Camera parameter calibration method, device and equipment

Abstract

Embodiments of the present application provide a camera parameter calibration method, device and equipment, the camera parameter calibration method comprises: obtaining a first image photographed by a camera to be calibrated; determining a gray image corresponding to the first image; processing the gray image through a first preset model to obtain internal parameters of the camera, the internal parameters comprising focal length, center and distortion information of the camera. The calibration method provided by the embodiments of the present application improves the accuracy of camera parameter calibration.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a camera parameter calibration method, device and equipment. BACKGROUND

[0002] The camera can be used for shooting images. In order to ensure that the images shot by the camera can correctly reflect the positional relationship of objects in the environment, the parameters of the camera need to be calibrated.

[0003] In the related art, unified parameter calibration is usually performed on the same batch of cameras in the production process of the cameras. When the production process of the cameras is unstable, the structures of the cameras in the same batch will change greatly. If unified parameter calibration is adopted, the accuracy of camera parameter calibration will be low. SUMMARY

[0004] The present application relates to a camera parameter calibration method, device and equipment, which improves the accuracy of camera parameter calibration.

[0005] In a first aspect, an embodiment of the present application provides a camera parameter calibration method, comprising:

[0006] obtaining a first image shot by a camera to be calibrated;

[0007] determining a gray image corresponding to the first image;

[0008] processing the gray image through a first preset model to obtain internal parameters of the camera, wherein the internal parameters comprise a focal length of the camera, a center of the camera and distortion information of the camera.

[0009] In a possible implementation, the first preset model comprises an input layer, a feature extraction layer, a first output layer and a second output layer; processing the gray image through the first preset model to obtain the internal parameters of the camera comprises:

[0010] performing normalization processing on the gray image through the input layer to obtain a normalized image;

[0011] processing the normalized image through the feature extraction layer to obtain image features of the normalized image;

[0012] processing the image features through the first output layer to obtain the focal length of the camera and the center of the camera;

[0013] processing the image features through the second output layer to obtain the distortion information of the camera.

[0014] In a possible implementation, the feature extraction layer comprises a plurality of convolution layers and a concatenation layer; processing the normalized image through the feature extraction layer to obtain image features of the normalized image comprises:

[0015] performing N times of convolution on the normalized image through the plurality of convolution layers to obtain N image features, wherein the convolution scales of the N times of convolution are different, and N is an integer greater than or equal to 2;

[0016] performing concatenation processing on the N image features through the concatenation layer to obtain the image features of the normalized image.

[0017] In a possible implementation, the first image is an image obtained by photographing a calibration site, and the calibration site comprises at least one preset corner point; the method further comprises:

[0018] processing the grayscale image through a second preset model to obtain pixel coordinates of each corner point in the grayscale image;

[0019] determining world coordinates of each corner point in a world coordinate system;

[0020] determining external parameters of the camera according to the pixel coordinates of each corner point in the grayscale image and the world coordinates of each corner point in the world coordinate system, wherein the external parameters comprise rotation information and translation information.

[0021] In a possible implementation, the second preset model comprises an input layer, a convolution layer, a concatenation layer and a deconvolution layer; processing the grayscale image through the second preset model to obtain pixel coordinates of each corner point in the grayscale image comprises:

[0022] performing normalization processing on the grayscale image through the input layer of the second preset model to obtain a second normalized image;

[0023] performing M times of convolution on the second normalized image through the convolution layer of the second preset model to obtain M image features, wherein the convolution scales of the M times of convolution are different, and M is an integer greater than or equal to 2;

[0024] performing concatenation processing on the M image features through the concatenation layer of the second preset model to obtain image features of the second normalized image;

[0025] processing the image features of the second normalized image through the deconvolution layer to obtain the pixel coordinates of each corner point in the grayscale image.

[0026] In a second aspect, the embodiments of the present application provide a camera parameter calibration device, comprising an acquisition module, a first determination module and a first processing module, wherein

[0027] The acquisition module is configured to acquire a first image captured by a camera to be calibrated.

[0028] The first determination module is configured to determine a gray image corresponding to the first image.

[0029] The first processing module is configured to process the gray image through a first preset model to obtain internal parameters of the camera, wherein the internal parameters comprise a focal length of the camera, a center of the camera and distortion information of the camera.

[0030] In a possible implementation, the first preset model comprises an input layer, a feature extraction layer, a first output layer and a second output layer; and the first processing module is specifically configured to:

[0031] perform normalization processing on the gray image through the input layer to obtain a normalized image;

[0032] perform processing on the normalized image through the feature extraction layer to obtain image features of the normalized image;

[0033] perform processing on the image features through the first output layer to obtain the focal length of the camera and the center of the camera;

[0034] perform processing on the image features through the second output layer to obtain the distortion information of the camera.

[0035] In a possible implementation, the feature extraction layer comprises a multi-convolution layer and a splicing layer; and the first processing module is specifically configured to:

[0036] perform N times of convolution on the normalized image through the multi-convolution layer to obtain N image features, wherein the convolution scales of the N times of convolution are different, and N is an integer greater than or equal to 2;

[0037] perform splicing processing on the N image features through the splicing layer to obtain the image features of the normalized image.

[0038] In a possible implementation, the first image is an image obtained by capturing a calibration site, and the calibration site comprises at least one preset corner point; and the device further comprises a second processing module, a second determination module and a third determination module, wherein the second processing module is configured to:

[0039] The second processing module is configured to process the gray image through a second preset model to obtain pixel coordinates of each corner point in the gray image.

[0040] The second determining module is configured to determine world coordinates of each corner point in a world coordinate system.

[0041] The third determining module is configured to determine external parameters of the camera according to the pixel coordinates of each corner point in the grayscale image and the world coordinates of each corner point in the world coordinate system, the external parameters including rotation information and translation information.

[0042] In a possible implementation, the second preset model includes an input layer, a convolution layer, a concatenation layer and a deconvolution layer; and the second processing module is specifically configured to:

[0043] perform normalization processing on the grayscale image through the input layer of the second preset model to obtain a second normalized image;

[0044] perform M times of convolution on the second normalized image through the convolution layer of the second preset model to obtain M image features, the M times of convolution having different convolution scales, and M being an integer greater than or equal to 2;

[0045] perform concatenation processing on the M image features through the concatenation layer of the second preset model to obtain an image feature of the second normalized image;

[0046] perform processing on the image feature of the second normalized image through the deconvolution layer to obtain the pixel coordinates of each corner point in the grayscale image.

[0047] In a third aspect, an embodiment of the present application provides a parameter calibration device for a camera, including a processor, a memory, and the parameter calibration method according to any one of the first aspect.

[0048] The memory stores computer-executable instructions.

[0049] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the parameter calibration method according to any one of the first aspect.

[0050] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to implement the parameter calibration method according to any one of the first aspect when executed by a processor.

[0051] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the parameter calibration method according to any one of the first aspect.

[0052] The embodiment of the present application provides a camera parameter calibration method, device and equipment, first, a first image photographed by a camera to be calibrated is acquired, a gray image of the first image is obtained after processing the first image, the internal parameters of the camera are obtained by processing the gray image through a first preset model. The above method calibrates each camera separately through a neural network, compared with the method of uniformly calibrating the same batch of cameras, the accuracy of camera parameter calibration is improved.

[0053] In a sixth aspect, the embodiment of the present application provides a model training method, comprising:

[0054] M sample images corresponding to each sample camera are acquired, the M sample images are images photographed by the sample camera from different angles on a preset object, and M is an integer greater than 1;

[0055] Sample internal parameters of each sample camera are calculated according to M sample images corresponding to each sample camera respectively;

[0056] A calibration image obtained by photographing a calibration site by each sample camera is acquired;

[0057] An initial model is trained according to the sample internal parameters of each sample camera and the calibration image obtained by photographing by each sample camera, to obtain an internal parameter model, and the internal parameter model is used to determine the internal parameters of the camera.

[0058] In a possible implementation, the initial model is trained according to the sample internal parameters of each sample camera and the calibration image obtained by photographing by each sample camera, to obtain the internal parameter model, comprising:

[0059] A plurality of groups of training samples are determined according to the sample internal parameters of each sample camera and the calibration image obtained by photographing by each sample camera, and each group of training samples includes a sample camera sample internal parameter and a calibration image obtained by photographing;

[0060] The initial model is trained according to the plurality of groups of training samples, to obtain the internal parameter model.

[0061] In a possible implementation, for any one group of training samples, the sample internal parameters of the one group of training samples include sample focal length, sample center and sample distortion information; and the initial model is trained according to the one group of training samples, comprising:

[0062] The calibration image in the one group of training samples is processed through the initial model, to obtain actual output information, and the actual output information includes actual focal length, actual center and actual distortion information;

[0063] According to the actual output information and the sample internal parameter in the set of training samples, the initial model is trained, and model parameters of the initial model are updated.

[0064] In a possible implementation, according to the actual output information and the sample internal parameter in the set of training samples, the initial model is trained, and model parameters of the initial model are updated, including:

[0065] According to the actual focal length, the actual center, the sample focal length and the sample center, a first loss function is determined.

[0066] According to the actual distortion information and the sample distortion information, a second loss function is determined.

[0067] According to the first loss function and the second loss function, model parameters of the initial model are updated.

[0068] In a seventh aspect, an embodiment of the present application provides a model training device, including a first acquisition module, a calculation module, a second acquisition module and a training module, wherein,

[0069] The first acquisition module is configured to acquire M sample images corresponding to each sample camera, the M sample images being images obtained by the sample camera from different angles on a preset object, and M being an integer greater than 1.

[0070] The calculation module is configured to calculate sample internal parameters of each sample camera according to the M sample images corresponding to each sample camera, respectively.

[0071] The second acquisition module is configured to acquire calibration images obtained by each sample camera on a calibration site.

[0072] The training module is configured to train an initial model according to the sample internal parameters of each sample camera and the calibration images obtained by each sample camera, to obtain an internal parameter model, and the internal parameter model is configured to determine internal parameters of a camera.

[0073] In a possible implementation, the training module is specifically configured to:

[0074] According to the sample internal parameters of each sample camera and the calibration images obtained by each sample camera, a plurality of sets of training samples are determined, and each set of training samples includes a sample camera sample internal parameter and a calibration image obtained by shooting.

[0075] The initial model is trained according to the plurality of sets of training samples, respectively, to obtain the internal parameter model.

[0076] In a possible implementation, the sample internal parameters of the training sample set include sample focal length, sample center and sample distortion information; and the training module is specifically configured to:

[0077] The calibration image in the training sample set is processed by the initial model to obtain actual output information, the actual output information including actual focal length, actual center and actual distortion information;

[0078] The initial model is trained according to the actual output information and the sample internal parameters in the training sample set, and the model parameters of the initial model are updated.

[0079] In a possible implementation, the training module is specifically configured to:

[0080] A first loss function is determined according to the actual focal length, the actual center, the sample focal length and the sample center;

[0081] A second loss function is determined according to the actual distortion information and the sample distortion information;

[0082] The model parameters of the initial model are updated according to the first loss function and the second loss function.

[0083] In an eighth aspect, an embodiment of the present application provides a model training device, including: a processor, a memory;

[0084] The memory stores computer execution instructions;

[0085] The processor executes the computer execution instructions stored in the memory, so that the processor executes the model training method in any one of the sixth aspect.

[0086] In a ninth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium storing computer execution instructions, when the computer execution instructions are executed by a processor, the computer execution instructions are used to implement the model training method in any one of the fifth aspect.

[0087] In a tenth aspect, an embodiment of the present application provides a computer program product, including a computer program, when the computer program is executed by a processor, the computer program can implement the model training method in any one of the sixth aspect.

[0088] The embodiment of the present application provides a model training method, device and equipment, first, M sample images of a preset object photographed from different angles by each sample camera are acquired, sample internal parameters of each sample camera are calculated according to the M sample images; a calibration image of a calibration site photographed by each sample camera is acquired, and an initial model is trained according to the sample internal parameters of each sample camera and the photographed calibration image, to obtain an internal parameter model. The above method can reduce the training time of the internal parameter model, and when the above internal parameter model is applied to determining the internal parameters of the camera, the accuracy of camera parameter calibration can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0089] Figure 1 A schematic diagram of four coordinate systems involved in the embodiment of the present application;

[0090] Figure 2 A schematic diagram of an application scenario provided by the embodiment of the present application;

[0091] Figure 3 A flowchart of a camera parameter calibration method provided by the embodiment of the present application Figure 1 ;

[0092] Figure 4 A schematic diagram of the network architecture of the first preset model provided by the embodiment of the present application;

[0093] Figure 5 A flowchart of a camera parameter calibration method provided by the embodiment of the present application Figure 2 ;

[0094] Figure 6 A schematic diagram of the network architecture of the second preset model provided by the embodiment of the present application;

[0095] Figure 7 A flowchart of a model training method provided by the embodiment of the present application Figure 1 ;

[0096] Figure 8 A structural schematic diagram of a camera parameter calibration device provided by the embodiment of the present application Figure 1 ;

[0097] Figure 9 A structural schematic diagram of a camera parameter calibration device provided by the embodiment of the present application Figure 2 ;

[0098] Figure 10 A structural schematic diagram of a camera parameter calibration device provided by the embodiment of the present application

[0099] Figure 11 A structural schematic diagram of a model training device provided by the embodiment of the present application;

[0100] Figure 12 This is a schematic diagram of the structure of a model training device provided in an embodiment of this application. Detailed Implementation

[0101] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the embodiments of this application. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0102] To facilitate understanding, the following will be combined with... Figure 1 The coordinate system involved in the embodiments of this application will be described in conjunction with... Figure 2 The application scenarios applicable to the embodiments of this application are described below.

[0103] Figure 1 This is a schematic diagram illustrating the four coordinate systems involved in the embodiments of this application. Please refer to... Figure 1 Including the world coordinate system O w -X w Y w C w Camera coordinate system O c -X c Y c C c Image coordinate system o(u0,v0)-xy and pixel coordinate system o uv -uv, where the world coordinate system O w -X w Y w C w It can describe the position of an object in the real world; camera coordinate system O c -X c Y c C c The image coordinate system is a coordinate system established on the camera, describing the object's position from the camera's perspective. It can serve as an intermediary between the world coordinate system and the image / pixel coordinate system. The image coordinate system o(u0,v0)-xy describes the projection and transmission relationship of the object from the camera coordinate system to the image coordinate system during the imaging process. The pixel coordinate system o uv -uv can describe the coordinates of the pixels of an object on a digital image after imaging.

[0104] p is the image point of point P in the world coordinate system, with coordinates (x, y) in the image coordinate system and (u, v) in the pixel coordinate system.

[0105] Figure 2 An application scenario schematic diagram is provided for an embodiment of the present application. Please refer to Figure 2 , which comprises an unmanned vehicle 201, and a vehicle-mounted camera is arranged in the unmanned vehicle.

[0106] A plurality of vehicle-mounted cameras can be arranged on the unmanned vehicle 201, for example, Figure 2 Three vehicle-mounted cameras A, B and C respectively installed at the front part of the vehicle, the side of the vehicle and the rear part of the vehicle are shown in FIG.

[0107] In order to ensure that the image information can accurately reflect the information in the environment, it is necessary to establish the corresponding relationship between the object pixel coordinates and the world coordinates, and the corresponding relationship between the object pixel coordinates and the world coordinates is determined by the parameters of the camera. For example, there is a point P on the cuboid 1, and the world coordinates of the point P are (X w , Y w , C w ). After imaging by the camera, the coordinates of the point P in the pixel coordinate system are (u, v), and the corresponding relationship between the two is as follows:

[0108]

[0109] Wherein, Z c is a scale factor (Z c is not 0), f is the effective focal length (the distance from the optical center to the image plane), and is the homogeneous coordinates of the spatial point in the camera coordinate system; f x =f / d x is the normalized focal length on the x-axis; f y =f / d y is the normalized focal length on the y-axis; f is the focal length of the camera; (u0, v0) represents the offset of the camera optical axis in the pixel coordinate system, that is, the coordinates of the image coordinate origin in the pixel coordinate system; R represents the rotation matrix, and T represents the offset vector.

[0110] In the related art, the unified parameter calibration is usually performed on the same batch of cameras in the production process of the camera. When the production process of the camera is unstable, the structure of the same batch of cameras will change greatly, and if the unified parameter calibration is adopted, the accuracy of the camera parameter calibration will be low.

[0111] To solve the above technical problems, the application provides a camera parameter calibration method, first, a first image shot by a camera to be calibrated is obtained, the first image is processed to obtain a gray image, the gray image is processed by a first preset model to obtain the internal parameters of the camera. The above method calibrates each camera separately through a neural network, compared with the method of uniformly calibrating the same batch of cameras, the accuracy of camera parameter calibration is improved.

[0112] The technical solutions shown in the application will be described in detail below through specific embodiments. It should be noted that the following embodiments can exist independently or be combined with each other, and the same or similar contents will not be repeated in different embodiments.

[0113] Figure 3 A flowchart of a camera parameter calibration method provided by an embodiment of the application Figure 1 . Please refer to Figure 3 , the method can include:

[0114] S301, obtaining a first image shot by a camera to be calibrated.

[0115] The execution subject of the embodiment of the application can be a vehicle or a server, or a camera parameter calibration device provided in the vehicle or the server, the camera parameter calibration device can be realized by software or by the combination of software and hardware.

[0116] The parameter can be the internal parameter of the camera.

[0117] The first image can be an image obtained by shooting a calibration site, and the calibration site includes a calibration image, and the calibration image can be a checkerboard image.

[0118] The first image can be a YUV image. YUV is a common image format, Y represents luminance, and uv combined can represent chrominance.

[0119] S302, determining a gray image corresponding to the first image.

[0120] If the first image is a YUV image, the gray image can be a Y channel image. The first image can also be an RGB image, and the first image is converted into a gray image through a gray calculation formula Gray = R 0.299 + G 0.587 + B 0.114, wherein R is the value of the R channel of the RGB image, G is the value of the G channel of the RGB image, and B is the value of the B channel of the RGB image.

[0121] S303, processing the gray-scale image through a first preset model to obtain the internal parameter of the camera.

[0122] The internal parameter includes a focal length of the camera, a center of the camera, and distortion information of the camera. The distortion information can include radial distortion information. The gray-scale image is input into the first preset model pre-trained to obtain the internal parameter of the camera. The first preset model is trained according to a large number of sample calibration images and sample internal parameters corresponding to each sample.

[0123] The first preset model can include an input layer, a feature extraction layer, a first output layer, and a second output layer.

[0124] Optionally, processing the gray-scale image through the first preset model to obtain the internal parameter of the camera includes the following steps.

[0125] The gray-scale image is normalized through the input layer to obtain a normalized image. The normalized image is processed through the feature extraction layer to obtain image features of the normalized image. The image features are processed through the first output layer to obtain the focal length of the camera and the center of the camera. The image features are processed through the second output layer to obtain the distortion information of the camera.

[0126] The feature extraction layer can include a plurality of convolution layers and a concatenation layer.

[0127] Optionally, processing the normalized image through the feature extraction layer to obtain the image features of the normalized image includes the following steps.

[0128] The normalized image is convolved N times through the plurality of convolution layers to obtain N image features. The convolution scales of the N times of convolution are different, and N is an integer greater than or equal to 2. The N image features are processed through the concatenation layer to obtain the image features of the normalized image.

[0129] For ease of understanding, the following describes the first preset model in combination with Figure 4 The network architecture of the first preset model is described.

[0130] Figure 4 The network architecture of the first preset model provided by the embodiments of the present application is shown in the schematic diagram. Please refer to Figure 4 , including an input layer, a plurality of convolution layers, a concatenation layer, a first output layer, and a second output layer, Figure 4The embodiment shown illustrates the number of convolutional layers as 2. The gray-scale image first enters the input layer, after mean value reduction and normalization processing, enters the multi-convolutional layer, and is convolved in two different size scales to extract image features of different scales; the two image features are spliced in the dimension, and then input into the first output layer or the second output layer, and the internal parameters of the camera are obtained through multiple convolutions, wherein the first output layer obtains the focal length f x , f y and the center u0, v0, and the second output layer obtains the distortion coefficients K1, K2, K3, K4 of the camera.

[0131] In Figure 3 the embodiment shown, a first image captured by the camera is first obtained, Y channel data of the first image is extracted, the first image is converted into a gray-scale image, and the gray-scale image is processed by a first preset model to obtain the internal parameters of the camera. The above method calibrates a single camera through a neural network, and compared with the method of uniformly calibrating the same batch of cameras, the accuracy of camera parameter calibration is improved.

[0132] Figures 3-4 The embodiment shown illustrates a camera internal parameter calibration method, and the camera external parameter calibration method is described below. Figures 5-6

[0133] Figure 5 A flowchart of a camera parameter calibration method provided by an embodiment of the present application is shown in Figure 2 . Please refer to Figure 5 , which can include the following steps.

[0134] S501, a first image captured by a camera to be calibrated is obtained.

[0135] The first image is an image obtained by capturing a calibration site, and the image includes at least one preset corner point in the calibration site. The corner point can be the intersection of two straight lines on a checkerboard image.

[0136] It should be noted that the execution process of S501 can refer to the execution process of S301, which will not be described here.

[0137] S502, a gray-scale image corresponding to the first image is determined.

[0138] It should be noted that the execution process of S502 can refer to the execution process of S302, which will not be described here.

[0139] S503, the gray-scale image is processed by a second preset model to obtain the pixel coordinates of each corner point in the gray-scale image.

[0140] ​The second preset model can include an input layer, a convolution layer, a splicing layer, and a deconvolution layer.

[0141] Optionally, the pixel coordinates of each corner point in the gray image are obtained by processing the gray image through the second preset model, including:

[0142] The gray image is normalized through the input layer of the second preset model to obtain a second normalized image; the second normalized image is convolved M times through the convolution layer of the second preset model to obtain M image features, the convolution scales of the M times of convolution are different, and M is an integer greater than or equal to 2; the M image features are spliced through the splicing layer of the second preset model to obtain the image features of the second normalized image; and the pixel coordinates of each corner point in the gray image are obtained by processing the image features of the second normalized image through the deconvolution layer.

[0143] For ease of understanding, the following will be combined with Figure 6 The network architecture of the second preset model is described.

[0144] Figure 6 The network architecture of the second preset model provided by the embodiments of the present application is shown in the schematic diagram. Please refer to Figure 6 , including an input layer, a convolution layer, a splicing layer, and a deconvolution layer, Figure 6 The embodiment shown in the figure is described with the number of convolution layers being 2. The gray image first enters the input layer, is processed by mean value reduction and normalization, and then enters the multi-convolution layer to perform convolution of two different sizes to extract image features of different scales; the two image features are spliced in the dimension, and then input into the deconvolution layer to obtain the pixel coordinates (u, v) of the corner points in the gray image through multiple deconvolutions.

[0145] S504, determining the world coordinates of each corner point in the world coordinate system.

[0146] S505, determining the external parameters of the camera according to the pixel coordinates of each corner point in the gray image and the world coordinates of each corner point in the world coordinate system.

[0147] The external parameters can include rotation information and translation information.

[0148] The external parameters of the camera can be determined through the corresponding relationship between the pixel coordinates and the world coordinates of the corner points.

[0149] For example, the pixel coordinates of the corner point A are (u, v), the world coordinates are (X w , Y w , C w ), and the external parameters, the rotation matrix R and the offset vector T, of the corner point A can be determined through the following formula:

[0150]

[0151] wherein, Z c is a scale factor (Z c is not 0), f is an effective focal length (distance from optical center to image plane), and is the homogeneous coordinate of a spatial point in the camera coordinate system; f x = f / d x is the normalized focal length in the x-axis; f y = f / d y is the normalized focal length in the y-axis; f is the focal length of the camera; and (u0, v0) represents the offset of the camera optical axis in the pixel coordinate system.

[0152] In Figure 5 the embodiment shown, a first image captured by a camera is first obtained, the first image including a preset corner point, Y channel data of the first image is extracted, the first image is converted into a grayscale image, the grayscale image is processed by a second preset model to obtain pixel coordinates of the corner point in the grayscale image, world coordinates of the corner point in a world coordinate system are obtained, and external parameters of the camera are determined according to a corresponding relationship between the pixel coordinates and the world coordinates. The above method calibrates the external parameters of the camera by using a neural network, and improves the accuracy of camera parameter calibration.

[0153] According to the embodiments shown in Figure 3 and Figure 4 , a first preset model is used in the parameter calibration method of the camera. In the following, the training method of the first preset model is described in combination with the embodiment shown in Figures 7-8 .

[0154] Figure 7 A flowchart of a model training method provided by an embodiment of the present application is shown in Figure 1 . Please refer to Figure 7 , the method can include the following steps.

[0155] S701, M sample images corresponding to each sample camera are obtained.

[0156] The M sample images can be images of a preset object captured from different angles by a sample camera, that is, the M sample images can be images with different distortions. M is an integer greater than 1.

[0157] The sample images can be YUV images.

[0158] The sample images can be checkerboard images. The checkerboard can be a 6 8 checkerboard, and the length of each square can be 3 cm.

[0159] S702, calculate the sample internal parameters of each sample camera according to the M sample images corresponding to each sample camera.

[0160] The intrinsic parameters of the camera can be calculated by using OpenCV.

[0161] The sample internal parameters can be the real internal parameters of the sample camera.

[0162] S703, obtain the calibration images obtained by each sample camera shooting the calibration site.

[0163] The calibration images can be grayscale images.

[0164] S704, train the initial model according to the sample internal parameters of each sample camera and the calibration images obtained by each sample camera, to obtain an intrinsic parameter model.

[0165] The intrinsic parameter model is used to determine the internal parameters of the camera.

[0166] The initial model can be trained according to the sample internal parameters of each sample camera and the calibration images obtained by each sample camera to obtain the intrinsic parameter model in the following manner:

[0167] Determine a plurality of groups of training samples according to the sample internal parameters of each sample camera and the calibration images obtained by each camera, each group of training samples including a sample camera sample internal parameter and a calibration image obtained by shooting; train the initial model according to the plurality of groups of training samples to obtain the intrinsic parameter model.

[0168] The sample internal parameters include sample focal length, sample center and sample distortion information.

[0169] The initial model can be trained according to a group of training samples in the following manner:

[0170] Process the calibration images in the group of training samples by the initial model to obtain actual output information, the actual output information including actual focal length, actual center and actual distortion information; determine a first loss function according to the actual focal length, the actual center, the sample focal length and the sample center; determine a second loss function according to the actual distortion information and the sample distortion information; update the model parameters of the initial model according to the first loss function and the second loss function.

[0171] The loss function can be:

[0172]

[0173] wherein y i represents the actual output information, f(x i ) represents the sample internal parameters.

[0174] In the intrinsic parameter model training, the model parameters need to be optimized by using a loss function. If the loss function does not converge, the model parameters of the intrinsic parameter model need to be repeatedly updated by using the training data until the loss function converges, the training stops, and the trained intrinsic parameter model is obtained.

[0175] In Figure 7 In the embodiment shown, M sample images corresponding to each sample camera are first acquired. The sample intrinsic parameters (real intrinsic parameters) of each sample camera are calculated according to the M sample images corresponding to each sample camera, respectively. The calibration images obtained by each sample camera shooting the calibration site are acquired. The initial model is trained according to the sample intrinsic parameters (real intrinsic parameters) of each sample camera and the calibration images obtained by each sample camera, and the intrinsic parameter model is obtained. The above method can reduce the training time of the intrinsic parameter model. When the above intrinsic parameter model is applied to determining the intrinsic parameters of the camera, the accuracy of the camera parameter calibration can be improved.

[0176] Figure 8 A structure diagram of a camera parameter calibration device provided by an embodiment of the present application Figure 1 . Please refer to Figure 8 The parameter calibration device 10 can include an acquisition module 11, a first determination module 12, and a first processing module 13, wherein

[0177] The acquisition module 11 is configured to acquire a first image shot by a camera to be calibrated.

[0178] The first determination module 12 is configured to determine a gray image corresponding to the first image.

[0179] The first processing module 13 is configured to process the gray image by using a first preset model to obtain intrinsic parameters of the camera, wherein the intrinsic parameters include a focal length of the camera, a center of the camera, and distortion information of the camera.

[0180] In a possible implementation, the first preset model includes an input layer, a feature extraction layer, a first output layer, and a second output layer. The first processing module 13 is specifically configured to:

[0181] perform normalization processing on the gray image by using the input layer to obtain a normalized image;

[0182] perform processing on the normalized image by using the feature extraction layer to obtain image features of the normalized image;

[0183] perform processing on the image features by using the first output layer to obtain the focal length of the camera and the center of the camera;

[0184] The image features are processed by the second output layer to obtain distortion information of the camera.

[0185] In a possible implementation, the feature extraction layer includes a multi-convolution layer and a concatenation layer; and the first processing module 13 is specifically configured to:

[0186] The normalized image is convolved N times by the multi-convolution layer to obtain N image features, the convolution scales of the N times of convolution are different, and N is an integer greater than or equal to 2;

[0187] The N image features are processed by the concatenation layer to obtain image features of the normalized image.

[0188] Figure 9 A structure diagram of a camera parameter calibration device provided by an embodiment of the present application Figure 2 . Please refer to Figure 9 , on the basis of Figure 8 , the first image is an image obtained by photographing a calibration site, and the calibration site includes at least one preset corner point; the device 10 further includes a second processing module 14, a second determination module 15, and a third determination module 16, wherein the

[0189] The second processing module 14 is configured to process the gray image by a second preset model, and each corner point has a pixel coordinate in the gray image;

[0190] The second determination module 15 is configured to determine a world coordinate of each corner point in a world coordinate system;

[0191] The third determination module 16 is configured to determine external parameters of the camera according to the pixel coordinate of each corner point in the gray image and the world coordinate of each corner point in the world coordinate system, wherein the external parameters include rotation information and translation information.

[0192] In a possible implementation, the second preset model includes an input layer, a convolution layer, a concatenation layer, and a deconvolution layer; and the second processing module 14 is specifically configured to:

[0193] The gray image is normalized by the input layer of the second preset model to obtain a second normalized image;

[0194] The second normalized image is convolved M times by the convolution layer of the second preset model to obtain M image features, the convolution scales of the M times of convolution are different, and M is an integer greater than or equal to 2;

[0195] The M image features are spliced by a splicing layer of the second preset model to obtain image features of the second normalized image.

[0196] The image features of the second normalized image are processed by the deconvolution layer to obtain pixel coordinates of each corner point in the grayscale image.

[0197] The parameter calibration device 10 of the camera provided by the embodiment of the present application can execute the technical solutions shown in the parameter calibration method embodiments of the camera described above, and the implementation principles and beneficial effects are similar, which will not be repeated here.

[0198] Figure 10 A structural schematic diagram of a parameter calibration device of a camera provided by the embodiment of the present application is shown in FIG. 20. Figure 10 The parameter calibration device 20 of the camera can include a memory 21 and a processor 22. The memory 21 and the processor 22 are connected to each other through a bus 23.

[0199] The memory 21 stores computer execution instructions.

[0200] The processor 22 executes the computer execution instructions stored in the memory, so that the processor executes the parameter calibration method of the camera described in any of the above.

[0201] Figure 10 The parameter calibration device of the camera shown in the embodiment can execute the technical solutions shown in the parameter calibration method embodiments of the camera described above, and the implementation principles and beneficial effects are similar, which will not be repeated here.

[0202] The embodiment of the present application provides a computer readable storage medium, the computer readable storage medium stores computer execution instructions, when the computer execution instructions are executed by the processor, the computer execution instructions are used to implement the parameter calibration method of the camera described in any of the above.

[0203] The embodiment of the present application provides a computer program product, including a computer program, when the computer program is executed by the processor, the computer program can implement the parameter calibration method of the camera described in any of the above.

[0204] Figure 11 A structural schematic diagram of a model training device provided by the embodiment of the present application is shown in FIG. 30. Figure 11 The model training device 30 can include a first acquisition module 31, a calculation module 32, a second acquisition module 33 and a training module 34, wherein,

[0205] The first obtaining module 31 is configured to obtain M sample images corresponding to each sample camera, the M sample images being images of a preset object captured by the sample camera from different angles, and M being an integer greater than 1.

[0206] The calculation module 32 is configured to calculate a sample internal parameter of each sample camera according to the M sample images corresponding to each sample camera.

[0207] The second obtaining module 33 is configured to obtain a calibration image of a calibration site captured by each sample camera.

[0208] The training module 34 is configured to train an initial model according to the sample internal parameter of each sample camera and the calibration image captured by each sample camera, to obtain an internal parameter model, the internal parameter model being used to determine an internal parameter of a camera.

[0209] In a possible implementation, the training module 34 is specifically configured to:

[0210] determine a plurality of groups of training samples according to the sample internal parameter of each sample camera and the calibration image captured by each sample camera, each group of training samples including a sample camera sample internal parameter and a captured calibration image;

[0211] train the initial model according to the plurality of groups of training samples, to obtain the internal parameter model.

[0212] In a possible implementation, for any one group of training samples, the sample internal parameter of the one group of training samples includes sample focal length, sample center and sample distortion information; and the training module 34 is specifically configured to:

[0213] process the calibration image in the one group of training samples by using the initial model, to obtain actual output information, the actual output information including actual focal length, actual center and actual distortion information;

[0214] train the initial model according to the actual output information and the sample internal parameter in the one group of training samples, to update a model parameter of the initial model.

[0215] In a possible implementation, the training module 34 is specifically configured to:

[0216] determine a first loss function according to the actual focal length, the actual center, the sample focal length and the sample center;

[0217] determine a second loss function according to the actual distortion information and the sample distortion information;

[0218] According to the first loss function and the second loss function, a model parameter of the initial model is updated.

[0219] The model training apparatus 30 provided by the embodiments of the present application can execute the technical solutions shown in the method embodiments, and the implementation principles and beneficial effects are similar, which will not be repeated here.

[0220] Figure 12 A structural schematic diagram of a model training device provided by the embodiments of the present application is provided. Please refer to Figure 12 The model training device 40 can include a memory 41 and a processor 42. The memory 41 and the processor 42 are connected to each other through a bus 43.

[0221] The memory 41 stores computer execution instructions.

[0222] The processor 42 executes the computer execution instructions stored in the memory 41, so that the processor 42 executes the action of the training method of the network.

[0223] Figure 12 The model training device 40 shown in the embodiments can execute the technical solutions shown in the above model training method embodiments, and the implementation principles and beneficial effects are similar, which will not be repeated here.

[0224] The embodiments of the present application provide a computer readable storage medium, which stores computer execution instructions, and when the computer execution instructions are executed by a processor, the computer execution instructions are used to implement the above model training method.

[0225] The embodiments of the present application provide a computer program product, which includes a computer program, and when the computer program is executed by a processor, the computer program can implement the above model training method.

[0226] The above description is only the preferred embodiments of the present application and the explanation of the applied technical principles. Those skilled in the art should understand that the disclosed range of the present application is not limited to the technical solutions formed by the specific combinations of the above technical features, and also covers other technical solutions formed by any combination of the above technical features or equivalent features without departing from the disclosed concept. For example, the above features are replaced with the technical features disclosed in the present application (but not limited to) having similar functions to form technical solutions.

[0227] Moreover, while operations are depicted in a particular order, this should not be understood as requiring such an order nor infringing on the scope of the application. Certain of the operations described in the specification can be combined with one another and can be performed in any order. In addition, certain of the operations described in the specification can be optional depending on the circumstances. Various other variations of these operations were also possible, as indicated to some extent by the specificity of the claims attached hereto.

[0228] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A method for calibrating parameters of a camera, characterized in that, The method comprises: obtaining a first image photographed by a camera to be calibrated; determining a gray image corresponding to the first image; processing the gray image through a first preset model to obtain internal parameters of the camera, the internal parameters comprising focal length, center and distortion information of the camera, and the distortion information comprising radial distortion information; the first preset model comprises an input layer, a plurality of convolution layers, a splicing layer, a first output layer and a second output layer; processing the gray image through the first preset model to obtain the internal parameters of the camera comprises: performing mean value reduction and normalization processing on the gray image through the input layer to obtain a normalized image; performing N times of convolution on the normalized image through the plurality of convolution layers to obtain N image features, the convolution scales of the N times of convolution being different, and N being an integer greater than or equal to 2; performing splicing processing on the N image features through the splicing layer to obtain image features of the normalized image; processing the image features through the first output layer to obtain the focal length and the center of the camera; processing the image features through the second output layer to obtain the distortion information of the camera.

2. The method of claim 1, wherein, The first image is an image obtained by photographing a calibration site, and the calibration site comprises at least one preset corner point; the method further comprises: processing the gray image through a second preset model to obtain pixel coordinates of each corner point in the gray image; determining world coordinates of each corner point in a world coordinate system; determining external parameters of the camera according to the pixel coordinates of each corner point in the gray image and the world coordinates of each corner point in the world coordinate system, the external parameters comprising rotation information and translation information.

3. The method of claim 2, wherein, The second preset model comprises an input layer, a convolution layer, a splicing layer and a deconvolution layer; processing the gray image through the second preset model to obtain the pixel coordinates of each corner point in the gray image comprises: performing normalization processing on the gray image through the input layer of the second preset model to obtain a second normalized image; performing M times of convolution on the second normalized image through the convolution layer of the second preset model to obtain M image features, the convolution scales of the M times of convolution being different, and M being an integer greater than or equal to 2; performing splicing processing on the M image features through the splicing layer of the second preset model to obtain image features of the second normalized image; processing the image features of the second normalized image through the deconvolution layer to obtain the pixel coordinates of each corner point in the gray image.

4. A model training method, comprising: The method comprises: obtaining M sample images corresponding to each sample camera, the M sample images being images photographed by the sample camera from different angles on a preset object, and M being an integer greater than 1; calculating sample internal parameters of each sample camera according to the M sample images corresponding to each sample camera, respectively; obtaining a calibration image photographed by each sample camera on a calibration site; The initial model is trained according to the sample internal parameters of each sample camera and the calibration images obtained by photographing each sample camera, and an internal parameter model is obtained, the internal parameter model being used to determine the internal parameters of the camera; The internal parameter model comprises an input layer, a plurality of convolution layers, a splicing layer, a first output layer and a second output layer; and the internal parameter model is specifically used for: normalizing the gray image through the input layer to obtain a normalized image; performing N times of convolution on the normalized image through the plurality of convolution layers to obtain N image features, the convolution scales of the N times of convolution being different, and N being an integer greater than or equal to 2; splicing the N image features through the splicing layer to obtain an image feature of the normalized image; processing the image feature through the first output layer to obtain the focal length of the camera and the center of the camera; processing the image feature through the second output layer to obtain distortion information of the camera, the distortion information comprising radial distortion information.

5. The method of claim 4, wherein, The initial model is trained according to the sample internal parameters of each sample camera and the calibration images obtained by photographing each sample camera, and an internal parameter model is obtained, including: a plurality of groups of training samples are determined according to the sample internal parameters of each sample camera and the calibration images obtained by photographing each sample camera, each group of training samples comprising a sample camera sample internal parameter and a photographed calibration image; the initial model is trained according to the plurality of groups of training samples to obtain the internal parameter model.

6. The method of claim 5, wherein, For any one group of training samples, the sample internal parameters of the one group of training samples comprise a sample focal length, a sample center and sample distortion information; training the initial model according to the one group of training samples comprises: processing the calibration image in the one group of training samples through the initial model to obtain actual output information, the actual output information comprising an actual focal length, an actual center and actual distortion information; training the initial model according to the actual output information and the sample internal parameters in the one group of training samples to update the model parameters of the initial model.

7. The method of claim 6, wherein, Training the initial model according to the actual output information and the sample internal parameters in the one group of training samples to update the model parameters of the initial model comprises: determining a first loss function according to the actual focal length, the actual center, the sample focal length and the sample center; determining a second loss function according to the actual distortion information and the sample distortion information; updating the model parameters of the initial model according to the first loss function and the second loss function.

8. A parameter calibration device of a camera, characterized in that, The method comprises an acquisition module, a determination module and a processing module, wherein: the acquisition module is configured to acquire a first image photographed by a camera to be calibrated; the determination module is configured to determine a gray image corresponding to the first image; The processing module is configured to process the grayscale image by using a first preset model to obtain internal parameters of the camera, the internal parameters comprising a focal length of the camera, a center of the camera, and distortion information of the camera, the distortion information comprising radial distortion information. The first preset model comprises an input layer, a plurality of convolution layers, a splicing layer, a first output layer, and a second output layer; and the processing module is specifically configured to: perform normalization processing on the grayscale image by using the input layer to obtain a normalized image; perform N times of convolution on the normalized image by using the plurality of convolution layers to obtain N image features, the N times of convolution having different convolution scales, and N being an integer greater than or equal to 2; perform splicing processing on the N image features by using the splicing layer to obtain an image feature of the normalized image; perform processing on the image feature by using the first output layer to obtain the focal length of the camera and the center of the camera; and perform processing on the image feature by using the second output layer to obtain the distortion information of the camera.

9. A model training apparatus comprising: The first obtaining module, the calculation module, the second obtaining module, and the training module, wherein: The first obtaining module is configured to obtain M sample images corresponding to each sample camera, the M sample images being images obtained by the sample camera from different angles on a preset object, and M being an integer greater than 1; The calculation module is configured to calculate sample internal parameters of each sample camera according to the M sample images corresponding to each sample camera, respectively; The second obtaining module is configured to obtain a calibration image obtained by each sample camera on a calibration site; The training module is configured to train an initial model according to the sample internal parameters of each sample camera and the calibration image obtained by each sample camera to obtain an internal parameter model, the internal parameter model being configured to determine internal parameters of a camera; The internal parameter model comprises an input layer, a plurality of convolution layers, a splicing layer, a first output layer, and a second output layer; and the internal parameter model is specifically configured to: perform normalization processing on a grayscale image by using the input layer to obtain a normalized image; perform N times of convolution on the normalized image by using the plurality of convolution layers to obtain N image features, the N times of convolution having different convolution scales, and N being an integer greater than or equal to 2; perform splicing processing on the N image features by using the splicing layer to obtain an image feature of the normalized image; perform processing on the image feature by using the first output layer to obtain the focal length of the camera and the center of the camera; and perform processing on the image feature by using the second output layer to obtain the distortion information of the camera, the distortion information comprising radial distortion information.

10. An electronic device, comprising: The processor, the memory, the memory storing computer execution instructions, and the processor executing the computer execution instructions stored in the memory to make the processor execute the method of any one of claims 1-3 or the method of any one of claims 4-7. ​ ​ ​ 11. A computer readable storage medium, characterized in that, The computer readable storage medium stores computer executing instructions for implementing the method of any one of claims 1-3 or the method of any one of claims 4-7 when the computer executing instructions are executed by the processor.

12. A computer program product, characterised in that, The computer program is executed by the processor to implement the method of any one of claims 1-3 or the method of any one of claims 4-7.

Images