Camera calibration method based on deep learning, electronic equipment, medium and product

Abstract

The invention discloses a camera calibration method based on deep learning, electronic equipment, a medium and a product, and relates to the technical field of image processing, and the method comprises the steps: determining a first coordinate corresponding to at least one calibration point in a three-dimensional model corresponding to a to-be-detected product; determining a first image pixel coordinate corresponding to the at least one calibration point in the product image of the to-be-tested product according to a deep learning model; according to the first coordinates and the first image pixel coordinates belonging to the same calibration point, determining a homography matrix representing a coordinate conversion relation; determining the Euclidean distance from at least one calibration point to the camera according to the homography matrix so as to determine the distance from the camera to the to-be-detected product; and in response to the fact that the distance is not matched with the optimal depth-of-field position of the camera, adjusting the position parameter of the camera, and re-executing acquisition of the product image of the to-be-detected product until the latest obtained distance is matched with the optimal depth-of-field position of the camera. The distance between the camera and the calibration object can be accurately determined so as to adjust the position of the camera and improve the shooting effect of the camera.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to camera calibration methods, electronic devices, storage media, and computer program products based on deep learning. Background Technology

[0002] In the field of visual calibration, it is usually necessary to accurately determine the spatial distance between the camera and the calibration object (such as product parts with calibration points, such as product accessories, plastic parts, etc.), such as the Z-axis distance (i.e., the distance in the Z-axis direction), in order to determine the positional relationship between the camera and the calibration object, so that the camera can accurately capture high-precision images of the calibration object. However, currently, the camera height is adjusted manually based on human experience, which makes it impossible to accurately determine the distance between the camera and the calibration object, thus affecting the camera's shooting effect.

[0003] Therefore, accurately determining the distance between the camera and the calibrated object in order to adjust the camera position and thus improve the camera's shooting effect has become an urgent problem to be solved.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this application is to provide a camera calibration method, electronic device, medium, and product based on deep learning, aiming to solve the technical problem of how to accurately determine the distance between the camera and the calibration object in order to adjust the camera position and thus support the improvement of the camera's shooting effect.

[0006] To achieve the above objectives, this application proposes a deep learning-based camera calibration method, which includes the following steps: Determine the 3D model corresponding to the product under test that has at least one calibration point marked, and determine the first coordinates corresponding to at least one calibration point in the 3D model; Collect product images of the product to be tested, and determine the first image pixel coordinates in the product image corresponding to at least one calibration point based on a pre-trained deep learning model. Based on the first coordinates and the first image pixel coordinates belonging to the same calibration point, determine the homography matrix representing the coordinate transformation relationship; Determine the Euclidean distance from at least one calibration point to the camera based on the homography matrix, and determine the distance from the camera to the product under test based on the Euclidean distance; In response to a mismatch between the distance and the camera's optimal depth-of-field position, the camera's position parameters are adjusted, and the acquisition of product images of the test product is repeated until the latest distance matches the camera's optimal depth-of-field position.

[0007] Optionally, the homography matrix represents the coordinate transformation relationship between the product coordinate system corresponding to the first coordinate, the image coordinate system corresponding to the first image pixel coordinate, and the camera coordinate system of the camera.

[0008] Optionally, at least two calibration points are located on the same plane of the product under test. The steps for determining the Euclidean distance from at least one calibration point to the camera based on the homography matrix include: The camera's rotation matrix and translation vector are determined based on the homography matrix; For any given calibration point, determine the second coordinate of the calibration point in the preset world coordinate system; Determine the transformation relationship between the world coordinate system and the camera coordinate system. Using the transformation relationship, based on the rotation matrix, translation vector, and second coordinate, determine the third coordinate of the calibration point in the camera coordinate system. The Euclidean distance from the calibration point to the camera is calculated based on the third coordinate and the origin of the camera coordinate system within the camera coordinate system. The origin of the camera coordinate system is the optical center of the camera.

[0009] Alternatively, deep learning-based camera calibration methods also include: The preset training product image is input into the preset neural network model, and the second image pixel coordinates corresponding to the first prediction calibration point are determined based on the output of the neural network model. The first loss function value is determined based on the preset focus loss function and the second image pixel coordinates. The model parameters of the neural network model are updated in reverse based on the first loss function value until the preset training cutoff condition is met, thus obtaining the pre-trained neural network model. The pre-trained neural network model is then used as the pre-trained deep learning model.

[0010] Optionally, the step of determining the second image pixel coordinates corresponding to the first prediction calibration point based on the output of the neural network model includes: Based on the encoder of the neural network model, the training product image is mapped to a high-dimensional feature space to obtain the first feature map. Multiple convolutional layers are then used to convolve the first feature map, and the convolution results of each convolutional layer are fused to obtain a fused convolutional feature map. The decoder based on the neural network model uses two parallel deconvolution layers (at least one of which is deconvolutional) to deconvolve the fused convolutional feature maps respectively, to obtain the center heatmap and the offset heatmap. The pixel coordinates of the second image are determined by using the heatmap of the center and the offset.

[0011] Optionally, the step of determining the pixel coordinates of the second image based on the center heatmap and the offset includes: For each central heatmap, Gaussian distribution detection is performed on the central heatmap, and the coordinates of the third image pixel in the training product image corresponding to the first prediction calibration point are determined based on the Gaussian distribution detection results. Based on the offset heatmap corresponding to the center heatmap, the pixel coordinates of the third image are updated to obtain the pixel coordinates of the second image.

[0012] Optionally, the step of performing Gaussian distribution detection on the central heatmap and determining the coordinates of the third image pixel in the training product image corresponding to the first prediction calibration point based on the Gaussian distribution detection result includes: The preset Gaussian distribution detection conditions are determined, wherein the Gaussian distribution detection conditions include: when the result generated by Gaussian distribution processing of the center heatmap contains a non-zero real matrix, the center of the circle in the key feature region corresponding to the real matrix in the training product image is determined as the third image pixel coordinate corresponding to the first prediction calibration point; Gaussian distribution detection is performed on the central heat map based on the Gaussian distribution detection conditions; If the Gaussian distribution detection result includes the existence of a non-zero real number matrix, then the first real number in the real number matrix that is greater than a preset real number threshold is determined; The pixel region corresponding to each first real number in the training product image is used as the calibration point feature region, and the pixel coordinates of the third image are determined based on the calibration point feature region.

[0013] In addition, to achieve the above objectives, this application also proposes an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the deep learning-based camera calibration method described above.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the deep learning-based camera calibration method described above.

[0015] In addition, to achieve the above objectives, this application also proposes a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the deep learning-based camera calibration method described above.

[0016] In this application, the first coordinates corresponding to at least one calibration point in the 3D model of the product under test are determined by identifying the first coordinates of the calibration point. Based on a pre-trained deep learning model, the first image pixel coordinates of the predicted calibration point in the product image of the product under test are determined. Pairing is performed between the spatial coordinates and the first image pixel coordinates to determine the first coordinates and first image pixel coordinates belonging to the same calibration point. This determines the homography matrix representing the coordinate transformation relationship. The Euclidean distance from the calibration point to the camera is determined based on the homography matrix, thereby determining the distance from the camera to the product under test. This distance is then compared with the camera depth of field. When the optimal position does not match, the camera's position parameters are adjusted, and the steps of acquiring product images by the camera are iterated again until the latest distance matches the camera's optimal depth of field position. This distance is then used as the final determined distance between the camera and the calibration object (i.e., the product under test). This allows the camera's calibration distance to match the camera's optimal depth of field position, thus accurately determining the distance between the camera and the calibration object. The camera position can be adjusted based on this distance, enabling the camera to capture high-precision images of the calibration object at a position that matches the camera's optimal depth of field position, thereby improving the camera's shooting effect. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating the first embodiment of the camera calibration method based on deep learning in this application; Figure 2 This is a schematic diagram of coordinate matching in the camera calibration method based on deep learning in this application; Figure 3 This is a schematic diagram of the neural network architecture in the camera calibration method based on deep learning in this application; Figure 4 This application presents a schematic diagram of a scene for predicting the location of a calibration point in a camera calibration method based on deep learning. Figure 5 A flowchart illustrating the camera calibration method based on deep learning in this application; Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the camera calibration method based on deep learning in the embodiments of this application.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0023] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions, such as a camera calibration device based on deep learning. The following description uses an electronic device as an example to illustrate this embodiment and the subsequent embodiments.

[0024] Optionally, when performing camera calibration and determining the distance from the calibration object (such as the product under test) to the camera, a sharpness / identification code recognition algorithm can be used. However, this method has certain drawbacks, such as poor processing performance on the inclined surface of the product under test, and the repeatability accuracy being greatly affected by environmental factors such as lighting. Furthermore, identification codes must be affixed for each calibration, making camera calibration impossible in confined spaces or scenarios where it is inconvenient to affix identification codes. Therefore, to circumvent these drawbacks, this application also provides a deep learning-based camera calibration method. This method can directly extract calibration points on the product under test using AI based on its features, and match them with calibration points in the digital model (i.e., the 3D model). Based on the matching relationship and perspective principles, the precise Z-axis distance from the camera to the calibration points is calculated, which is the distance from the camera to the product under test. This allows for accurate determination of the distance between the camera and the calibration object, enabling adjustment of the camera position and ultimately improving the camera's shooting performance.

[0025] Based on this, embodiments of this application provide a camera calibration method based on deep learning, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the camera calibration method based on deep learning in this application.

[0026] In this embodiment, the camera calibration method based on deep learning includes steps S10 to S50.

[0027] Step S10: Determine the 3D model corresponding to the product to be tested that has at least one calibration point marked, and determine the first coordinates corresponding to at least one calibration point in the 3D model; Optionally, calibration points can be pre-selected locations on the product under test and / or 3D model that have identifiable features (such as corners, center of holes, texture features, etc.).

[0028] Optionally, at least two calibration points are located on the same plane of the product under test. The number of calibration points can be multiple, for example, no fewer than four. Each calibration point contains surface-defined geometric / texture features used for identification.

[0029] Optionally, the product to be tested can be a product for which camera calibration is to be performed, such as wearable devices, TWS earphones, or ceramic parts.

[0030] Optionally, the 3D model of the product under test can be a virtual geometric model corresponding to the product under test, or a digital model, or simply a digital model, which can include the geometric shape and size information of the product under test.

[0031] Optionally, the first coordinate can be the coordinate of the calibration point in the product coordinate system, and each calibration point can correspond to a first coordinate.

[0032] Optionally, the product coordinate system can be a coordinate system constructed using any point in the 3D model of the product to be tested.

[0033] Optionally, a digital model (i.e., a 3D model) corresponding to the product under test can be pre-set, and the coordinates of the calibration points in the product coordinate system can be obtained and used as the primary coordinates. For example, the primary coordinates of four calibration points in the product coordinate system can be selected.

[0034] Step S20: Collect product images of the product to be tested, and determine the coordinates of the first image pixels in the product image corresponding to at least one calibration point based on the pre-trained deep learning model. Optionally, the product image can be a product image taken of the product under test, or it can be an image obtained by taking a picture of a product plane with calibration points.

[0035] Optionally, the pre-trained deep learning model can be a pre-set and trained deep learning model, and can be a neural network for calibration point detection. The neural network can include an encoder and a decoder. The encoder is used to extract features from the product image, and the decoder is used to predict the position information of the calibration point based on the extracted features to determine the pixel coordinates of the calibration point in the product image, and use them as the first image pixel coordinates.

[0036] Optionally, the decoder can use split convolution for prediction to reduce the computational load of the model while maintaining model accuracy, thus ensuring the real-time availability of the obtained first image pixel coordinates. Furthermore, it can combine the Gaussian distribution of the predicted points to accurately extract the sub-pixel coordinates of the calibration points, achieving high-precision extraction of the calibration points.

[0037] Optionally, the pre-trained deep learning model can be an attention-based model, such as an attention-based neural network including an encoder and a decoder, to increase model complexity, perform deep fitting of the calibration points of the product under test, and improve the stability of calibration point extraction.

[0038] Optionally, the correspondence between the extracted points and the digital model annotation points can be determined based on the calibration points marked in the model output, that is, the correspondence between the first coordinates and the first image pixel coordinates can be determined, and then the first coordinates and the first image pixel coordinates belonging to the same calibration point can be determined.

[0039] Optionally, product images of the product under test can be captured by a camera, which can be positioned directly above the product under test, such as above it in the z-axis direction.

[0040] Alternatively, the product image can be input into a pre-trained deep learning model, such as a pre-trained neural network. This neural network can be processed using separable convolution to output a center heatmap (hereinafter referred to as the second center heatmap) and an offset heatmap (hereinafter referred to as the second offset heatmap).

[0041] Optionally, each second center heatmap represents a predicted calibration point, and each second offset heatmap represents the offset value between the predicted calibration point and the actual calibration point.

[0042] Optionally, the second center heatmap can be processed by Gaussian distribution to accurately extract the pixel coordinates of the corresponding predicted calibration point in the product image. Then, based on the second offset heatmap corresponding to the second center heatmap, the pixel coordinates can be updated (for example, by adding or subtracting the offset value corresponding to the second offset heatmap based on the pixel coordinates) to obtain the pixel coordinates of the first image.

[0043] Optionally, using a deep learning model as an example of a neural network model, since each channel in the output layer of the neural network model corresponds to a fixed product key point (i.e., a calibration point), it is necessary to determine the spatial coordinates (first coordinates) of the corresponding product key point in each layer, and establish the coordinate correspondence between the first image pixel coordinates predicted by the neural network model and the spatial coordinates. For example, as... Figure 2 As shown, point-to-point matching and coordinate association can be performed between real images (such as product images) and calibration object models (such as 3D models of the product to be tested). For example, if the first image pixel coordinates in the real image are P0(u5, v5), then the corresponding first coordinates are P0(X5, Y5).

[0044] Step S30: Determine the homography matrix representing the coordinate transformation relationship based on the first coordinates belonging to the same calibration point and the first image pixel coordinates; Optionally, the homography matrix represents the coordinate transformation relationship between the product coordinate system corresponding to the first coordinate, the image coordinate system corresponding to the first image pixel coordinate, and the camera coordinate system of the camera.

[0045] Optionally, the homography matrix can characterize the transformation from the image coordinate system to the camera coordinate system and then to the product coordinate system. The transformation between the product and camera coordinate systems can be decomposed from the homography matrix based on the camera intrinsic parameters.

[0046] Optionally, the camera coordinate system can be a camera coordinate system with the camera's optical center as the origin, and it can be a three-dimensional coordinate system. The product coordinate system can also be a three-dimensional coordinate system.

[0047] Optionally, the image coordinate system can be a UV coordinate system with the upper left corner of the product image as the origin, where u is the horizontal axis and v is the vertical axis, and it can be a two-dimensional coordinate system.

[0048] Optionally, for the same calibration point of the product under test, the first coordinate of the calibration point in the product coordinate system and the first image pixel coordinate in the product image can be determined, and a pairing correspondence between the two can be established to obtain the first coordinate and the first image pixel coordinate belonging to the same calibration point, and then the coordinate transformation relationship between the product coordinate system and the image coordinate system can be determined based on this.

[0049] Optionally, the coordinate transformation relationship between the world coordinate system and the product coordinate system can be determined based on the position of the product under test in the world coordinate system, and the coordinate transformation relationship between the world coordinate system and the camera coordinate system can be determined based on the position of the camera in the world coordinate system. Then, the coordinate transformation relationship between the product coordinate system, the image coordinate system, and the camera coordinate system can be determined based on the world coordinate system, thereby obtaining the homography matrix.

[0050] For example, if there are n (n is a positive integer, such as 4) calibration points in the product coordinate system, and the first coordinate of each calibration point in the product coordinate system is... The coordinates of the first image pixel in the image coordinate system are Where i is an integer from 1 to n; T is the transpose sign; These are the coordinates of calibration point i in the X, Y, and Z directions of the product coordinate system, respectively. These are the coordinates of calibration point i in the u-axis and v-axis directions in the image coordinate system, respectively.

[0051] If n calibration points are located on the same plane (i.e.) Then, for any calibration point on this plane, its projection relationship can be determined as shown in Formula 1 below. Formula 1; Where K is the camera intrinsic parameter matrix; R is the rotation matrix; and t is the translation vector; is the scale factor; p is the pixel coordinate of the first image; P is the first coordinate.

[0052] And because Therefore, Formula 1 can be simplified to Formula 2.

[0053] Formula 2; Where u and v are the x and y coordinates of the first image pixel coordinates, respectively; X and Y are the x and y coordinates of the first coordinate, respectively. and These are elements at different positions in the rotation matrix R, such as the first and second columns of the rotation matrix R.

[0054] Based on Formula 2, the corresponding homography matrix H can be obtained as shown in Formula 3 below.

[0055] Formula 3; in, , and These are elements at different positions in the homography matrix H.

[0056] Step S40: Determine the Euclidean distance from at least one calibration point to the camera based on the homography matrix, and determine the distance from the camera to the product under test based on the Euclidean distance. Optionally, the distance can be the Z-axis distance between the camera and the product under test.

[0057] Optionally, for any calibration point, the homography matrix can be decomposed by combining the first coordinates of the calibration point and the first image pixel coordinates to determine the Euclidean distance from at least one calibration point to the camera's optical center. This Euclidean distance can then be used as the distance from the camera to the product under test. Optionally, the average value of the Euclidean distances corresponding to all calibration points can be calculated, and the calculated average value can be used as the distance. Alternatively, a random selection can be made from the Euclidean distances corresponding to all calibration points.

[0058] Optionally, step S40, which involves determining the Euclidean distance from at least one calibration point to the camera's optical center based on the homography matrix, includes steps a10-a40.

[0059] Step a10: Determine the camera's rotation matrix and translation vector based on the homography matrix; Step a20: For any calibration point, determine the second coordinate of the calibration point in the preset world coordinate system; Step a30: Determine the transformation relationship between the world coordinate system and the camera coordinate system. Using the transformation relationship, based on the rotation matrix, translation vector, and second coordinate, determine the third coordinate of the calibration point in the camera coordinate system. Step a40: Calculate the Euclidean distance based on the third coordinate and the origin of the camera coordinate system within the camera coordinate system to obtain the Euclidean distance from the calibration point to the camera, where the origin of the camera coordinate system is the camera optical center.

[0060] Alternatively, the homography matrix can be solved to obtain the distance.

[0061] Optionally, for the i-th calibration point, according to Formulas 1-3 above, Formula 4 can be obtained as follows.

[0062] Formula 4; Eliminate the scale factor corresponding to the i-th calibration point from Formula 4. This leads to formulas 5 and 6.

[0063] Formula 5; Formula 6; in, , , , , , , , , These are elements at different positions in the homography matrix.

[0064] Rearranging Equations 5 and 6 into homogeneous form yields Equation 7.

[0065] Formula 7; For n calibration points, they can be constructed according to Formula 7. Given a matrix A, solve for H by decomposing matrix A using SVD.

[0066] Alternatively, the homography matrix H can be decomposed into a rotation matrix R and a translation vector t.

[0067] Optionally, it is known Then, by decomposing the homography matrix, we can obtain the following formula 8-10.

[0068] Formula 8; Formula 9; Formula 10; Alternatively, the scaling factor can be determined based on the orthogonality of the rotation matrix R. The orthogonality of the rotation matrix R can be expressed as: , .

[0069] Alternatively, the scaling factor can be calculated according to the following formula 11.

[0070] Formula 11; Alternatively, the third column of the rotation matrix can be calculated according to Formula 12 below; thus obtaining the complete rotation matrix R= .

[0071] Formula 12; Optionally, due to the influence of noise, the rotation matrix R needs to be orthogonalized. SVD can be used for orthogonalization, as shown in Formula 13 below.

[0072] Formula 13; Where U is the left singular vector matrix; V is the right singular vector matrix.

[0073] Optionally, the transformation from world coordinates in the world coordinate system to camera coordinates in the camera coordinate system can be performed using Equation 14. In other words, the transformation relationship between the world coordinate system and the camera coordinate system can be shown in Equation 14.

[0074] Formula 14; in, These are the coordinates in the camera coordinate system, such as the coordinates of the calibration point in the camera coordinate system. These are coordinates in the world coordinate system, such as the coordinates of the calibration point in the world coordinate system.

[0075] For calibration points on the same plane ( Formula 15 can be obtained from Formula 14.

[0076] Formula 15; in, These are the coordinates in the camera coordinate system. X and Y can be the horizontal and vertical coordinates in the world coordinate system.

[0077] Expanding formula 15, we get formulas 16-18.

[0078] Formula 16; Formula 17; Formula 18; in, , , , , , These are elements at different positions in the rotation matrix. This is the translation vector along the horizontal axis in the camera coordinate system. This is the translation vector along the vertical axis in the camera coordinate system. This is the translation vector along the Z-axis in the camera coordinate system.

[0079] Optionally, for any calibration point, the second coordinate of the calibration point in the world coordinate system can be determined, and the second coordinate, rotation matrix and translation vector can be input into Formulas 16-18 for calculation to obtain the horizontal axis coordinate, vertical axis coordinate and Z-axis coordinate of the calibration point in the camera coordinate system, that is, the third coordinate. The Euclidean distance between the third coordinate and the camera optical center can be calculated to obtain the corresponding Euclidean distance, which can be used as the corresponding distance.

[0080] Optionally, it can be This refers to the distance between the camera and the product under test along the Z-axis, such as the height distance.

[0081] In this embodiment, the rotation matrix and translation vector of the camera are determined based on the homography matrix, and the second coordinate of the calibration point in the world coordinate system is determined. Then, the third coordinate of the calibration point in the camera coordinate system is determined by combining the transformation relationship between the world coordinate system and the camera coordinate system. Based on the third coordinate and the Euclidean distance of the camera optical center, the Euclidean distance from the calibration point to the camera is determined. Thus, the calibration point can be transformed to the camera coordinate system to accurately determine the Euclidean distance between the calibration point of the product under test and the camera.

[0082] In step S50, in response to the mismatch between the distance and the camera's optimal depth-of-field position, the camera's position parameters are adjusted, and the acquisition of product images of the product under test is re-executed until the latest obtained distance matches the camera's optimal depth-of-field position.

[0083] Alternatively, the optimal depth-of-field position for the camera can be the focus position where the camera image is clearest and the calibration point extraction accuracy is highest.

[0084] Optionally, if a certain distance, such as the height distance, is not at the camera's optimal depth-of-field position, or if the absolute value of the distance difference between the two is greater than or equal to a preset error threshold (e.g., 0.1 nm), then it can be determined that the distance does not match the camera's optimal depth-of-field position. In this case, the camera's position parameters, such as the height position, can be adjusted. The product image is then re-acquired through the camera, and step S20 is executed until the final distance matches the camera's optimal depth-of-field position, i.e., the final distance is at the camera's optimal depth-of-field position, or the absolute value of the difference between the two is less than the preset error threshold. At this point, the final distance can be used as the calibrated distance, such as the calibrated Z-axis distance.

[0085] Optionally, the optimal calibration point height (i.e., the camera's height position) can be calculated iteratively to extract the calibration point at the optimal depth of field, thus ensuring the camera's imaging quality and the accuracy of calibration point extraction.

[0086] Optionally, calibration points to be measured can be selected in the 3D digital model of the product, and the location and type information (calibration point type is numerical or image type) of the calibration points can be pre-labeled to obtain the first coordinates of these calibration points in the product coordinate system. Alternatively, a neural network for calibration point detection can be set up, and the model parameters of this neural network can be trained based on the product's image data. The trained neural network (i.e., a pre-trained deep learning model) can be used to detect the calibration point information and obtain the corresponding first image pixel coordinates, which can be 2D point coordinates. By utilizing the correspondence between 3D and 2D points, the homography matrix H from the camera to the product coordinate system is solved. Through intrinsic parameters and the homography matrix H, the rotation matrix R and translation vector t are decomposed to transform the first coordinate corresponding to the product coordinate system subscript point to the camera coordinate system. The Euclidean distance from this point to the camera optical center is calculated and used as the distance from the camera to the product under test. The calculation is iterative. When this distance is not at the camera's optimal depth of field position, the camera is moved to the optimal depth of field position based on this distance. Then, the camera re-captures the image and recalculates the distance until the error between the distance and the camera's optimal depth of field position is less than a preset error threshold (e.g., 0.1nm).

[0087] Optionally, to ensure measurement accuracy, an iterative approach can be adopted. The optimal depth of field for the camera is set to the standard height. The difference between the current height and the standard height is calculated, and then the camera is moved to that position. This process is repeated until the maximum number of iterations is reached or the difference between the current height and the standard height is less than a threshold. This ensures that the 2D camera and the calibration board remain at the same height during machine calibration data acquisition.

[0088] In this embodiment, by determining the first coordinates of at least one calibration point in the 3D model corresponding to the product under test with at least one calibration point, and by determining the first image pixel coordinates of the calibration point in the product image of the product under test based on a pre-trained deep learning model, pairing the spatial coordinates and the first image pixel coordinates is performed to determine the first coordinates and the first image pixel coordinates belonging to the same calibration point. This determines the homography matrix representing the coordinate transformation relationship, and the Euclidean distance from the calibration point to the camera is determined based on the homography matrix. This determines the distance from the camera to the product under test. When this distance does not match the optimal depth-of-field position of the camera, the position parameters of the camera are adjusted, and the step of acquiring the product image by the camera is iterated again until the latest distance matches the optimal depth-of-field position of the camera. This distance is then used as the final determined distance between the camera and the calibration object (i.e., the product under test). This allows the calibration distance of the camera to match the optimal depth-of-field position of the camera, thus accurately determining the distance between the camera and the calibration object. The camera position can be adjusted based on this distance, enabling the camera to capture a high-precision image of the calibration object at a position that matches the optimal depth-of-field position of the camera, thereby improving the camera's shooting effect.

[0089] Based on the first embodiment of this application, a second embodiment of this application is proposed. In the second embodiment of this application, the same or similar contents as those in the above embodiment can be referred to the above description, and will not be repeated hereafter.

[0090] Optionally, the deep learning-based camera calibration method includes steps b10-b20.

[0091] Step b10: Input the preset training product image into the preset neural network model, and determine the second image pixel coordinates corresponding to the first prediction calibration point based on the output of the neural network model; Step b20: Determine the value of the first loss function based on the preset focus loss function and the pixel coordinates of the second image, and update the model parameters of the neural network model in reverse according to the value of the first loss function until the preset training cutoff condition is met, so as to obtain the pre-trained neural network model and use the pre-trained neural network model as the pre-trained deep learning model.

[0092] Optionally, a training dataset can be constructed that includes multiple training product images and the actual image pixel coordinates corresponding to the calibration points in the training product images.

[0093] Optionally, the training product image can be input into a pre-set neural network model (i.e., the neural network in the first embodiment) for model training. Based on the output of the neural network model, the image pixel coordinates corresponding to the first prediction calibration point in the training product image are determined and used as the second image pixel coordinates.

[0094] Optionally, the actual image pixel coordinates corresponding to the first prediction calibration point in the training dataset and the second image pixel coordinates can be checked and input into a pre-set focus loss function to calculate the loss function value, thus obtaining the first loss function value. If the first loss function value is greater than a preset loss function threshold, the model parameters of the neural network model are updated, and the neural network model is retrained until the preset training cutoff condition is met (such as reaching a certain number of training iterations, model convergence, or the first loss function value being less than the preset loss function threshold, etc.), thus obtaining the pre-trained neural network model, and using the pre-trained neural network model as the pre-trained deep learning model.

[0095] Optionally, the focus loss function is as shown in Formula 19 below.

[0096] Formula 19; in, y is the loss function value of the focus loss function; m is the number of calibration points in the image (such as the training product image); y is the true value (such as the pixel coordinates of the actual image). These are the predicted values ​​from the neural network model (e.g., the pixel coordinates of the second image). and These are two different hyperparameters, which can be set to 0.5 and 2 respectively.

[0097] In this embodiment, a pre-trained neural network model is trained using training product images combined with a focus loss function to obtain a pre-trained neural network model, which is then used as a pre-trained deep learning model, thereby ensuring the effectiveness of the obtained pre-trained deep learning model.

[0098] Optionally, step b10, which involves determining the second image pixel coordinates corresponding to the first prediction calibration point based on the output of the neural network model, includes steps c10-c30.

[0099] Step c10: Based on the encoder of the neural network model, the training product image is mapped to a high-dimensional feature space to obtain a first feature map. Multiple convolutional layers are used to perform convolution processing on the first feature map respectively, and the convolution results of each convolutional layer are fused to obtain a fused convolutional feature map. Step c20: Based on the decoder of the neural network model, at least two parallel deconvolution layers are used to deconvolve the fused convolutional feature maps to obtain the center heatmap and the offset heatmap. Step c30: Determine the pixel coordinates of the second image based on the heatmap obtained from the center heatmap and the offset.

[0100] Optionally, the neural network can be a fully convolutional neural network, including an encoder that can be a lightweight feature extraction and reconstruction encoder and a decoder that can be a key point prediction decoder, which can effectively improve the computational efficiency of calibration point detection.

[0101] Optionally, after the training product image is input into the neural network, the encoder of the neural network encodes the training product image. It can first map the training product image from a low-dimensional image space to a high-dimensional feature space to obtain a first feature map. Then, multiple convolutional layers (e.g., 4 convolutional layers) are used to convolve the first feature map respectively, and the convolution results of each convolutional layer are fused to obtain a fused convolutional feature map. This is then input into the decoding end of the neural network. The decoding end is also equipped with convolutional layers (i.e., deconvolutional layers) with the opposite function to those in the encoding end. It can also adopt a separate convolution method, using two parallel deconvolutional layers (e.g., 4 convolutional layers) to perform deconvolution processing respectively, to obtain two heatmaps, namely a center heatmap and an offset heatmap. Then, based on these two heatmaps, the image pixel coordinates of the predicted calibration point in the training product image are determined, i.e., the second image pixel coordinates.

[0102] For example, such as Figure 3 As shown, a 1024x1024x3 image (such as a training product image) is input into a neural network. In the feature extraction and reconstruction encoder (hereinafter referred to as the encoder), a ResNet-50 backbone network can be used for feature extraction to map the image space to a high-dimensional feature space, that is, to map the training product image to a high-dimensional feature space, thus obtaining the first feature map. Depthwise separable convolution operations can also be introduced to lightweight the ResNet-50 backbone network. The improved ResNet-50 mainly contains four residual convolutional blocks (e.g., 1×1 Conv), outputting four multi-scale intermediate feature maps with sizes of 32×32×2048, 64×64×256, 128×128×128, and 256×256×64, respectively denoted as _____. , , and Since features at different scales contribute differently to the final output, a weighted FPN algorithm is used to weight and fuse the obtained multi-scale feature maps. During weighted fusion, convolutional fusion can be performed using 2×Upsample and 3×3 Conv convolutional layers, and the feature map resolution is restored using three deconvolutional layers. Furthermore, during the restoration process, a 3×3 Maxpooling layer can be incorporated to obtain the desired feature map resolution. , , and The encoder output is a feature map of size 256×256×64. Previous FPN-based feature fusion methods treated features at different scales consistently. Weighted FPN achieves simple and fast bidirectional cross-scale feature fusion by introducing weight and contextual information into features at different scales. In this embodiment, "f" corresponds to the "f" in the figure. ".

[0103] by Taking the FPN feature fusion process at the level as an example, the calculation formulas are shown in Formulas 20 and 21 below.

[0104] Formula 20; Formula 21; in, These are intermediate features of the first layer; and These are two different multi-scale intermediate feature maps that have undergone convolution processing; , , , , For different learnable weight parameters.

[0105] Optionally, features at all other levels are constructed in a similar manner, with normalization and activation function operations performed after each convolutional layer. During feature extraction, the resolution of the feature map decreases continuously due to multiple downsampling operations performed by the convolutional layers, thus gradually ignoring the information of the calibration points. To address this issue, deconvolution operations are used to reconstruct the resolution of the feature map, and a feature pyramid structure is introduced to fuse multi-scale feature information, thereby preserving and enhancing the features of the calibration points.

[0106] Optionally, the decoder consists of two parallel convolutional layers (e.g., 3×3 Conv), outputting two heatmaps (a 256×256×4 center and a 256×256×2 offset), namely a center heatmap and an offset heatmap. The center heatmap is 128×128×C in size, where C is the number of channels, and each channel represents a calibration point. According to the perspective projection principle, the required calibration point information can be satisfied when there are four or more channels with calibration points. To comprehensively consider the stability and timeliness of the algorithm, the number of output layer channels is set to 4. The coordinates of the center point reflect the position of the calibration point, and the pixel values ​​reflect the confidence level of the prediction result. The offset heatmap is used to predict the offset value of the calibration point to compensate for the error caused by the feature map downsampling operation.

[0107] In this embodiment, when training the neural network model, the encoder and decoder of the neural network model can be combined for model training. Separate convolution is set in the decoder so that the decoder can output a center heatmap and an offset heatmap. The second image pixel coordinates of the predicted calibration point are determined based on these two heatmaps, thereby ensuring the effectiveness of the predicted second image pixel coordinates and improving the training effect of the neural network model.

[0108] Optionally, step c30, which involves determining the pixel coordinates of the second image based on the center heatmap and the offset, includes steps d10-d20.

[0109] Step d10: For each center heatmap, perform Gaussian distribution detection on the center heatmap, and determine the coordinates of the third image pixel in the training product image corresponding to the first prediction calibration point based on the Gaussian distribution detection results. Optionally, the neural network output can have multiple center heatmaps, with each channel corresponding to one center heatmap. Each center heatmap can also correspond to an offset heatmap. The same operation can be performed on each center heatmap; the following example illustrates the processing of only one center heatmap.

[0110] Optionally, in order to more reliably predict the target (i.e., the second image pixel coordinates of the predicted calibration point in the training product image), the predicted calibration point position can be represented by a Gaussian distribution. That is, the Gaussian distribution detection of the central heat map can be used to determine the corresponding image pixel coordinates in the training product image, and these coordinates can be used as the third image pixel coordinates. If the offset value of the offset heat map corresponding to the central heat map is 0, the third image pixel coordinates can be directly used as the second image pixel coordinates.

[0111] Optionally, step d10, which involves performing Gaussian distribution detection on the central heatmap and determining the coordinates of the third image pixel in the training product image corresponding to the first prediction calibration point based on the Gaussian distribution detection result, includes steps e10-e40.

[0112] Step e10: Determine the preset Gaussian distribution detection conditions, wherein the Gaussian distribution detection conditions include: when the result generated by Gaussian distribution processing of the center heatmap contains a non-zero real matrix, determine the center of the circle in the key feature region corresponding to the real matrix in the training product image as the third image pixel coordinates corresponding to the first prediction calibration point. Step e20: Perform Gaussian distribution detection on the central heat map according to the Gaussian distribution detection conditions; Step e30: In response to the Gaussian distribution detection result including the existence of a non-zero real number matrix, the first real number in the real number matrix that is greater than a preset real number threshold is determined; Step e40: The pixel region where each first real number in the training product image is located is taken as the calibration point feature region, and the pixel coordinates of the third image are determined based on the calibration point feature region.

[0113] Optionally, when performing Gaussian distribution detection, Gaussian distribution detection conditions can be assumed in advance. For example, the region centered on the pixel coordinates of the predicted calibration point and with a radius of a preset number of pixels (e.g., 10 pixels) can be used as the key feature region. A real-valued matrix of the corresponding calibration point features is generated using a Gaussian distribution. After Gaussian distribution processing, the real values ​​outside this key feature region are zero. Within this key feature region, pixels closer to the center of the circle have a higher Gaussian probability, i.e., a larger real value. Each central heatmap contains feature information for only one calibration point. When a calibration point is not visible in the central heatmap, all values ​​in the key feature region of the heatmap corresponding to that calibration point are 0. Each pixel value in the central heatmap is created by a two-dimensional Gaussian distribution. Furthermore, the center position is also the peak position of the Gaussian circle. The closer the pixel position is to the center of the circle, the higher the probability value; pixels outside the circle have a value of 0.

[0114] Optionally, a threshold of 0.5 can be set as the predicted response value for the calibration point (i.e., a preset real number threshold). Regions with values ​​greater than 0.5 are considered the feature regions where the calibration point is located; regions with values ​​less than 0.5 are considered to lack a calibration point. In other words, if a Gaussian distribution is applied to the central heatmap, resulting in a matrix of non-zero real numbers, the pixel regions corresponding to this matrix can be key feature regions. If the real number matrix does not contain a first real number greater than the preset real number threshold, then the predicted calibration point corresponding to the central heatmap does not exist. If a first real number exists, the pixel region composed of the pixels corresponding to each first real number can be determined as the feature region containing the calibration point. Within this feature region, the pixel coordinates that may be the predicted calibration point are determined and used as the third pixel coordinates.

[0115] In this embodiment, the coordinates of the third image pixels corresponding to the central heatmap are determined by using a Gaussian distribution on the central heatmap, thereby ensuring the accuracy and effectiveness of the predicted coordinates of the third image pixels of the calibration point.

[0116] Step d20: Based on the offset heatmap corresponding to the center heatmap, update the pixel coordinates of the third image to obtain the pixel coordinates of the second image.

[0117] Optionally, for at least one center heatmap, if the corresponding third pixel coordinates are determined according to steps e10-e40, then the offset heatmap corresponding to that center heatmap can be determined. That is, the center heatmap and offset heatmap belonging to the same channel are determined, and the offset value corresponding to the offset heatmap is determined. If the offset value is 0, then the third image pixel coordinates are used as the second image pixel coordinates. If the offset value is positive, then the offset value is added to the third image pixel coordinates to obtain the second image pixel coordinates. If the offset value is negative, then the offset value is subtracted from the third image pixel coordinates to obtain the second image pixel coordinates. The final obtained second image pixel coordinates are then used as the image pixel coordinates of the final predicted calibration point.

[0118] For example, such as Figure 4 As shown, if there are 4 calibration points in the image of the product (such as a training product image or a product image), such as Figure 4 (a) shows P0, P1, P2, and P3 respectively. The neural network model then predicts the corresponding heatmaps (including a center heatmap and an offset heatmap), as shown below. Figure 4 As shown in (b) above, the positions of these four calibration points in the image are predicted based on the heatmap, for example, as... Figure 4 As shown in (c), the values ​​are P0: 0.78; P1: 0.73; P2: 0.91; P3: 0.82.

[0119] In this embodiment, Gaussian distribution detection is performed on the center heatmap to predict the third image pixel coordinates of the calibration point. In order to more accurately predict the image pixel coordinates of the calibration point, the offset heatmap is also used for coordinate update processing to obtain the second image pixel coordinates, thereby making the predicted second image pixel coordinates of the calibration point more accurate and effective.

[0120] In addition, to aid in understanding the deep learning-based camera calibration principle in the embodiments of this application, examples are provided below.

[0121] For example, such as Figure 5As shown, when starting camera calibration, calibration points can be selected from the product 3D model and their positions and types can be labeled. This obtains the coordinates in the product coordinate system, meaning the 3D model corresponding to the product with at least one calibration point is determined, and the first coordinates corresponding to at least one calibration point in the 3D model are determined. An encoder-decoder type calibration point detection neural network can be designed, i.e., a deep learning model is constructed and trained to obtain a pre-trained deep learning model. The encoder of the neural network extracts image features, and the decoder predicts the pixel coordinates of the calibration points. This means the pre-trained deep learning model can be used to determine the first image pixel coordinates in the product image corresponding to at least one calibration point. The homography matrix is ​​then solved using the correspondence between the predicted points and the model calibration points. Based on the first coordinates and first image pixel coordinates belonging to the same calibration point, the homography matrix representing the coordinate transformation relationship is determined. Combined with the camera intrinsic parameter matrix, the points in the product coordinate system are transformed to the camera coordinate system, obtaining the Z-axis distance (i.e., the height distance between the camera and the calibration point). If the calibration point height is at the optimal depth of field position of the camera, the calibration process ends. If not, the camera is moved to the optimal depth of field position according to the calculated height, and the image is retaken and the height is recalculated until the calibration point height is at the optimal depth of field position of the camera, thus completing the camera calibration and ending the corresponding calibration process.

[0122] Furthermore, this application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the deep learning-based camera calibration method in Embodiment 1 above.

[0123] The following is for reference. Figure 6 The figure illustrates a structural diagram of an electronic device suitable for implementing embodiments of this application. The electronic devices in the embodiments of this application may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The devices shown in the figure are merely examples and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0124] The electronic device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for device operation. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. While electronic devices with various systems are shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0125] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0126] The electronic device provided in this application employs the deep learning-based camera calibration method described in the above embodiments, which solves the technical problem of how to accurately determine the distance between the camera and the calibration object in order to adjust the camera position and thus improve the camera's shooting effect. Compared with the prior art, the beneficial effects of the electronic device provided in this application are the same as those of the deep learning-based camera calibration method provided in the above embodiments, and other technical features of this electronic device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0127] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0128] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0129] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the deep learning-based camera calibration method in the above embodiments.

[0130] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0131] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0132] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by an electronic device, enable the electronic device to perform the steps in the aforementioned deep learning-based camera calibration method.

[0133] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0134] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0135] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0136] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned deep learning-based camera calibration method. This solves the technical problem of accurately determining the distance between the camera and the calibration object to adjust the camera position, thereby improving the camera's shooting performance. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the deep learning-based camera calibration method provided in the above embodiments, and will not be repeated here.

[0137] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the deep learning-based camera calibration method described above.

[0138] The computer program product provided in this application solves the technical problem of how to accurately determine the distance between the camera and the calibration object in order to adjust the camera position and thus improve the camera's shooting effect. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the deep learning-based camera calibration method provided in the above embodiments, and will not be repeated here.

[0139] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A camera calibration method based on deep learning, characterized in that, The deep learning-based camera calibration method includes the following steps: Determine the three-dimensional model corresponding to the product under test that has at least one calibration point marked, and determine the first coordinates corresponding to the at least one calibration point in the three-dimensional model; Collect a product image of the product to be tested, and determine the coordinates of the first image pixel in the product image corresponding to the at least one calibration point based on a pre-trained deep learning model; Based on the first coordinates and first image pixel coordinates belonging to the same calibration point, a homography matrix representing the coordinate transformation relationship is determined; The Euclidean distance from the at least one calibration point to the camera is determined based on the homography matrix, and the distance from the camera to the product under test is determined based on the Euclidean distance. In response to the mismatch between the distance and the camera's optimal depth-of-field position, the position parameters of the camera are adjusted, and the acquisition of product images of the product under test is re-executed until the latest distance matches the camera's optimal depth-of-field position.

2. The camera calibration method based on deep learning as described in claim 1, characterized in that, The homography matrix represents the coordinate transformation relationship between the product coordinate system corresponding to the first coordinate, the image coordinate system corresponding to the first image pixel coordinate, and the camera coordinate system of the camera.

3. The camera calibration method based on deep learning as described in claim 2, characterized in that, The at least two calibration points are located on the same plane of the product under test. The step of determining the Euclidean distance from the at least one calibration point to the camera based on the homography matrix includes: The rotation matrix and translation vector of the camera are determined based on the homography matrix. For any given calibration point, determine the second coordinate of the calibration point in a preset world coordinate system; Determine the transformation relationship between the world coordinate system and the camera coordinate system, and using the transformation relationship, based on the rotation matrix, the translation vector, and the second coordinate, determine the third coordinate of the calibration point in the camera coordinate system; The Euclidean distance from the calibration point to the camera is calculated based on the third coordinate and the origin of the camera coordinate system within the camera coordinate system, wherein the origin of the camera coordinate system is the optical center of the camera.

4. The camera calibration method based on deep learning as described in claim 1, characterized in that, The deep learning-based camera calibration method also includes: The preset training product image is input into the preset neural network model, and the second image pixel coordinates corresponding to the first prediction calibration point are determined based on the output of the neural network model. The first loss function value is determined based on the preset focus loss function and the pixel coordinates of the second image. The model parameters of the neural network model are updated in reverse based on the first loss function value until the preset training cutoff condition is met, thereby obtaining a pre-trained neural network model. The pre-trained neural network model is then used as a pre-trained deep learning model.

5. The camera calibration method based on deep learning as described in claim 4, characterized in that, The step of determining the second image pixel coordinates corresponding to the first prediction calibration point based on the output of the neural network model includes: Based on the encoder of the neural network model, the training product image is mapped to a high-dimensional feature space to obtain a first feature map. Multiple convolutional layers are used to perform convolution processing on the first feature map respectively, and the convolution results of each convolutional layer are fused to obtain a fused convolutional feature map. The decoder of the neural network model uses two parallel deconvolution layers (at least one) to perform deconvolution processing on the fused convolutional feature map to obtain a center heatmap and an offset heatmap. The pixel coordinates of the second image are determined using the heatmap based on the central heatmap and the offset.

6. The camera calibration method based on deep learning as described in claim 5, characterized in that, The step of determining the pixel coordinates of the second image based on the heatmap obtained from the center heatmap and the offset includes: For each of the central heatmaps, Gaussian distribution detection is performed on the central heatmap, and the coordinates of the third image pixel in the training product image corresponding to the first prediction calibration point are determined based on the Gaussian distribution detection result. Based on the offset heatmap corresponding to the central heatmap, the pixel coordinates of the third image are updated to obtain the pixel coordinates of the second image.

7. The camera calibration method based on deep learning as described in claim 6, characterized in that, The step of performing Gaussian distribution detection on the central heatmap and determining the coordinates of the third image pixel in the training product image corresponding to the first prediction calibration point based on the Gaussian distribution detection result includes: A preset Gaussian distribution detection condition is determined, wherein the Gaussian distribution detection condition includes: when the result generated by Gaussian distribution processing of the center heatmap contains a non-zero real number matrix, the center of the circle in the key feature region corresponding to the real number matrix in the training product image is determined as the third image pixel coordinate corresponding to the first prediction calibration point; Gaussian distribution detection is performed on the central heat map according to the Gaussian distribution detection conditions; If the Gaussian distribution detection result includes the existence of a non-zero real number matrix, then the first real number in the real number matrix that is greater than a preset real number threshold is determined; The pixel region where each of the first real numbers in the training product image is located is taken as the calibration point feature region, and the pixel coordinates of the third image are determined based on the calibration point feature region.

8. An electronic device, characterized in that, The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the deep learning-based camera calibration method as described in any one of claims 1 to 7.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the camera calibration method based on deep learning as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the deep learning-based camera calibration method as described in any one of claims 1 to 7.

Images