A camera parameter calibration method, device and equipment

Abstract

The application provides a camera parameter calibration method, device and equipment. The method comprises the following steps: determining pixel coordinates of a calibration point in an image coordinate system based on a two-dimensional image; determining physical coordinates of the calibration point in a system coordinate system based on a position relationship between an optical identifier and the calibration point; generating a plurality of coordinate point pairs corresponding to a plurality of calibration points; configuring an initial focal length value for a camera focal length; determining a target parameter value corresponding to a to-be-calibrated parameter of a two-dimensional camera based on the initial focal length value and the plurality of coordinate point pairs; determining a target focal length value corresponding to the camera focal length of the two-dimensional camera based on the target parameter value, a vertical height between the two-dimensional camera and a measurement object, and the initial focal length value; and calibrating the target focal length value and the target parameter value for the two-dimensional camera. Through the technical solution of the application, the calibration operation can be simplified, the calibration time can be reduced, and the labor cost for calibration can be saved.

Description

Technical Field

[0001] This application relates to the field of vision technology, and in particular to a method, apparatus and device for calibrating camera parameters. Background Technology

[0002] In the field of vision-based multidimensional information measurement and recognition, multiple cameras can be used to acquire information about objects in different dimensions. Then, temporal correlation prediction technology is used to bind the recognition information from multiple cameras on the same object to obtain complete information about the object. For example, in the application of package volume measurement and optical symbol recognition in logistics systems, 3D cameras can be used for package volume measurement and classification, multiple small-field-of-view 2D cameras can be used for optical symbol recognition, and large-field-of-view 2D cameras can be used to obtain panoramic images of packages. Then, the 3D positions of multiple cameras are correlated, and temporal correlation prediction technology is combined to obtain the volume value, category information, optical symbol information, and panoramic image of each package. Based on this information, subsequent operations such as piece-rate billing, preventing missed scans, package verification, and classified loading can be performed.

[0003] To correlate the 3D positions of multiple cameras, it is necessary to calibrate the intrinsic and extrinsic parameters of each camera and correlate their 3D positions based on these parameters. However, there is no reasonable calibration method for each camera in the relevant technologies, resulting in problems such as cumbersome calibration operations, long calibration times, and high labor costs. Summary of the Invention

[0004] This application provides a method for calibrating camera parameters, wherein a calibration object is placed on the measurement object, the calibration object including optical identifiers and multiple calibration points, and the method includes:

[0005] A calibration two-dimensional image of the object being measured is acquired using a two-dimensional camera. The pixel coordinates of the calibration point in the image coordinate system are determined based on the calibration two-dimensional image. The physical coordinates of the calibration point in the system coordinate system are determined based on the positional relationship between the optical identifier and the calibration point.

[0006] Generate multiple coordinate point pairs corresponding to multiple calibration points. For each coordinate point pair corresponding to a calibration point, the coordinate point pair includes the pixel coordinates and physical coordinates corresponding to the calibration point.

[0007] Configure an initial focal length value for the camera focal length, and determine the target parameter value corresponding to the calibration parameters of the two-dimensional camera based on the initial focal length value and the multiple coordinate point pairs;

[0008] Based on the target parameter value, the vertical height between the two-dimensional camera and the measurement object, and the initial focal length value, the target focal length value corresponding to the camera focal length of the two-dimensional camera is determined;

[0009] The target focal length value and the target parameter value are calibrated for the two-dimensional camera.

[0010] This application provides a camera parameter calibration device, wherein a calibration object is placed on the measurement object, the calibration object including an optical identifier and a plurality of calibration points, and the device includes:

[0011] The acquisition module is used to acquire a calibration two-dimensional image of the measurement object using a two-dimensional camera, determine the pixel coordinates of the calibration point in the image coordinate system based on the calibration two-dimensional image, and determine the physical coordinates of the calibration point in the system coordinate system based on the positional relationship between the optical identifier and the calibration point.

[0012] The generation module is used to generate multiple coordinate point pairs corresponding to multiple calibration points. For each coordinate point pair corresponding to a calibration point, the coordinate point pair includes the pixel coordinates and physical coordinates corresponding to the calibration point.

[0013] The determination module is used to configure an initial focal length value for the camera focal length, determine the target parameter value corresponding to the calibration parameter of the two-dimensional camera based on the initial focal length value and the multiple coordinate point pairs, and determine the target focal length value corresponding to the camera focal length of the two-dimensional camera based on the target parameter value, the vertical height between the two-dimensional camera and the measurement object, and the initial focal length value.

[0014] The calibration module is used to calibrate the target focal length value and the target parameter value for the two-dimensional camera.

[0015] This application provides an electronic device, including: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; wherein the processor is used to execute the machine-executable instructions to implement the camera parameter calibration method described above in this application.

[0016] As can be seen from the above technical solutions, in this embodiment, camera calibration can be performed simply by laying a calibration object (such as calibration cloth or calibration plate) carrying optical identifiers, effectively reducing the cost of manufacturing and carrying the calibration object. By acquiring a calibration two-dimensional image of the object to be measured using a two-dimensional camera, and combining the vertical height between the two-dimensional camera and the object (i.e., the mounting height of the two-dimensional camera), the intrinsic and extrinsic parameters of the two-dimensional camera can be calculated, and the intrinsic and extrinsic parameters of the two-dimensional camera can be calibrated, greatly improving implementation efficiency and saving labor costs. By associating each camera with the system coordinate system, while decoupling the cameras from each other, the scalability of camera calibration is improved. By configuring an initial focal length value for the camera, and based on the initial focal length value, the target parameter values ​​corresponding to the parameters to be calibrated of the two-dimensional camera are determined based on multiple coordinate point pairs, thereby reducing the amount of calculation, saving computing resources, and improving calibration speed. Based on the above camera calibration method, the calibration operation can be simplified, the calibration time reduced, and the calibration labor costs saved. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments of this application or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings of the embodiments of this application.

[0018] Figure 1 This is a flowchart illustrating a camera parameter calibration method according to one embodiment of this application.

[0019] Figure 2 This is a schematic diagram of an application scenario in one embodiment of this application;

[0020] Figure 3 This is a schematic diagram of the calibration process of a 3D camera in one embodiment of this application;

[0021] Figure 4 This is a schematic diagram of the calibration process of a two-dimensional camera in one embodiment of this application;

[0022] Figure 5 This is a flowchart illustrating a camera parameter calibration method according to one embodiment of this application.

[0023] Figure 6 This is a schematic diagram of the camera parameter calibration device in one embodiment of this application;

[0024] Figure 7 This is a hardware structure diagram of an electronic device according to one embodiment of this application. Detailed Implementation

[0025] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “the,” and “the” as used in this application and claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to any and all possible combinations comprising one or more of the associated listed items.

[0026] It should be understood that although the terms first, second, third, etc., may be used to describe various information in embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" may also be interpreted as "when," "when," or "in response to a determination."

[0027] This application proposes a method for calibrating camera parameters. A calibration object (such as a calibration cloth or calibration plate) is placed on the measurement object. This calibration object may include optical identifiers and multiple calibration points. See [link to relevant documentation]. Figure 1 The diagram shown is a flowchart of the method, which may include:

[0028] Step 101: Acquire a calibration two-dimensional image of the object to be measured using a two-dimensional camera, determine the pixel coordinates of the calibration point in the image coordinate system based on the calibration two-dimensional image, and determine the physical coordinates of the calibration point in the system coordinate system based on the positional relationship between the optical identifier and the calibration point.

[0029] In one possible implementation, determining the physical coordinates of the calibration point in the system coordinate system based on the positional relationship between the optical identifier and the calibration point may include, but is not limited to, resolving the physical coordinates of the target position of the optical identifier in the system coordinate system from the optical identifier; wherein the target position of the optical identifier may include, but is not limited to, the upper left corner, upper right corner, lower left corner, lower right corner, or center position of the optical identifier. Then, based on the physical coordinates of the target position of the optical identifier in the system coordinate system and the positional relationship between the target position of the optical identifier and the calibration point, the physical coordinates of the calibration point in the system coordinate system can be determined.

[0030] Step 102: Generate multiple coordinate point pairs corresponding to multiple calibration points. For each coordinate point pair corresponding to a calibration point, the coordinate point pair may include the pixel coordinates and physical coordinates corresponding to the calibration point.

[0031] Step 103: Configure an initial focal length value for the camera, and determine the target parameter values ​​corresponding to the calibration parameters of the two-dimensional camera based on the initial focal length value and multiple coordinate point pairs.

[0032] For example, the parameters to be calibrated may include, but are not limited to, the camera center, distortion coefficients, rotation parameters, and translation parameters. Based on the initial focal length value and multiple coordinate point pairs, the target parameter values ​​corresponding to the calibrated parameters of the 2D camera are determined. This may include, but is not limited to: configuring an initial center value for the camera center and initial coefficient values ​​for the distortion coefficients; and determining the initial values ​​of the rotation parameters and translation parameters based on the initial focal length value, initial center value, initial coefficient values, and multiple coordinate point pairs. The initial center value, initial coefficient values, initial values ​​of the rotation parameters, and initial values ​​of the translation parameters are then optimized (e.g., through nonlinear optimization) to obtain the target center value corresponding to the camera center, the target coefficient value corresponding to the distortion coefficients, the target rotation parameter value corresponding to the rotation parameters, and the target translation parameter value corresponding to the translation parameters.

[0033] Step 104: Based on the target parameter value corresponding to the parameter to be calibrated, the vertical height between the 2D camera and the measurement object, and the initial focal length value, determine the target focal length value corresponding to the camera focal length of the 2D camera.

[0034] For example, based on the target parameter value corresponding to the parameter to be calibrated, the vertical height between the 2D camera and the measured object, and the initial focal length value, the target focal length value corresponding to the 2D camera's focal length is determined. This can include, but is not limited to, determining a candidate focal length value corresponding to the 2D camera's focal length based on the target parameter value, the vertical height between the 2D camera and the measured object, and the initial focal length value. If the difference between the candidate focal length value and the initial focal length value is less than a preset threshold, the candidate focal length value can be determined as the target focal length value corresponding to the camera's focal length; otherwise, if the difference between the candidate focal length value and the initial focal length value is not less than a preset threshold, the candidate focal length value is determined as the initial focal length value, and the operation of determining the target parameter value corresponding to the parameter to be calibrated of the 2D camera based on the initial focal length value and multiple coordinate point pairs is returned.

[0035] For example, the parameters to be calibrated may include, but are not limited to: camera center, distortion coefficient, rotation parameters, and translation parameters, and the translation parameters include X-axis translation parameters, Y-axis translation parameters, and Z-axis translation parameters; based on this, based on the target parameter values, the vertical height between the 2D camera and the measurement object, and the initial focal length value, the candidate focal length value corresponding to the camera focal length of the 2D camera is determined, which may include, but is not limited to: determining an adjustment factor based on the target parameter values ​​corresponding to the vertical height and the Z-axis translation parameters, and adjusting the initial focal length value based on the adjustment factor to obtain the candidate focal length value corresponding to the camera focal length of the 2D camera.

[0036] Step 105: Calibrate the target focal length and target parameter values ​​for the 2D camera.

[0037] For example, target parameter values ​​may include, but are not limited to, target center values ​​corresponding to the camera center, target coefficient values ​​corresponding to the distortion coefficients, target rotation parameter values ​​corresponding to the rotation parameters, and target translation parameter values ​​corresponding to the translation parameters. Clearly, by calibrating the 2D camera to the target focal length value corresponding to the camera focal length, the target center value corresponding to the camera center, and the target coefficient values ​​corresponding to the distortion coefficients, we are essentially calibrating the 2D camera's intrinsic parameters. Similarly, by calibrating the 2D camera to the target rotation parameter values ​​corresponding to the rotation parameters and the target translation parameter values ​​corresponding to the translation parameters, we are essentially calibrating the 2D camera's extrinsic parameters.

[0038] In one possible implementation, a 3D point cloud for calibration of the measurement object can be acquired using a 3D camera, and the 3D spatial coordinates corresponding to the calibration points can be determined based on the calibration point cloud. The physical coordinates of the calibration point in the system coordinate system are determined based on the positional relationship between the optical identifier and the calibration point. Multiple coordinate point pairs corresponding to multiple calibration points are generated. For each coordinate point pair corresponding to a calibration point, the coordinate point pair can include the 3D spatial coordinates and physical coordinates corresponding to the calibration point. Based on the calibrated intrinsic parameters of the 3D camera and the multiple coordinate point pairs, the target parameter values ​​corresponding to the camera extrinsic parameters can be determined, and the target parameter values ​​of the camera extrinsic parameters are calibrated for the 3D camera; that is, the camera extrinsic parameters are calibrated for the 3D camera.

[0039] In one possible implementation, when a target object is placed on the measurement object and the measurement object is moving at a constant linear speed, a two-dimensional image of the target object can be acquired using a two-dimensional camera, and a three-dimensional point cloud of the target object can be acquired using a three-dimensional camera. The three-dimensional spatial coordinates of the target object can be determined based on the target three-dimensional point cloud, and these three-dimensional spatial coordinates can be converted into the initial physical coordinates of the target object in the system coordinate system based on the camera intrinsic and extrinsic parameters of the three-dimensional camera. Based on the measurement object's motion speed, target duration, and the initial physical coordinates, the target physical coordinates of the target object in the system coordinate system are determined. Then, the target physical coordinates can be converted into target pixel coordinates of the target object in the image coordinate system based on the camera intrinsic and extrinsic parameters of the two-dimensional camera, and the pixel position corresponding to the target pixel coordinates can be located from the target two-dimensional image corresponding to the target duration.

[0040] As can be seen from the above technical solutions, in this embodiment, camera calibration can be performed simply by laying a calibration object (such as calibration cloth or calibration plate) carrying optical identifiers, effectively reducing the cost of manufacturing and carrying the calibration object. By acquiring a calibration two-dimensional image of the object to be measured using a two-dimensional camera, and combining the vertical height between the two-dimensional camera and the object (i.e., the mounting height of the two-dimensional camera), the intrinsic and extrinsic parameters of the two-dimensional camera can be calculated, and the intrinsic and extrinsic parameters of the two-dimensional camera can be calibrated, greatly improving implementation efficiency and saving labor costs. By associating each camera with the system coordinate system, while decoupling the cameras from each other, the scalability of camera calibration is improved. By configuring an initial focal length value for the camera, and based on the initial focal length value, the target parameter values ​​corresponding to the parameters to be calibrated of the two-dimensional camera are determined based on multiple coordinate point pairs, thereby reducing the amount of calculation, saving computing resources, and improving calibration speed. Based on the above camera calibration method, the calibration operation can be simplified, the calibration time reduced, and the calibration labor costs saved.

[0041] The technical solutions described above in the embodiments of this application will be explained below in conjunction with specific application scenarios.

[0042] In the field of vision-based multidimensional information measurement and recognition, multiple cameras can be used to acquire information about objects in different dimensions. Then, temporal correlation prediction technology is used to bind the recognition information from multiple cameras on the same object to obtain complete information about the object. For example, in the application of package volume measurement and optical symbol recognition in logistics systems, 3D cameras can be used for package volume measurement and classification, multiple small-field-of-view 2D cameras can be used for optical symbol recognition, and large-field-of-view 2D cameras can be used to obtain panoramic images of packages. Then, the 3D positions of multiple cameras are correlated, and temporal correlation prediction technology is combined to obtain the volume value, category information, optical symbol information, and panoramic image of each package. Based on this information, subsequent operations such as piece-rate billing, preventing missed scans, package verification, and classified loading can be performed.

[0043] To correlate the 3D positions of multiple cameras, it is necessary to calibrate the intrinsic and extrinsic parameters of each camera and correlate their 3D positions based on these parameters. However, there is no reasonable calibration method for each camera in the relevant technologies, resulting in problems such as cumbersome calibration operations, long calibration times, and high labor costs.

[0044] To address the above findings, this embodiment proposes a multi-camera calibration method based on a single-frame image. This method requires laying a calibration cloth or calibration board with optical identifiers on the measurement object, ensuring the cloth (board) covers the measurement area of ​​all cameras and that each camera can see the calibration grid and optical identifiers. The mounting height of each 2D camera (i.e., the vertical distance between the 2D camera and the measurement object) is measured and stored in the 2D camera data. With the calibration cloth (board) stationary, each 2D camera acquires a single frame of calibration 2D image. Based on the calibration 2D image and the camera's mounting height, the intrinsic and extrinsic parameters of the 2D cameras can be calibrated. In this method, the calibration object is a calibration cloth (board) that can be printed on-site, and its laying requires manual intervention, i.e., laying the calibration cloth (board) on the measurement object. Based on this, the two-dimensional camera only needs to know the installation height and acquire a single frame of calibration two-dimensional image of the object being measured to complete the internal and external parameter calibration of the two-dimensional camera. This effectively reduces the cost of manufacturing and carrying the calibration object, greatly improves the implementation efficiency, simplifies the calibration operation of camera parameters, reduces calibration time, and saves calibration manpower costs.

[0045] In one possible implementation, see Figure 2 The diagram shown is an application scenario illustration of an embodiment of this application. A calibration cloth or calibration plate is laid on the measurement object. The following description will use the calibration cloth as an example.

[0046] Because it is necessary to perform temporal correlation and prediction of objects on the measurement object and to correlate information from multiple cameras, the measurement object can be an object moving in a straight line at a constant speed, such as a logistics conveyor belt moving in a straight line at a constant speed, a cross belt moving in a straight line at a constant speed, a robot moving in a straight line at a constant speed, or a vehicle moving in a straight line at a constant speed. There is no restriction on the type of measurement object, and the measurement object can also be called a measurement system.

[0047] A calibration cloth can be laid on the object being measured. The length and width of the calibration cloth must cover the entire measurement area. The calibration cloth can be printed in sections and spliced ​​together. There are no restrictions on the material of the calibration cloth; it simply needs to be laid flat and statically across the entire measurement area. For example, if the object being measured is a conveyor belt or cross-belt, the calibration cloth can be laid on top of the conveyor belt or cross-belt. If the object being measured is a robot or vehicle, the calibration cloth can be laid on top of the robot or vehicle, along its movement path, so that the calibration cloth is laid flat within the robot's or vehicle's movement range.

[0048] The calibration cloth can be a calibration cloth with calibration grids, which can be a checkerboard grid or a dotted grid; there are no restrictions on the type of calibration grid. Figure 2In this example, we can use a chessboard as an example. Multiple calibration points on the calibration cloth can be determined based on the calibration grid. For instance, for a chessboard calibration grid, the intersection of the black and white chessboard grids can be used as the calibration point. For a dot calibration grid, the center of the circle can be used as the calibration point. Of course, the above are just examples; as long as the calibration points can be determined from the calibration cloth, it is acceptable.

[0049] The calibration cloth can be a calibration cloth with optical identifiers. These optical identifiers can be QR codes (describing a unique location), numbers (describing a unique location), or other information capable of describing a unique location. There are no restrictions on the type of optical identifier, as long as it can describe a unique location. Since the optical identifier is used to describe a unique location, the physical coordinates of the target location in the system coordinate system can be extracted from it.

[0050] For example, a system coordinate system can be pre-constructed. This system coordinate system, also known as the world coordinate system or physical coordinate system, is a coordinate system specific to the object being measured. A location on the calibration cloth can be used as the origin of the system coordinate system. Once the origin of the system coordinate system is known, the positional relationship between the target position of the optical identifier and the origin can be determined. This allows us to determine the physical coordinates of the target position of the optical identifier within the system coordinate system. Therefore, the physical coordinates of the target position of the optical identifier within the system coordinate system can be described using the optical identifier, and thus, the physical coordinates of the target position of the optical identifier within the system coordinate system can be extracted from the optical identifier.

[0051] The target location of an optical identifier can include, but is not limited to, the upper left corner, upper right corner, lower left corner, lower right corner, or center position. These are just a few examples of target locations, and no limitation is imposed. Taking the center position of the optical identifier as an example, we can determine the positional relationship between the center position and the origin, and thus the physical coordinates of the center position in the system coordinate system. Therefore, the physical coordinates of the center position of this optical identifier in the system coordinate system can be described using the optical identifier.

[0052] The calibration cloth can be a calibration cloth with one optical identifier or a calibration cloth with multiple optical identifiers, and each optical identifier is used to describe the physical coordinates of its center position in the system coordinate system. Figure 2 The following explanation uses several optical identifiers as examples.

[0053] See Figure 2As shown, the example uses one 3D camera and three 2D cameras. The three 2D cameras can include two 2D barcode readers and one 2D panoramic camera. The 2D barcode readers are used to read the barcode information or other code information on the object, and the 2D panoramic camera is used to capture panoramic images of the object.

[0054] During the installation of the 2D camera, there are no restrictions on the installation angle or height; the 2D camera can be installed arbitrarily. Similarly, during the installation of the 3D camera, there are no restrictions on the installation angle or height; the 3D camera can be installed arbitrarily. After the 2D camera is installed, it is necessary to measure the vertical height between the 2D camera and the object being measured (i.e., the mounting height of the 2D camera). This vertical height can be the height of the 2D camera's optical center, which is perpendicular to the object being measured. Figure 2 In the diagram, the vertical distance between one 2D barcode reader camera and the object being measured is h1, the vertical distance between another 2D barcode reader camera and the object being measured is h2, and the vertical distance between the 2D panoramic camera and the object being measured is h3.

[0055] For both 2D and 3D cameras, camera intrinsic parameters can include camera focal length (fx, fy), camera center (cx, cy), and distortion coefficients (k1, k2, p1, p2, k3). For 2D cameras, the camera intrinsic parameters are used to transform a point Pc(x,y,z) in the camera coordinate system to a point Pi(x,y) in the image coordinate system, as shown in formula (1), which represents the transformation relationship between the camera coordinate system and the image coordinate system. If the calibration accuracy of the 2D camera is not strict, the camera intrinsic parameters of the 2D camera can be simplified, such as fy = fx, p1 = 0, p2 = 0, k3 = 0. In this case, see formula (2), which represents the transformation relationship between the camera coordinate system and the image coordinate system.

[0056] Pi(x,y)=[fx,fy,cx,cy][k1,k2,p1,p2,k3]Pc(x,y,z) Formula (1)

[0057] Pi(x,y)=[f,f,cx,cy][k1,k2]Pc(x,y,z) Formula (2)

[0058] As can be seen from formula (1), a point Pc(x,y,z) in the camera coordinate system can be converted to a point Pi(x,y) in the image coordinate system based on camera intrinsic parameters such as camera focal length (fx,fy), camera center (cx,cy), and distortion coefficients (k1,k2,p1,p2,k3). Similarly, a point Pi(x,y) in the image coordinate system can be converted to a point Pc(x,y,z) in the camera coordinate system based on the same camera intrinsic parameters. Clearly, based on points Pi(x,y) in the image coordinate system and Pc(x,y,z) in the camera coordinate system—that is, multiple coordinate point pairs—the camera intrinsic parameters such as camera focal length (fx,fy), camera center (cx,cy), and distortion coefficients (k1,k2,p1,p2,k3) can also be determined.

[0059] As can be seen from formula (2), a point Pc(x,y,z) in the camera coordinate system can be converted to a point Pi(x,y) in the image coordinate system based on camera intrinsic parameters such as camera focal length f, camera center (cx,cy), and distortion coefficients (k1,k2). Similarly, a point Pi(x,y) in the image coordinate system can be converted to a point Pc(x,y,z) in the camera coordinate system based on camera intrinsic parameters such as camera focal length f, camera center (cx,cy), and distortion coefficients (k1,k2). Based on points Pi(x,y) in the image coordinate system and points Pc(x,y,z) in the camera coordinate system, i.e., multiple coordinate point pairs, camera intrinsic parameters such as camera focal length f, camera center (cx,cy), and distortion coefficients (k1,k2) can also be determined.

[0060] For both 2D and 3D cameras, camera extrinsic parameters can include rotation parameters R (Rx, Ry, Rz) and translation parameters T (Tx, Ty, Tz). Rx is the X-axis rotation parameter, Ry is the Y-axis rotation parameter, Rz is the Z-axis rotation parameter, Tx is the X-axis translation parameter, Ty is the Y-axis translation parameter, and Tz is the Z-axis translation parameter. The camera extrinsic parameters are used to transform a point Ps(x,y,z) in the system coordinate system to a point Pc(x,y,z) in the camera coordinate system, as shown in formula (3), which represents the transformation relationship between the system coordinate system and the camera coordinate system.

[0061] Pc(x,y,z)=[R|T]Ps(x,y,0) Formula (3)

[0062] As can be seen from formula (3), a point Ps(x,y,z) in the system coordinate system can be converted to a point Pc(x,y,z) in the camera coordinate system based on camera extrinsic parameters such as rotation parameter R and translation parameter T. Conversely, a point Pc(x,y,z) in the camera coordinate system can be converted to a point Ps(x,y,z) in the system coordinate system based on camera extrinsic parameters such as rotation parameter R and translation parameter T. Based on multiple coordinate point pairs, such as points Ps(x,y,z) in the system coordinate system and points Pc(x,y,z) in the camera coordinate system, the camera extrinsic parameters such as rotation parameter R and translation parameter T can also be determined.

[0063] For the calibration process of a 3D camera, see [link to documentation]. Figure 3 As shown, the following steps may be included:

[0064] Step 301: Acquire a calibration 3D point cloud of the measurement object using a 3D camera, and determine the 3D spatial coordinates of the calibration points (which can be multiple calibration points) based on the calibration 3D point cloud.

[0065] For example, a calibration cloth bearing optical identifiers and calibration grids is laid on the object being measured, covering the measurement area of ​​the 3D camera, ensuring the camera can see the calibration grids and optical identifiers. With the calibration cloth stationary, the 3D camera acquires a calibration 3D point cloud for the object being measured; that is, the calibration 3D point cloud includes 3D points for the object. Since the 3D camera can see the calibration grids, multiple calibration points are determined based on these grids, such as the intersection of black and white checkerboard grids.

[0066] Since there are many calibration points, K calibration points can be selected from all calibration points. The value of K can be configured based on experience, indicating that the calibration operation is completed based on K sets of coordinate pairs.

[0067] In summary, after obtaining the calibrated 3D point cloud, which includes the 3D spatial coordinates of all 3D points, the 3D spatial coordinates of K calibration points can be determined based on the calibrated 3D point cloud.

[0068] Step 302: Based on the positional relationship between the optical identifier and the calibration point, determine the physical coordinates of the calibration point in the system coordinate system. For example, for each of the K calibration points, determine the physical coordinates of the calibration point in the system coordinate system based on the positional relationship between the optical identifier and the calibration point.

[0069] For example, the physical coordinates of the target position of the optical identifier in the system coordinate system can be parsed from the optical identifier; wherein the target position can be the upper left corner, the upper right corner, the lower left corner, the lower right corner, or the center position. Then, based on the physical coordinates of the target position of the optical identifier in the system coordinate system and the positional relationship between the target position of the optical identifier and the calibration point, the physical coordinates of the calibration point in the system coordinate system can be determined.

[0070] For example, for each of the K calibration points, the optical identifier closest to that calibration point is selected from all optical identifiers. Of course, other optical identifiers can also be selected; there are no restrictions on this. The optical identifier can be a QR code, a number, or other information that can describe a unique location. Therefore, the physical coordinates of the target location of the optical identifier in the system coordinate system are parsed from the optical identifier.

[0071] The positional relationship between the target position of the optical identifier and the calibration point is known. For example, the calibration point is separated from the target position of the optical identifier by 4 chessboard squares, and the calibration point is located to the left of the target position of the optical identifier. Therefore, based on the positional relationship between the target position of the optical identifier and the calibration point, and the physical coordinates of the target position of the optical identifier in the system coordinate system, the physical coordinates of the calibration point in the system coordinate system can be determined. There are no restrictions on the method of determining these physical coordinates.

[0072] Step 303: Generate multiple coordinate point pairs corresponding to multiple calibration points. For each coordinate point pair corresponding to a calibration point, the coordinate point pair may include the three-dimensional spatial coordinates and physical coordinates corresponding to the calibration point.

[0073] For example, for each of the K calibration points, in step 301, the three-dimensional spatial coordinates corresponding to the calibration point can be obtained, and in step 302, the physical coordinates of the calibration point in the system coordinate system can be obtained, thereby obtaining the coordinate point pair corresponding to the calibration point. The coordinate point pair may include the three-dimensional spatial coordinates and the physical coordinates. In this way, K coordinate point pairs corresponding to K calibration points can be obtained.

[0074] Step 304: Based on the calibrated intrinsic parameters of the 3D camera and multiple coordinate point pairs, the target parameter values ​​corresponding to the camera extrinsic parameters can be determined, and the target parameter values ​​of the camera extrinsic parameters can be calibrated for the 3D camera.

[0075] For example, when a 3D camera leaves the factory, its intrinsic parameters are pre-calibrated, such as focal length (fx, fy), camera center (cx, cy), and distortion coefficients (k1, k2, p1, p2, k3). Since these intrinsic parameters are already pre-calibrated, no further calibration is needed. Therefore, only the extrinsic parameters of the 3D camera need to be calibrated. For instance, the target values ​​for the rotation parameters R (Rx, Ry, Rz) can be calibrated, such as the target values ​​for the X-axis, Y-axis, and Z-axis rotation parameters. Similarly, the target values ​​for the translation parameters T (Tx, Ty, Tz) can be calibrated, such as the target values ​​for the X-axis, Y-axis, and Z-axis translation parameters.

[0076] For example, by combining formulas (1) and (3), we can obtain the transformation relationship shown in formula (4), which is the transformation relationship between three-dimensional spatial coordinates and system coordinate system.

[0077] Pi(x,y,z)=[fx,fy,cx,cy][k1,k2,p1,p2,k3][R|T]Ps(x,y,z) Formula (4)

[0078] In formula (4), the camera focal length (fx, fy), camera center (cx, cy) and distortion coefficients (k1, k2, p1, p2, k3) are camera intrinsic parameters, which are known values. The rotation parameter R and translation parameter T are camera extrinsic parameters, which are unknown values ​​and are parameter values ​​that need to be calibrated. The coordinate point pairs can include three-dimensional spatial coordinates (corresponding to Pi) and physical coordinates in the system coordinate system (corresponding to Ps). After substituting K coordinate point pairs into formula (4), the target values ​​of the rotation parameter R and the translation parameter T can be obtained, and then the target values ​​of the rotation parameter and translation parameter can be calibrated for the three-dimensional camera.

[0079] For the calibration process of a 2D camera, see [link / reference]. Figure 4 As shown, the following steps may be included:

[0080] Step 401: Acquire a calibration two-dimensional image of the object being measured using a two-dimensional camera, and determine the pixel coordinates of the calibration points (which can be multiple calibration points) in the image coordinate system based on the calibration two-dimensional image. The calibration two-dimensional image can be an RGB image or a grayscale image.

[0081] For example, a calibration cloth bearing optical identifiers and calibration grids is laid on the object being measured, covering the measurement area of ​​the 2D camera, ensuring the camera can see the calibration grids and optical identifiers. With the calibration cloth stationary, a 2D calibration image of the object is acquired by the 2D camera; this image includes the pixel coordinates of the object. Since the 2D camera can see the calibration grids, multiple calibration points are determined based on these grids, such as the intersection of black and white checkerboard grids.

[0082] Since there are many calibration points, M calibration points can be selected from all calibration points. The value of M can be configured based on experience, indicating that the calibration operation is completed based on M sets of coordinate pairs.

[0083] In summary, after obtaining the calibration two-dimensional image, the pixel coordinates of each calibration point in the image coordinate system can be determined based on the calibration two-dimensional image, that is, the pixel coordinates of M calibration points in the image coordinate system are obtained.

[0084] Step 402: Based on the positional relationship between the optical identifier and the calibration point, determine the physical coordinates of the calibration point in the system coordinate system. For example, for each of the M calibration points, determine the physical coordinates of the calibration point in the system coordinate system based on the positional relationship between the optical identifier and the calibration point.

[0085] For example, the physical coordinates of the target position of the optical identifier in the system coordinate system can be parsed from the optical identifier; wherein the target position can be the upper left corner, the upper right corner, the lower left corner, the lower right corner, or the center position. Then, based on the physical coordinates of the target position of the optical identifier in the system coordinate system and the positional relationship between the target position of the optical identifier and the calibration point, the physical coordinates of the calibration point in the system coordinate system can be determined.

[0086] For example, for each of the M calibration points, the optical identifier closest to that calibration point is selected from all optical identifiers. Of course, other optical identifiers can also be selected; there are no restrictions on this. The optical identifier can be a QR code, a number, or other information that can describe a unique location. Therefore, the physical coordinates of the target location of the optical identifier in the system coordinate system are parsed from the optical identifier.

[0087] The positional relationship between the target position of the optical identifier and the calibration point is known. For example, the calibration point is separated from the target position of the optical identifier by 4 chessboard squares, and the calibration point is located to the left of the target position of the optical identifier. Therefore, based on the positional relationship between the target position of the optical identifier and the calibration point, and the physical coordinates of the target position of the optical identifier in the system coordinate system, the physical coordinates of the calibration point in the system coordinate system can be determined. There are no restrictions on the method of determining these physical coordinates.

[0088] Step 403: Generate multiple coordinate point pairs corresponding to multiple calibration points. For each coordinate point pair corresponding to a calibration point, the coordinate point pair may include the pixel coordinates and physical coordinates corresponding to the calibration point.

[0089] For example, for each of the M calibration points, in step 401, the pixel coordinates of the calibration point in the image coordinate system can be obtained, and in step 402, the physical coordinates of the calibration point in the system coordinate system can be obtained, thereby obtaining the coordinate point pair corresponding to the calibration point. The coordinate point pair may include the pixel coordinates and the physical coordinates, thereby obtaining the M coordinate point pairs corresponding to the M calibration points.

[0090] Step 404: Configure an initial focal length value (i.e., a fixed focal length value) for the camera focal length, and determine the target parameter values ​​corresponding to the calibration parameters of the two-dimensional camera based on the initial focal length value and multiple coordinate point pairs.

[0091] For example, when a 2D camera leaves the factory, its intrinsic and extrinsic parameters are not calibrated. Therefore, the intrinsic and extrinsic parameters outside the camera's focal length are used as parameters to be calibrated. These parameters include the camera center (cx, cy), distortion coefficients (k1, k2, p1, p2, k3), rotation parameters R (Rx, Ry, Rz), and translation parameters T (Tx, Ty, Tz). Based on this, the target center value corresponding to the camera center is determined (e.g., the target center value corresponding to camera center cx and camera center cy), and the target coefficient values ​​corresponding to the distortion coefficients are determined (e.g., the target coefficient value corresponding to distortion coefficient k1 and distortion coefficient k2). The target coefficient values ​​corresponding to the distortion coefficients p1, p2, and k3 are determined. The target values ​​of the rotation parameters R (Rx, Ry, Rz) are determined (e.g., the target values ​​of the rotation parameters corresponding to the X-axis, Y-axis, and Z-axis rotation parameters). The target values ​​of the translation parameters T (Tx, Ty, Tz) are determined (e.g., the target values ​​of the translation parameters corresponding to the X-axis, Y-axis, and Z-axis translation parameters).

[0092] In one possible implementation, by combining formulas (1) and (3), the transformation relationship shown in formula (5) can be obtained. Formula (5) is the transformation relationship between the image coordinate system and the system coordinate system.

[0093] Pi(x,y)=[fx,fy,cx,cy][k1,k2,p1,p2,k3][R|T]Ps(x,y,0) Formula (5)

[0094] In formula (5), the camera focal length (fx, fy) is the initial focal length value, which can be configured according to experience, such as fx = fy = 1000. Of course, the initial focal length value can also be other values. There is no restriction on the initial focal length value. By configuring the camera focal length (fx, fy) as the initial focal length value, the amount of calculation in the calibration process can be reduced, the computing resources can be saved, the calibration speed can be improved, and only one frame of calibration two-dimensional image needs to be acquired to complete the calibration.

[0095] For example, the camera center (cx, cy) and distortion coefficients (k1, k2, p1, p2, k3) are camera intrinsic parameters, which are unknown values. The rotation parameter R and translation parameter T are camera extrinsic parameters, which are also unknown values. The above parameters are the parameters to be calibrated. The coordinate point pairs can include pixel coordinates (corresponding to Pi) in the image coordinate system and physical coordinates (corresponding to Ps) in the system coordinate system. Obviously, after substituting the M coordinate point pairs into formula (5), we can obtain the target center value corresponding to the camera center (cx, cy), the target coefficient value corresponding to the distortion coefficients (k1, k2, p1, p2, k3), the rotation parameter target value corresponding to the rotation parameter R (Rx, Ry, Rz), and the translation parameter target value corresponding to the translation parameter T (Tx, Ty, Tz).

[0096] In another possible implementation, in addition to configuring an initial focal length value for the camera, an initial center value can also be configured for the camera center, and initial coefficient values ​​can be configured for the distortion coefficients. For example, the center of the calibrated 2D image can be used as the initial center value, that is, half of the horizontal resolution of the calibrated 2D image can be used as the initial center value of the camera center cx, and half of the vertical resolution of the calibrated 2D image can be used as the initial center value of the camera center cy. The initial coefficient values ​​of the distortion coefficients can be configured to 0, such as k1=0, k2=0, p1=0, p2=0, k3=0. In this case, by combining formulas (1) and (3), the transformation relationship shown in formula (6) can be obtained, which is the transformation relationship between the image coordinate system and the system coordinate system. Since the initial coefficient value of the distortion coefficients is 0, formula (6) removes the distortion coefficients compared to formula (5).

[0097] Pi(x,y)=[fx,fy,cx,cy][R|T]Ps(x,y,0) Formula (6)

[0098] In formula (6), the camera focal length (fx, fy) is the initial focal length value, which can be configured based on experience, such as fx = fy = 1000. The camera center (cx, cy) is the center of the calibrated two-dimensional image, which is a known value. The distortion coefficients (k1, k2, p1, p2, k3) are 0, which is a known value. The rotation parameter R and the translation parameter T are camera extrinsic parameters, which are unknown values. The coordinate point pairs can include the pixel coordinates in the image coordinate system (corresponding to Pi) and the physical coordinates in the system coordinate system (corresponding to Ps). Obviously, after substituting the M coordinate point pairs into formula (6), the initial values ​​of the rotation parameters R (Rx, Ry, Rz) and the initial values ​​of the translation parameters T (Tx, Ty, Tz) can be obtained. That is, the initial values ​​of the rotation parameters and the initial values ​​of the translation parameters are determined based on the initial focal length value, the initial center value, the initial coefficient value, and multiple coordinate point pairs.

[0099] Then, the initial center value, initial coefficient value, initial value of rotation parameter, and initial value of translation parameter can be optimized (e.g., nonlinear optimization) to obtain the target center value corresponding to the camera center (cx, cy), the target coefficient value corresponding to the distortion coefficient (k1, k2, p1, p2, k3), the target value of the rotation parameter R (Rx, Ry, Rz) corresponding to the rotation parameter, and the target value of the translation parameter T (Tx, Ty, Tz) corresponding to the translation parameter.

[0100] For example, based on the least squares error of formula (5), nonlinear optimization methods (such as the LM method, which stands for Levenberg-Marquardt method and is a method for least squares estimation of regression parameters in nonlinear regression) can be used to jointly optimize the camera center (cx, cy), distortion coefficients (k1, k2, p1, p2, k3), rotation parameters R (Rx, Ry, Rz) and translation parameters T (Tx, Ty, Tz) to obtain the target center value corresponding to the camera center (cx, cy), the target coefficient value corresponding to the distortion coefficients (k1, k2, p1, p2, k3), the target value of the rotation parameter corresponding to the rotation parameter R (Rx, Ry, Rz), and the target value of the translation parameter corresponding to the translation parameter T (Tx, Ty, Tz). There are no restrictions on this optimization process.

[0101] For example, regarding the computational feasibility of formula (6), since three points can determine a plane, and each point can provide two equations (x, y) as in formula (6), six parameters can be calculated to meet the calculation requirements of camera extrinsic parameters (R|T). cx, cy, k1, and k2 are mainly related to distortion, and their calculation accuracy depends on whether the calibration grid covers enough camera field of view. They are less related to the number of images. The calibration cloth (calibration plate) in this embodiment just meets the conditions, so these parameters can be calibrated through a single frame image.

[0102] In summary, based on the initial focal length of the camera, we can obtain the target center value corresponding to the camera center, the target coefficient value corresponding to the distortion coefficient, the target rotation parameter value corresponding to the rotation parameter R (Rx, Ry, Rz), and the target translation parameter value corresponding to the translation parameter T (Tx, Ty, Tz).

[0103] Step 405: Based on the target parameter values, the vertical height between the 2D camera and the measurement object, and the initial focal length value, determine the candidate focal length values ​​corresponding to the camera focal length (fx, fy) of the 2D camera.

[0104] For example, the parameters to be calibrated may include the camera center, distortion coefficients, rotation parameters, and translation parameters. The translation parameters include X-axis translation parameters, Y-axis translation parameters, and Z-axis translation parameters. That is, the target value of the translation parameter corresponding to the Z-axis translation parameter can be obtained. Based on this, an adjustment factor can be determined based on the vertical height and the target value of the translation parameter corresponding to the Z-axis translation parameter. The initial focal length value is then adjusted based on the adjustment factor to obtain the candidate focal length value corresponding to the camera focal length. For example, the candidate focal length value can be determined using formula (7). Of course, formula (7) is just an example, and this determination method is not limited.

[0105]

[0106] In formula (7), f represents the initial focal length value, that is, the camera focal lengths fx and fy both correspond to the initial focal length value, h represents the vertical height between the two-dimensional camera and the object being measured, and Tz represents the target value of the translation parameter corresponding to the Z-axis translation parameter. Used to represent the adjustment factor, f ′ Used to represent candidate focal length values, that is, both camera focal length fx and camera focal length fy correspond to the candidate focal length value.

[0107] For example, based on the principle of triangulation, it is known that Tz in the inverse matrix of camera extrinsic parameters (R|T) has a nearly linear relationship with the camera focal length f. Tz is the height of the camera center perpendicular to the plane of the object being measured. Therefore, the initial value of the camera focal length f (i.e., the initial focal length value) can be arbitrarily set. After calculating the target center value corresponding to the camera center (cx, cy), the target coefficient value corresponding to the distortion coefficient (k1, k2, p1, p2, k3), the target value of the rotation parameter R (Rx, Ry, Rz), and the target value of the translation parameter T (Tx, Ty, Tz) according to formula (5) or formula (6), Tz in the inverse matrix of camera extrinsic parameters (R|T) can be obtained. Then, the initial focal length value is corrected according to formula (7) to obtain the candidate focal length value. The correct camera intrinsic parameters and camera extrinsic parameters are obtained through iterative updates.

[0108] Step 406: Determine whether the difference between the candidate focal length value and the initial focal length value is less than a preset threshold.

[0109] If yes, then proceed to step 407; otherwise, proceed to step 408.

[0110] For example, after obtaining the candidate focal length value, the difference between the candidate focal length value and the initial focal length value can be calculated, and it can be determined whether the difference between the candidate focal length value and the initial focal length value is less than a preset threshold. The preset threshold can be configured based on experience, and there is no restriction on the value of the preset threshold.

[0111] Step 407: Determine the candidate focal length value as the target focal length value corresponding to the camera focal length.

[0112] In summary, we can obtain the target focal length value corresponding to the camera focal length (fx, fy), the target center value corresponding to the camera center (cx, cy), the target coefficient value corresponding to the distortion coefficients (k1, k2, p1, p2, k3), the target rotation parameter value corresponding to the rotation parameter R (Rx, Ry, Rz), and the target translation parameter value corresponding to the translation parameter T (Tx, Ty, Tz). In this way, we can calibrate the intrinsic and extrinsic parameters of the 2D camera.

[0113] Step 408: Determine the candidate focal length value as the initial focal length value corresponding to the camera focal length, and return to step 404. Based on the updated initial focal length value and multiple coordinate points, redetermine the target parameter values ​​corresponding to the calibration parameters of the 2D camera. That is, the target parameter values ​​will be updated, thereby updating the camera focal length (fx, fy), camera center (cx, cy), distortion coefficients (k1, k2, p1, p2, k3), rotation parameters R (Rx, Ry, Rz), and translation parameters T through iterative updates until the correct camera intrinsic and extrinsic parameters are obtained, and calibrate the camera intrinsic and extrinsic parameters for the 2D camera.

[0114] In summary, the camera calibration process involves calibrating the intrinsic and extrinsic parameters of a 2D camera and the extrinsic parameters of a 3D camera. After calibration, relevant applications can be executed based on these parameters. For example, in the application of package volume measurement and optical symbol recognition in logistics systems, a 3D camera can be used for package volume measurement and classification, multiple small-field-of-view 2D cameras can be used for optical symbol recognition, and a large-field-of-view 2D camera can be used to obtain panoramic images of packages. Based on the intrinsic and extrinsic parameters of the 2D and 3D cameras, the 3D positions of multiple cameras are correlated, and temporal correlation prediction technology is used to obtain the volume value, category information, optical symbol information, and panoramic image of each package. This information is then used for subsequent operations such as piece-rate billing, preventing missed scans, package verification, and sorting and loading.

[0115] In one possible implementation, a target object is placed on the measurement object, and during the process of the measurement object moving at a constant speed in a straight line, a two-dimensional image of the target object can be acquired by a two-dimensional camera, and a three-dimensional point cloud of the target object can be acquired by a three-dimensional camera.

[0116] Then, the three-dimensional spatial coordinates corresponding to the target object are determined based on the target three-dimensional point cloud, and the three-dimensional spatial coordinates are converted into the initial physical coordinates of the target object in the system coordinate system based on the camera intrinsic and extrinsic parameters of the three-dimensional camera. For example, refer to the conversion relationship shown in formula (4). Formula (4) is the conversion relationship between three-dimensional spatial coordinates and the system coordinate system. In formula (4), the camera intrinsic and extrinsic parameters are known values. After substituting the three-dimensional spatial coordinates corresponding to the target object into formula (4), the initial physical coordinates of the target object in the system coordinate system can be obtained. For example, the three-dimensional camera can identify the three-dimensional spatial coordinates Pc(x,y,z) of the target object. Formula (4) can be transformed to obtain the conversion relationship shown in formula (8). In formula (8), only the camera extrinsic parameters of the three-dimensional camera are involved, and the camera intrinsic parameters of the three-dimensional camera are not involved. In this way, the three-dimensional spatial coordinates Pc(x,y,z) of the target object can be converted into the initial physical coordinates Ps(x,y,z) of the target object in the system coordinate system based on formula (8).

[0117] Ps(x,y,z)=[R|T) -1 Pc(x,y,z) Formula (8)

[0118] After obtaining the initial physical coordinates of the target object in the system coordinate system, the physical coordinates of the target object after the target duration (denoted as the target physical coordinates) can be predicted. That is, the target object will move to the position corresponding to the target physical coordinates after the target duration (e.g., after an interval of 10 seconds). For example, the target physical coordinates of the target object in the system coordinate system can be determined based on the motion speed of the measured object, the target duration, and the initial physical coordinates. For example, the target physical coordinates of the target object in the system coordinate system can be determined using formula (9). In formula (9), Ps(x,y) represents the initial physical coordinates of the target object in the system coordinate system, that is, z in the initial physical coordinates Ps(x,y,z) is set to 0. Ps′(x,y) represents the target physical coordinates of the target object in the system coordinate system. s represents the motion speed of the measured object. There are no restrictions on the method of obtaining this motion speed. For example, the motion speed can be obtained by reading the tachometer, or the motion speed can be measured using a speedometer. t represents the target duration, which can be configured according to requirements.

[0119] Ps′(x,y)=s·t+Ps(x,y) Formula (9)

[0120] After obtaining the target object's physical coordinates in the system coordinate system, the target pixel coordinates of the target object after the target time period can be predicted based on the target physical coordinates. This allows for the association of the target pixel coordinates with the target physical coordinates, thus enabling the prediction of the target object. Furthermore, the target pixel coordinates and their corresponding 3D spatial coordinates can also be associated; that is, the target pixel coordinates in the target 2D image can be associated with the target object's 3D spatial coordinates in the target 3D point cloud.

[0121] To predict the target pixel coordinates of a target object after a target duration based on its physical coordinates, the physical coordinates can be converted into the target pixel coordinates of the target object in the image coordinate system based on the camera intrinsic and extrinsic parameters of the 2D camera. For example, refer to the conversion relationship shown in formula (5), which is the conversion relationship between the image coordinate system and the system coordinate system. In formula (5), both the camera intrinsic and extrinsic parameters are known values. After substituting the physical coordinates of the target object into formula (5), the target pixel coordinates of the target object in the image coordinate system can be obtained. Based on this, the pixel position corresponding to the target pixel coordinates can be located from the target 2D image (i.e., the 2D image after the target duration) corresponding to the target duration. This pixel position is the position of the target object in the image coordinate system. The 2D camera can recognize the image information of the object. During information binding, the 3D information of the object in the system coordinate system is passively projected into the image based on the camera extrinsic and camera intrinsic parameters, and then the information binding is performed.

[0122] In one possible implementation, while a target object is placed on the measurement object and the measurement object is moving at a constant speed in a straight line, a two-dimensional image of the target object can be acquired using a two-dimensional camera, and a three-dimensional point cloud of the target object can be acquired using a three-dimensional camera. Then, the three-dimensional spatial coordinates corresponding to the target object are determined based on the target three-dimensional point cloud, and the three-dimensional spatial coordinates are converted into the target physical coordinates of the target object in the system coordinate system based on the camera intrinsic and extrinsic parameters of the three-dimensional camera.

[0123] After obtaining the target object's physical coordinates in the system coordinate system, the target pixel coordinates in the target 2D image can be determined based on these physical coordinates, thus associating the pixel coordinates with the physical coordinates. Furthermore, the target pixel coordinates can be associated with the corresponding 3D spatial coordinates of the physical coordinates; that is, the target pixel coordinates in the 2D image can be associated with the 3D spatial coordinates of the target object in the 3D point cloud. To determine the target pixel coordinates based on the physical coordinates, these physical coordinates can be converted into the target pixel coordinates in the image coordinate system based on the camera's intrinsic and extrinsic parameters.

[0124] As can be seen from the above technical solutions, in this embodiment, camera calibration can be performed simply by laying a calibration cloth or calibration plate carrying optical identifiers, effectively reducing the cost of manufacturing and carrying the calibration object. By acquiring a calibration two-dimensional image of the object to be measured using a two-dimensional camera, and combining the vertical height between the two-dimensional camera and the object, the intrinsic and extrinsic parameters of the two-dimensional camera can be calculated, and the intrinsic and extrinsic parameters of the two-dimensional camera can be calibrated, greatly improving implementation efficiency and saving labor costs. By associating each camera with the system coordinate system, while decoupling the cameras from each other, the scalability of camera calibration is improved. By configuring an initial focal length value for the camera, and based on the initial focal length value, the target parameter values ​​corresponding to the parameters to be calibrated of the two-dimensional camera are determined based on multiple coordinate point pairs, thereby reducing the amount of calculation, saving computing resources, and improving calibration speed. Based on the above camera calibration method, the calibration operation can be simplified, the calibration time reduced, and the calibration labor costs saved. All camera measurement and identification information is associated with the system plane described by the calibration cloth. The temporal association is based on the known system motion speed (reading the tachometer or using a speedometer to measure). There are no restrictions on the type and number of cameras. Each camera is independent of the others, and expansion is convenient.

[0125] In one possible implementation, see Figure 5 As shown, camera parameter calibration methods may include:

[0126] Step 501: Set up multiple cameras according to application requirements.

[0127] Step 502: Measure the installation height of each 2D camera.

[0128] Step 503: Lay out a calibration cloth with optical identifiers to cover the measurement space of all cameras.

[0129] Step 504: Control each camera to acquire a calibration image, such as an RGB image or a grayscale image.

[0130] Step 505: Perform extrinsic parameter calibration of the 3D camera.

[0131] Step 506: Perform intrinsic and extrinsic parameter calibration of the 2D camera.

[0132] Step 507: Obtain the speed of the object being measured.

[0133] Step 508: Perform information association between multiple cameras.

[0134] Based on the same concept as the above method, this application proposes a camera parameter calibration device, in which a calibration object is placed on the measurement object. The calibration object includes an optical identifier and multiple calibration points. See [link to relevant documentation]. Figure 6 The diagram shown is a structural schematic of the device, which may include:

[0135] The acquisition module 61 is used to acquire a calibration two-dimensional image of the measurement object using a two-dimensional camera, determine the pixel coordinates of the calibration point in the image coordinate system based on the calibration two-dimensional image, and determine the physical coordinates of the calibration point in the system coordinate system based on the positional relationship between the optical identifier and the calibration point.

[0136] The generation module 62 is used to generate multiple coordinate point pairs corresponding to multiple calibration points. For each coordinate point pair corresponding to a calibration point, the coordinate point pair includes the pixel coordinates and physical coordinates corresponding to the calibration point.

[0137] The determination module 63 is used to configure an initial focal length value for the camera focal length, determine the target parameter value corresponding to the calibration parameter of the two-dimensional camera based on the initial focal length value and the plurality of coordinate point pairs, and determine the target focal length value corresponding to the camera focal length of the two-dimensional camera based on the target parameter value, the vertical height between the two-dimensional camera and the measurement object, and the initial focal length value.

[0138] The calibration module 64 is used to calibrate the target focal length value and the target parameter value for the two-dimensional camera.

[0139] For example, when the acquisition module 61 determines the physical coordinates of the calibration point in the system coordinate system based on the positional relationship between the optical identifier and the calibration point, it is specifically used to: parse the physical coordinates of the target position of the optical identifier in the system coordinate system from the optical identifier; wherein, the target position of the optical identifier is the upper left corner, or the upper right corner, or the lower left corner, or the lower right corner, or the center position of the optical identifier; and determine the physical coordinates of the calibration point in the system coordinate system based on the physical coordinates of the target position in the system coordinate system and the positional relationship between the target position of the optical identifier and the calibration point.

[0140] For example, when determining the target focal length value corresponding to the camera focal length of the two-dimensional camera based on the target parameter value, the vertical height between the two-dimensional camera and the measurement object, and the initial focal length value, the determining module 63 is specifically used to: determine the candidate focal length value corresponding to the camera focal length of the two-dimensional camera based on the target parameter value, the vertical height between the two-dimensional camera and the measurement object, and the initial focal length value; if the difference between the candidate focal length value and the initial focal length value is less than a preset threshold, then the candidate focal length value is determined as the target focal length value corresponding to the camera focal length; otherwise, the candidate focal length value is determined as the initial focal length value, and the operation of determining the target parameter value corresponding to the calibration parameter of the two-dimensional camera based on the initial focal length value and the multiple coordinate points is returned.

[0141] For example, the parameters to be calibrated include camera center, distortion coefficient, rotation parameters, and translation parameters, wherein the translation parameters include X-axis translation parameters, Y-axis translation parameters, and Z-axis translation parameters; when the determining module 63 determines the candidate focal length value corresponding to the camera focal length of the two-dimensional camera based on the target parameter value, the vertical height between the two-dimensional camera and the measurement object, and the initial focal length value, it is specifically used to: determine an adjustment factor based on the target parameter value corresponding to the vertical height and the Z-axis translation parameter; and adjust the initial focal length value based on the adjustment factor to obtain the candidate focal length value.

[0142] For example, the parameters to be calibrated include the camera center, distortion coefficients, rotation parameters, and translation parameters. When the determining module 63 determines the target parameter values ​​corresponding to the parameters to be calibrated for the two-dimensional camera based on the initial focal length value and the multiple coordinate point pairs, it is specifically used to: configure an initial center value for the camera center and configure an initial coefficient value for the distortion coefficients; determine the initial value of the rotation parameter corresponding to the rotation parameter and the initial value of the translation parameter corresponding to the translation parameter based on the initial focal length value, the initial center value, the initial coefficient value, and the multiple coordinate point pairs; optimize the initial center value, the initial coefficient value, the initial value of the rotation parameter, and the initial value of the translation parameter to obtain the target center value corresponding to the camera center, the target coefficient value corresponding to the distortion coefficient, the target value of the rotation parameter corresponding to the rotation parameter, and the target value of the translation parameter corresponding to the translation parameter.

[0143] For example, the acquisition module 61 is further configured to acquire a calibration 3D point cloud for the measurement object using a 3D camera, and determine the 3D spatial coordinates corresponding to the calibration point based on the calibration 3D point cloud; determine the physical coordinates of the calibration point in the system coordinate system based on the positional relationship between the optical identifier and the calibration point; the generation module 62 is further configured to generate multiple coordinate point pairs corresponding to multiple calibration points, and for each coordinate point pair corresponding to a calibration point, the coordinate point pair includes the 3D spatial coordinates and physical coordinates corresponding to the calibration point; the calibration module 64 is further configured to determine the target parameter value corresponding to the camera extrinsic parameters based on the calibrated camera intrinsic parameters of the 3D camera and the multiple coordinate point pairs, and to calibrate the target parameter value of the camera extrinsic parameters of the 3D camera.

[0144] For example, when a target object is placed on the measurement object and the measurement object moves at a constant linear speed, the acquisition module 61 is further configured to acquire a target two-dimensional image of the measurement object using the two-dimensional camera and acquire a target three-dimensional point cloud of the measurement object using the three-dimensional camera; the determination module 63 is further configured to determine the three-dimensional spatial coordinates corresponding to the target object based on the target three-dimensional point cloud, and convert the three-dimensional spatial coordinates into the initial physical coordinates of the target object in the system coordinate system based on the camera intrinsic and extrinsic parameters of the three-dimensional camera; based on the movement speed of the measurement object, the target duration, and the initial physical coordinates, determine the target physical coordinates of the target object in the system coordinate system, and convert the target physical coordinates into the target pixel coordinates of the target object in the image coordinate system based on the camera intrinsic and extrinsic parameters of the two-dimensional camera, and locate the pixel position corresponding to the target pixel coordinates from the target two-dimensional image corresponding to the target duration.

[0145] Based on the same concept as the above method, this application proposes an electronic device, see [link to previous application]. Figure 7As shown, the electronic device may include: a processor 71 and a machine-readable storage medium 72, the machine-readable storage medium 72 storing machine-executable instructions that can be executed by the processor 71; the processor 71 is used to execute the machine-executable instructions to implement the camera parameter calibration method disclosed in the above example of this application.

[0146] Based on the same application concept as the above method, this application embodiment also provides a machine-readable storage medium storing a plurality of computer instructions, which, when executed by a processor, can implement the camera parameter calibration method disclosed in the above examples of this application.

[0147] The aforementioned machine-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0148] The systems, devices, modules, or units described in the above embodiments can be implemented by a computer entity or by a product with a certain function. A typical implementation device is a computer, which can be a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0149] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0150] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0151] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0152] Furthermore, these computer program instructions can also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0153] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0154] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. A method for calibrating camera parameters, characterized in that, A calibration object is placed on the object being measured, the calibration object including an optical identifier and multiple calibration points, the method comprising: A calibration two-dimensional image of the object being measured is acquired using a two-dimensional camera. The pixel coordinates of the calibration point in the image coordinate system are determined based on the calibration two-dimensional image. The physical coordinates of the calibration point in the system coordinate system are determined based on the positional relationship between the optical identifier and the calibration point. Generate multiple coordinate point pairs corresponding to multiple calibration points. For each coordinate point pair corresponding to a calibration point, the coordinate point pair includes the pixel coordinates and physical coordinates corresponding to the calibration point. Configure an initial focal length value for the camera focal length, and determine the target parameter value corresponding to the calibration parameters of the two-dimensional camera based on the initial focal length value and the multiple coordinate point pairs; Based on the target parameter value, the vertical height between the 2D camera and the measurement object, and the initial focal length value, a target focal length value corresponding to the camera focal length of the 2D camera is determined; wherein, an adjustment factor is determined based on the target parameter value corresponding to the vertical height and the Z-axis translation parameter in the parameters to be calibrated, and the initial focal length value is adjusted based on the adjustment factor to obtain a candidate focal length value; if the difference between the candidate focal length value and the initial focal length value is less than a preset threshold, then the candidate focal length value is determined as the target focal length value; The target focal length value and the target parameter value are calibrated for the two-dimensional camera.

2. The method according to claim 1, characterized in that, Determining the physical coordinates of the calibration point in the system coordinate system based on the positional relationship between the optical identifier and the calibration point includes: The physical coordinates of the target position of the optical identifier in the system coordinate system are parsed from the optical identifier; wherein, the target position of the optical identifier is the upper left corner, or the upper right corner, or the lower left corner, or the lower right corner, or the center position of the optical identifier; Based on the physical coordinates of the target location in the system coordinate system, the positional relationship between the target location of the optical identifier and the calibration point, the physical coordinates of the calibration point in the system coordinate system are determined.

3. The method according to claim 1, characterized in that, After adjusting the initial focal length value based on the adjustment factor to obtain the candidate focal length value, the method further includes: If the difference between the candidate focal length value and the initial focal length value is not less than a preset threshold, then the candidate focal length value is determined as the initial focal length value, and the operation of determining the target parameter value corresponding to the calibration parameter of the two-dimensional camera based on the initial focal length value and the multiple coordinate point pairs is returned.

4. The method according to claim 3, characterized in that, The parameters to be calibrated include camera center, distortion coefficient, rotation parameters, and translation parameters. The translation parameters include X-axis translation parameters, Y-axis translation parameters, and Z-axis translation parameters.

5. The method according to claim 1, characterized in that, The parameters to be calibrated include the camera center, distortion coefficients, rotation parameters, and translation parameters. Determining the target parameter values ​​corresponding to the parameters to be calibrated for the 2D camera based on the initial focal length value and the multiple coordinate point pairs includes: Configure an initial center value for the camera center and an initial coefficient value for the distortion coefficients; Based on the initial focal length value, the initial center value, the initial coefficient value, and the multiple coordinate point pairs, determine the initial values ​​of the rotation parameters corresponding to the rotation parameters and the initial values ​​of the translation parameters corresponding to the translation parameters; The initial center value, the initial coefficient value, the initial value of the rotation parameter, and the initial value of the translation parameter are optimized to obtain the target center value corresponding to the camera center, the target coefficient value corresponding to the distortion coefficient, the target value of the rotation parameter corresponding to the rotation parameter, and the target value of the translation parameter corresponding to the translation parameter.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: A 3D point cloud for calibration of the object being measured is acquired using a 3D camera, and the 3D spatial coordinates of the calibration point are determined based on the calibration point cloud; the physical coordinates of the calibration point in the system coordinate system are determined based on the positional relationship between the optical identifier and the calibration point. Generate multiple coordinate point pairs corresponding to multiple calibration points. For each coordinate point pair corresponding to a calibration point, the coordinate point pair includes the three-dimensional spatial coordinates and physical coordinates corresponding to the calibration point. Based on the calibrated intrinsic parameters of the 3D camera and the multiple coordinate point pairs, the target parameter values ​​corresponding to the camera extrinsic parameters are determined, and the target parameter values ​​of the camera extrinsic parameters are calibrated for the 3D camera.

7. The method according to claim 6, characterized in that, When a target object is placed on the object being measured, and the object being measured moves at a constant linear velocity, the method further includes: The two-dimensional camera acquires a two-dimensional image of the target object for the measurement object, and the three-dimensional camera acquires a three-dimensional point cloud of the target object for the measurement object. Based on the target 3D point cloud, determine the corresponding 3D spatial coordinates of the target object; based on the camera intrinsic and extrinsic parameters of the 3D camera, convert the 3D spatial coordinates into the initial physical coordinates of the target object in the system coordinate system; based on the motion speed of the measured object, the target duration, and the initial physical coordinates, determine the target physical coordinates of the target object in the system coordinate system. Based on the camera intrinsic and extrinsic parameters of the two-dimensional camera, the physical coordinates of the target are converted into the target pixel coordinates of the target object in the image coordinate system, and the pixel position corresponding to the target pixel coordinates is located from the target two-dimensional image corresponding to the target duration.

8. A camera parameter calibration device, characterized in that, A calibration object is placed on the object being measured. The calibration object includes an optical identifier and multiple calibration points. The device includes: The acquisition module is used to acquire a calibration two-dimensional image of the measurement object using a two-dimensional camera, determine the pixel coordinates of the calibration point in the image coordinate system based on the calibration two-dimensional image, and determine the physical coordinates of the calibration point in the system coordinate system based on the positional relationship between the optical identifier and the calibration point. The generation module is used to generate multiple coordinate point pairs corresponding to multiple calibration points. For each coordinate point pair corresponding to a calibration point, the coordinate point pair includes the pixel coordinates and physical coordinates corresponding to the calibration point. The determination module is used to configure an initial focal length value for the camera focal length, and determine the target parameter value corresponding to the calibration parameter of the two-dimensional camera based on the initial focal length value and the multiple coordinate point pairs; The determining module is used to determine a target focal length value corresponding to the camera focal length of the two-dimensional camera based on the target parameter value, the vertical height between the two-dimensional camera and the measurement object, and the initial focal length value; wherein, an adjustment factor is determined based on the target parameter value corresponding to the vertical height and the Z-axis translation parameter in the parameters to be calibrated, and the initial focal length value is adjusted based on the adjustment factor to obtain a candidate focal length value; if the difference between the candidate focal length value and the initial focal length value is less than a preset threshold, then the candidate focal length value is determined as the target focal length value; The calibration module is used to calibrate the target focal length value and the target parameter value for the two-dimensional camera.

9. The apparatus according to claim 8, characterized in that, in, When the acquisition module determines the physical coordinates of the calibration point in the system coordinate system based on the positional relationship between the optical identifier and the calibration point, it is specifically used to: parse the physical coordinates of the target position of the optical identifier in the system coordinate system from the optical identifier; wherein, the target position of the optical identifier is the upper left corner, or upper right corner, or lower left corner, or lower right corner, or center position of the optical identifier; and determine the physical coordinates of the calibration point in the system coordinate system based on the physical coordinates of the target position in the system coordinate system and the positional relationship between the target position of the optical identifier and the calibration point. The determining module is further configured to, after adjusting the initial focal length value based on the adjustment factor to obtain a candidate focal length value, if the difference between the candidate focal length value and the initial focal length value is not less than a preset threshold, determine the candidate focal length value as the initial focal length value, and return to perform the operation of determining the target parameter value corresponding to the calibration parameter of the two-dimensional camera based on the initial focal length value and the multiple coordinate points; The parameters to be calibrated include camera center, distortion coefficient, rotation parameters, and translation parameters, wherein the translation parameters include X-axis translation parameters, Y-axis translation parameters, and Z-axis translation parameters. The parameters to be calibrated include the camera center, distortion coefficients, rotation parameters, and translation parameters. When the determining module determines the target parameter values ​​corresponding to the parameters to be calibrated for the 2D camera based on the initial focal length value and the multiple coordinate point pairs, it specifically performs the following: configuring an initial center value for the camera center and initial coefficient values ​​for the distortion coefficients; determining initial values ​​for the rotation parameters and translation parameters corresponding to the rotation parameters based on the initial focal length value, the initial center value, the initial coefficient values, and the multiple coordinate point pairs; and optimizing the initial center value, the initial coefficient values, the initial rotation parameter values, and the initial translation parameter values ​​to obtain the target center value corresponding to the camera center, the target coefficient value corresponding to the distortion coefficients, the target rotation parameter value corresponding to the rotation parameters, and the target translation parameter value corresponding to the translation parameters. The acquisition module is further configured to acquire a calibration 3D point cloud of the measurement object using a 3D camera, and determine the 3D spatial coordinates corresponding to the calibration point based on the calibration 3D point cloud; determine the physical coordinates of the calibration point in the system coordinate system based on the positional relationship between the optical identifier and the calibration point; the generation module is further configured to generate multiple coordinate point pairs corresponding to multiple calibration points, wherein for each coordinate point pair corresponding to a calibration point, the coordinate point pair includes the 3D spatial coordinates and physical coordinates corresponding to the calibration point; the calibration module is further configured to determine the target parameter value corresponding to the camera extrinsic parameters based on the calibrated intrinsic parameters of the 3D camera and the multiple coordinate point pairs, thereby calibrating the target parameter value of the camera extrinsic parameters of the 3D camera. Wherein, when a target object is placed on the measurement object and the measurement object moves at a uniform linear speed, the acquisition module is further configured to acquire a target two-dimensional image of the measurement object using the two-dimensional camera and acquire a target three-dimensional point cloud of the measurement object using the three-dimensional camera; the determination module is further configured to determine the three-dimensional spatial coordinates corresponding to the target object based on the target three-dimensional point cloud, and convert the three-dimensional spatial coordinates into the initial physical coordinates of the target object in the system coordinate system based on the camera intrinsic and extrinsic parameters of the three-dimensional camera; based on the movement speed of the measurement object, the target duration, and the initial physical coordinates, determine the target physical coordinates of the target object in the system coordinate system, and convert the target physical coordinates into the target pixel coordinates of the target object in the image coordinate system based on the camera intrinsic and extrinsic parameters of the two-dimensional camera, and locate the pixel position corresponding to the target pixel coordinates from the target two-dimensional image corresponding to the target duration.

10. An electronic device, characterized in that, include: A processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; The processor is configured to execute the machine-executable instructions to implement the method of any one of claims 1-7.

Images