A camera extrinsic parameter calibration method and system combined with a posture detection module

Abstract

The embodiment of the present application relates to the technical field of intelligent excavator, and discloses a camera external parameter calibration method combined with a posture detection module, comprising: acquiring image information under a current scene through a camera arranged at an intelligent excavator, and determining a calibration point image and position information of the calibration point in the image information; acquiring posture angle information detected by a posture detection module group arranged at the intelligent excavator; acquiring vehicle information of the intelligent excavator to be calibrated, determining a third vector according to a first vector and a second vector, determining three-dimensional coordinate information of the calibration point in a gravity coordinate system according to posture angle information of the first posture detection module and the third vector, determining an external parameter matrix of the camera arranged at the intelligent excavator according to the three-dimensional coordinate information of the calibration point in the gravity coordinate system, the position information of the calibration point and a coordinate transformation algorithm, and performing data storage on the external parameter matrix. The scheme of the embodiment of the present application can improve the calibration efficiency of the overall scheme.

Description

Technical Field

[0001] This invention relates to the field of intelligent excavator technology, and specifically to a camera extrinsic parameter calibration method and system that incorporates an attitude detection module. Background Technology

[0002] Currently, camera extrinsic calibration can accurately determine the camera's three-dimensional spatial position relative to a known world coordinate system. For example, in robot vision applications, knowing the camera's specific coordinates within its working environment (world coordinate system) is crucial for the robot to accurately grasp target objects. In intelligent excavators, the movement path needs to be planned based on the visual information fed back by the camera, and accurate camera position information allows the excavator to precisely locate the target, avoiding digging deviations and other issues.

[0003] Current methods for extrinsic parameter calibration of cameras have the following drawbacks: First, they generally rely on tools such as calibration boards, RTK poles, QR codes, or drones, making the calibration process complex. Second, they require precise measurement of the distance between the camera and the calibration board. Third, this complex data collection process must be repeated for each new intelligent excavator installed, resulting in low efficiency and hindering automation. Fourth, because it requires holding the calibration board on-site and measuring the distance between the camera and the board, on-site data collection and calibration are necessary. Therefore, designing an efficient method for camera extrinsic parameter calibration has become a pressing technical problem for those skilled in the art. Summary of the Invention

[0004] To address the aforementioned shortcomings, this invention discloses a camera extrinsic parameter calibration method that incorporates an attitude detection module, which can quickly and accurately complete camera calibration and improve calibration efficiency.

[0005] The first aspect of this invention discloses a camera extrinsic parameter calibration method incorporating an attitude detection module, comprising:

[0006] The system uses a camera installed on the intelligent excavator to acquire image information of the current scene and to determine the calibration point image and the location information of the calibration point in the image information.

[0007] The attitude angle information detected by the attitude detection module group set at the intelligent excavator is obtained; the attitude detection module group includes a first attitude detection module, a second attitude detection module and a third attitude detection module, wherein the first attitude detection module is set at the vehicle base, the second attitude detection module is set at the excavator boom and the third attitude detection module is set at the excavator stick.

[0008] Obtain vehicle information of the intelligent excavator to be calibrated, including the boom length and stick length of the intelligent excavator, and determine a first vector based on the attitude angle information detected by the second attitude detection module and the boom length, and determine a second vector based on the attitude angle information detected by the third attitude detection module and the stick length.

[0009] A third vector is determined based on the first vector and the second vector, and the third vector is used to characterize the vector from the boom start point to the stick end point; the three-dimensional coordinate information of the calibration point in the gravity coordinate system is determined based on the attitude angle information of the first attitude detection module and the third vector, wherein the boom start point in the gravity coordinate system is the origin of the gravity coordinate system;

[0010] The extrinsic parameter matrix of the camera installed at the intelligent excavator is determined based on the three-dimensional coordinate information of the calibration point in the gravity coordinate system, the position information of the calibration point, and the coordinate transformation algorithm, and the extrinsic parameter matrix is ​​stored.

[0011] As an optional implementation, in the first aspect of the present invention, the step of acquiring image information of the current scene through a camera installed at the intelligent excavator, and determining the calibration point image and the location information of the calibration point in the image information, includes:

[0012] By using cameras installed on the intelligent excavator, image information of the excavator's working mechanism at different working positions in the current scene is obtained, and the calibration point images and position information of the calibration points in the image information of the excavator's working mechanism at different working positions are determined.

[0013] As an optional implementation, in the first aspect of the present invention, the step of acquiring image information of the excavator's working mechanism at different working positions in the current scene using a camera installed at the intelligent excavator, and determining the calibration point image and the location information of the calibration point in the image information of the excavator's working mechanism at different working positions, includes:

[0014] The excavator's working mechanism is controlled according to a pre-stored movement trajectory path so that the excavator's working mechanism is in different working states;

[0015] The system uses cameras installed on the intelligent excavator to acquire image information of the excavator's working mechanism at different working positions in the current scene.

[0016] The target detection algorithm is used to identify the image information of the excavator working mechanism in different working positions to determine the calibration point image in the corresponding image information. The two-dimensional coordinate information of the calibration point image is determined according to the center point position of the calibration point image, and the two-dimensional coordinate information is used as the position information of the calibration point; wherein, the calibration point is the end point of the stick.

[0017] As an optional implementation, in the first aspect of the present invention, determining the extrinsic parameter matrix of the camera installed at the intelligent excavator based on the three-dimensional coordinate information of the calibration point in the gravity coordinate system, the position information of the calibration point, and the coordinate transformation algorithm includes:

[0018] The three-dimensional spatial coordinates of the corresponding calibration points of the working mechanism in different working states and the two-dimensional pixel coordinates of the above calibration points on the image plane are obtained, and the one-to-one correspondence between the three-dimensional spatial coordinates and the two-dimensional pixel coordinates is determined.

[0019] Obtain the intrinsic parameter matrix of the camera, which includes focal length information and principal point coordinates;

[0020] Initialize the camera's rotation matrix and translation vector; based on the currently estimated rotation matrix, translation vector, and camera intrinsic parameter matrix, project the three-dimensional spatial coordinates onto the corresponding image plane to obtain the reprojection points;

[0021] The sum of squared reprojection errors of all three-dimensional spatial coordinates is used as the objective function, and an optimization algorithm is used to iteratively update the rotation matrix and translation vector, so that the objective function gradually decreases.

[0022] When the convergence condition is met, the iteration stops. The rotation matrix and translation vector obtained at this time are the rotation and translation relationship of the camera relative to the world coordinate system. The convergence condition includes the reprojection error being less than a certain preset value or the number of iterations reaching the upper limit.

[0023] As an optional implementation, in the first aspect of the present invention, after determining the extrinsic parameter matrix of the camera installed at the intelligent excavator, the method further includes:

[0024] Receive the coordinates of ground points measured by the RTK pole and generate the corresponding gravity lines;

[0025] The system acquires ground point images captured by the camera and calculates the three-dimensional coordinates of the corresponding ground points based on the extrinsic parameter matrix.

[0026] The corresponding Z-axis information is determined based on the three-dimensional coordinates, and the Z-axis is compared with the gravity line generated by the RTK rod to determine the deviation between the two. If the deviation is within the set range, the external parameter matrix is ​​stored.

[0027] As an optional implementation, in a first aspect of the present invention, the attitude detection module includes a tilt sensor.

[0028] As an optional implementation, in a first aspect of the present invention, the second attitude detection module is disposed in the middle of the excavator boom, and the third attitude detection module is disposed in the middle of the excavator stick.

[0029] A second aspect of this invention discloses a camera extrinsic parameter calibration system incorporating an attitude detection module, comprising:

[0030] Image acquisition module: used to acquire image information of the current scene through a camera set at the intelligent excavator, and to determine the calibration point image and the location information of the calibration point in the image information;

[0031] First acquisition module: used to acquire attitude angle information detected by the attitude detection module group set at the intelligent excavator; the attitude detection module group includes a first attitude detection module, a second attitude detection module and a third attitude detection module, wherein the first attitude detection module is set at the vehicle base, the second attitude detection module is set at the excavator boom, and the third attitude detection module is set at the excavator stick.

[0032] The second acquisition module is used to acquire vehicle information of the intelligent excavator to be calibrated. The vehicle information includes the boom length and stick length of the intelligent excavator. The first vector is determined based on the attitude angle information detected by the second attitude detection module and the boom length. The second vector is determined based on the attitude angle information detected by the third attitude detection module and the stick length.

[0033] The determination module is used to determine a third vector based on the first vector and the second vector, wherein the third vector is used to characterize the vector from the boom start point to the stick end point; and to determine the three-dimensional coordinate information of the calibration point in the gravity coordinate system based on the attitude angle information of the first attitude detection module and the third vector, wherein the boom start point in the gravity coordinate system is the origin of the gravity coordinate system.

[0034] The extrinsic parameter calculation module is used to determine the extrinsic parameter matrix of the camera set at the intelligent excavator based on the three-dimensional coordinate information of the calibration point in the gravity coordinate system, the position information of the calibration point, and the coordinate transformation algorithm, and to store the extrinsic parameter matrix.

[0035] A third aspect of the present invention discloses an electronic device, comprising: a memory storing executable program code; a processor coupled to the memory; the processor calling the executable program code stored in the memory to execute the camera extrinsic parameter calibration method incorporating an attitude detection module disclosed in the first aspect of the present invention.

[0036] A fourth aspect of the present invention discloses a computer-readable storage medium storing a computer program, wherein the computer program causes a computer to execute the camera extrinsic parameter calibration method incorporating an attitude detection module disclosed in the first aspect of the present invention.

[0037] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0038] The camera extrinsic parameter calibration method in this embodiment of the invention, which incorporates an attitude detection module, acquires attitude angle information through the attitude detection module group and combines it with the vehicle information of the intelligent excavator to accurately determine various vectors, thereby precisely calculating the three-dimensional coordinate information of the calibration point in the gravity coordinate system. Based on this precise three-dimensional coordinate information and the position information of the calibration point in the image, the camera extrinsic parameter matrix determined by the coordinate transformation algorithm has high accuracy, which helps to improve the accuracy of subsequent applications based on camera images. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart illustrating the camera extrinsic parameter calibration method incorporating an attitude detection module disclosed in an embodiment of the present invention.

[0041] Figure 2 This is a schematic diagram of the process for acquiring images of different calibration points disclosed in an embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram of the process for obtaining the extrinsic parameter matrix disclosed in an embodiment of the present invention;

[0043] Figure 4 This is a schematic diagram of the calibration and verification process disclosed in an embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram of the sensor installation position disclosed in an embodiment of the present invention;

[0045] Figure 6 This is a schematic diagram of a calibration-free system disclosed in an embodiment of the present invention;

[0046] Figure 7 This is a schematic diagram of calibration points disclosed in an embodiment of the present invention;

[0047] Figure 8 This is a schematic diagram of a camera extrinsic parameter calibration system combined with an attitude detection module provided in an embodiment of the present invention;

[0048] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] It should be noted that the terms "first," "second," "third," "fourth," etc., in the specification and claims of this invention are used to distinguish different objects, not to describe a specific order. The terms "comprising" and "having," and any variations thereof, in the embodiments of this invention are intended to cover non-exclusive inclusion. Exemplarily, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0051] Current camera extrinsic calibration methods have the following drawbacks: First, they generally rely on tools such as calibration boards, RTK poles, QR codes, or drones, making the calibration process complex. Second, they require precise measurement of the distance between the camera and the calibration board. Third, this complex data acquisition process must be repeated for each new intelligent excavator installed, resulting in low efficiency and hindering automation. Fourth, on-site data acquisition and calibration require measuring the distance between the camera and the calibration board, necessitating on-site data collection and calibration. This invention discloses a camera extrinsic calibration method, system, electronic device, and storage medium incorporating an attitude detection module. By acquiring attitude angle information through the attitude detection module and combining it with vehicle information from the intelligent excavator, it can accurately determine various vectors and precisely calculate the three-dimensional coordinates of the calibration point in the gravity coordinate system. Based on this precise three-dimensional coordinate information and the position information of the calibration point in the image, the camera extrinsic matrix determined using a coordinate transformation algorithm has high accuracy, contributing to improved accuracy in subsequent applications based on camera images.

[0052] Example 1

[0053] Please see Figure 1 , Figure 1This is a flowchart illustrating a camera extrinsic parameter calibration method incorporating an attitude detection module, as disclosed in an embodiment of the present invention. The execution entity of the method described in this embodiment is an execution entity composed of software and / or hardware. This execution entity can receive relevant information via wired or / or wireless means and can send certain instructions. It may also have certain processing and storage functions. This execution entity can control multiple devices, such as remote physical servers or cloud servers and related software, or local hosts or servers and related software that perform related operations on devices located in a certain location. In some scenarios, multiple storage devices can also be controlled; these storage devices may be placed in the same location as the device or in different locations. Figures 1-7 As shown, the camera extrinsic parameter calibration method based on the attitude detection module includes the following steps:

[0054] S101: Obtain image information of the current scene through a camera installed at the intelligent excavator, and determine the calibration point image and the location information of the calibration point in the image information;

[0055] S102: Obtain the attitude angle information detected by the attitude detection module group set at the intelligent excavator; the attitude detection module group includes a first attitude detection module, a second attitude detection module and a third attitude detection module, wherein the first attitude detection module is set at the vehicle base, the second attitude detection module is set at the excavator boom, and the third attitude detection module is set at the excavator stick.

[0056] S103: Obtain the vehicle information of the intelligent excavator to be calibrated, the vehicle information including the boom length and stick length of the intelligent excavator, and determine the first vector based on the attitude angle information detected by the second attitude detection module and the boom length, and determine the second vector based on the attitude angle information detected by the third attitude detection module and the stick length.

[0057] S104: Determine a third vector based on the first vector and the second vector, the third vector being used to characterize the vector from the boom start point to the stick end point; determine the three-dimensional coordinate information of the calibration point in the gravity coordinate system based on the attitude angle information of the first attitude detection module and the third vector, wherein the boom start point in the gravity coordinate system is the origin of the gravity coordinate system;

[0058] S105: Determine the extrinsic parameter matrix of the camera set at the intelligent excavator based on the three-dimensional coordinate information of the calibration point in the gravity coordinate system, the position information of the calibration point, and the coordinate transformation algorithm, and store the extrinsic parameter matrix.

[0059] Specifically, in this embodiment of the invention, three tilt sensors are installed on the excavator, namely 0, 1, and 2 in the figure (inc0, inc1, and inc2 respectively), as shown below. Figure 5 As shown, the tilt sensor only measures angles, and the angle is combined with the length to calculate the corresponding vector. Since the lengths of AB and BC are fixed for each excavator model, the calculation method for the 3D coordinates of point A using the tilt sensor is as follows: Using point C as the origin, calculate the coordinates (x, y, z) of point A, and then transform them to the gravity coordinate system of inc0. The calculation process is as follows: Based on the attitude angle of inc1 and the length of CB, calculate the vector CB; based on the attitude angle of inc2 and the length of BA, calculate the vector BA; based on vectors CB and BA, calculate the vector CA; based on the attitude angle of inc0, transform CA to the gravity coordinate system of inc0 to obtain the coordinates of point A in the gravity coordinate system of inc0; thus, the three-dimensional coordinates of the calibration point can be determined.

[0060] The intelligent excavator of this invention typically operates in complex and ever-changing construction sites. This method utilizes an attitude detection module to acquire the attitude angle information of various parts of the excavator in real time, enabling it to adapt to the camera extrinsic parameter calibration requirements under different attitudes and positions of the excavator. Even when the excavator is in complex attitudes such as tilting or rotating, the three-dimensional coordinates of the calibration point can be accurately calculated using the attitude angle information, ensuring the reliability of the camera extrinsic parameter calibration and enabling the camera to accurately establish a mapping relationship with the actual spatial position under various working conditions.

[0061] This method can acquire attitude angle and image information in real time and quickly calculate the camera extrinsic parameter matrix. During the actual operation of the intelligent excavator, the camera extrinsic parameters can be updated and calibrated promptly according to changes in the working state, meeting real-time requirements. For example, when the excavator performs different digging actions, the camera extrinsic parameters can be quickly adjusted to ensure that the camera can always accurately capture images of the target area and provide accurate spatial information. The determined camera extrinsic parameter matrix is ​​stored for convenient subsequent use. If the camera needs to be used again for related operations in different working stages or scenarios of the excavator, the stored extrinsic parameter matrix can be directly called without the need for a complex recalibration process, saving time and computing resources. Simultaneously, the stored extrinsic parameter matrix data can also be used to evaluate and optimize the camera system, further improving its performance.

[0062] The modular design of the attitude detection module group gives the system good maintainability and scalability. If a particular attitude detection module fails, it can be easily replaced and debugged. Furthermore, the number and location of attitude detection modules can be increased or adjusted according to actual needs to adapt to different models or working requirements of intelligent excavators, improving the system's versatility and flexibility.

[0063] The precise camera extrinsic calibration of this invention can provide more accurate visual information for intelligent excavators, helping operators better understand the excavator's working status and surrounding environment, thereby improving operational safety and efficiency. For example, during excavation and loading operations, the camera can accurately identify the position and shape of the target object, helping operators to control the excavator more precisely, reducing misoperation and accidents, and improving work efficiency.

[0064] More preferably, the step of acquiring image information of the current scene through a camera installed at the intelligent excavator, and determining the calibration point image and the location information of the calibration point in the image information, includes:

[0065] By using cameras installed on the intelligent excavator, image information of the excavator's working mechanism at different working positions in the current scene is obtained, and the calibration point images and position information of the calibration points in the image information of the excavator's working mechanism at different working positions are determined.

[0066] When the excavator's working mechanism is in different working positions according to this embodiment of the invention, the camera's viewing angle and attitude will change. By acquiring image information from multiple different positions and determining calibration point information, the camera's situation under various possible working states can be more comprehensively covered. Calculating the camera's extrinsic parameter matrix based on this diverse data allows the calibration results to more accurately reflect the relationship between the camera and the actual space under different working positions, improving the accuracy and reliability of extrinsic parameter calibration and reducing errors caused by changes in working position.

[0067] Accurate extrinsic parameter calibration is fundamental to achieving precise operation planning and control of intelligent excavators. Acquiring image information from different working positions and determining the location of calibration points helps to more accurately determine the relative positional relationship between the excavator's working mechanism and its surrounding environment. For example, when planning excavation paths, accurate camera extrinsic parameters at different working positions allow for more precise control of the excavator's movements, avoiding collisions and misoperations, and improving work efficiency and quality.

[0068] Specifically, the excavator moves the boom, arm, and bucket, for example, up, down, left, and right, acquiring 6-10 sets of images and corresponding angles inc0, inc1, and inc2. For example, 6-10 sets of images at any given position are collected, and the coordinates of point A in the gravity coordinate system at inc0 are calculated using the method described above. Point A is then labeled on the image, obtaining its coordinates (x, y) in the image.

[0069] More preferably, such as Figure 2As shown, the step of acquiring image information of the excavator's working mechanism at different working positions in the current scene through a camera set at the intelligent excavator, and determining the calibration point image and the location information of the calibration point in the image information of the excavator's working mechanism at different working positions, includes:

[0070] S1011: Control the excavator working mechanism according to the pre-stored movement trajectory path so that the excavator working mechanism is in different working states;

[0071] S1012: Obtain image information of the excavator's working mechanism at different working positions in the current scene by using a camera installed at the intelligent excavator;

[0072] S1013: Based on the target detection algorithm, identify the image information of the excavator working mechanism at different working positions to determine the calibration point image in the corresponding image information, determine its two-dimensional coordinate information based on the center point position of the calibration point image, and use the two-dimensional coordinate information as the position information of the calibration point; wherein, the calibration point is the end point of the stick.

[0073] The solution in this invention controls the excavator's working mechanism based on a pre-stored movement trajectory path, enabling the excavator's working mechanism to change its working position in a predetermined and scientifically planned manner. Compared to random movement, this method ensures that the working mechanism traverses various possible working states, thereby guaranteeing that the image information captured by the camera is comprehensive and representative. For example, it can cover different dimensions of working scenarios such as excavation depth, angle, and extension range, providing a richer and more complete data foundation for subsequent external parameter calibration.

[0074] The working mechanism moves along a preset trajectory, ensuring a consistent motion pattern each time data is collected. This helps reduce image data discrepancies caused by the randomness of the mechanism's movement, improving the consistency and stability of data acquisition. This makes subsequent analysis and calculations based on this data more reliable and reduces the generation of errors.

[0075] The system explicitly acquires image information from different working positions via cameras, and these positions are key locations determined by a preset trajectory. This allows the cameras to accurately capture images of the excavator's working mechanism in various important states, avoiding interference from irrelevant images and improving the quality and relevance of the image information. For example, at certain digging angles or extension lengths, relevant images can be acquired promptly and accurately, providing clear and effective data for subsequent calibration point identification.

[0076] Target detection algorithms are used to identify image information and determine calibration point images. These algorithms possess high accuracy and robustness, capable of accurately identifying calibration points (the end point of the boom) in complex image backgrounds. Even under adverse conditions such as changing lighting and occlusion, they can reliably locate calibration points, significantly improving the accuracy of calibration point identification. Using the center point of the calibration point image as the two-dimensional coordinate information of the calibration point is a simple and intuitive positioning method, reducing computational complexity. Furthermore, the center point typically represents the actual location of the calibration point well, improving the efficiency of determining calibration point location information while maintaining a certain level of accuracy. This is particularly important for intelligent excavator applications requiring rapid external parameter calibration, providing accurate location information for subsequent operation control and planning. This method enables rapid automated annotation. Specifically, as shown... Figure 6 and Figure 7 As shown. In actual implementation, the calibration point here can be other location points.

[0077] More preferably, such as Figure 2 As shown, the step of determining the extrinsic parameter matrix of the camera installed at the intelligent excavator based on the three-dimensional coordinate information of the calibration point in the gravity coordinate system, the position information of the calibration point, and the coordinate transformation algorithm includes:

[0078] S1051: Obtain the three-dimensional spatial coordinates of the corresponding calibration points of the working mechanism in different working states and the two-dimensional pixel coordinates of the above calibration points on the image plane, and determine that the three-dimensional spatial coordinates and the two-dimensional pixel coordinates correspond one-to-one.

[0079] S1052: Obtain the intrinsic parameter matrix of the camera, the intrinsic parameter matrix including focal length information and principal point coordinates;

[0080] S1053: Initialize the camera's rotation matrix and translation vector; based on the currently estimated rotation matrix, translation vector, and camera intrinsic parameter matrix, project the three-dimensional spatial coordinates onto the corresponding image plane to obtain the reprojection point;

[0081] S1054: Take the sum of squared reprojection errors of all three-dimensional spatial coordinates as the objective function, and use an optimization algorithm to iteratively update the rotation matrix and translation vector, so that the objective function gradually decreases;

[0082] S1055: When the convergence condition is met, stop the iteration. The rotation matrix and translation vector obtained at this time are the rotation and translation relationship of the camera relative to the world coordinate system. The convergence condition includes the reprojection error being less than a certain preset value or the number of iterations reaching the upper limit.

[0083] This invention explicitly obtains the three-dimensional spatial coordinates of the calibration points and their corresponding two-dimensional pixel coordinates on the image plane under different working states of the working mechanism, ensuring a one-to-one correspondence between the two. This strict data correspondence is the foundation for subsequent calculations, avoiding deviations in the calculation of the extrinsic parameter matrix due to data matching errors. Accurate data correspondence enables camera extrinsic parameter calibration to be based on reliable data, thereby improving the accuracy and reliability of the extrinsic parameter matrix and providing an accurate spatial mapping relationship for the vision system of intelligent excavators.

[0084] Obtaining the camera's intrinsic parameter matrix (including focal length information and principal point coordinates) fully considers the camera's inherent properties. The intrinsic parameter matrix is ​​a crucial parameter in the camera's imaging process, and different cameras have different intrinsic parameter matrices. Incorporating the intrinsic parameter matrix when calculating the extrinsic parameter matrix allows for a more accurate description of the camera's imaging process, making the projection calculation from 3D spatial coordinates to 2D pixel coordinates more precise. This, in turn, improves the accuracy of the extrinsic parameter matrix calculation, enabling the camera to more accurately reflect the spatial information of the actual scene. The camera's rotation matrix and translation vector are initialized, and the 3D spatial coordinates are projected onto the image plane based on the currently estimated parameters and the intrinsic parameter matrix to obtain reprojection points. Proper initialization provides a good starting point for subsequent iterative optimization, reducing the number of iterations and computational load. Accurate projection calculation yields reprojection points that can be compared with the actual observation points, providing data support for calculating reprojection errors, providing optimization direction for the algorithm, and facilitating the rapid finding of accurate extrinsic parameter matrices.

[0085] This invention uses the sum of squared reprojection errors of all three-dimensional spatial coordinates as the objective function, and employs an optimization algorithm to iteratively update the rotation matrix and translation vector to reduce the objective function. This optimization algorithm-based method continuously adjusts the rotation matrix and translation vector, gradually reducing the error between the projected point and the actual observation point. Through multiple iterations, the optimal extrinsic parameter matrix can be gradually approximated, improving its accuracy. This allows the camera to more accurately establish a correspondence with the actual space under different operating conditions, meeting the high-precision visual information requirements of intelligent excavators. A convergence condition is set (reprojection error less than a preset value or the number of iterations reaches an upper limit). Iteration stops when these conditions are met. This setting ensures the convergence and stability of the algorithm, preventing it from getting stuck in infinite iteration. By reasonably judging the convergence condition, the computation time and resource consumption can be controlled while ensuring the accuracy of the extrinsic parameter matrix. Simultaneously, stopping iteration when the reprojection error is less than a preset value ensures that the obtained extrinsic parameter matrix meets certain accuracy requirements, providing a reliable foundation for subsequent applications of intelligent excavators.

[0086] More preferably, such as Figure 4 As shown, after determining the extrinsic parameter matrix of the camera installed at the intelligent excavator, the method further includes:

[0087] S106: Receives the coordinates of ground points measured by the RTK rod and generates the corresponding gravity lines;

[0088] S107: Acquire ground point images captured by the camera, and calculate the three-dimensional coordinates of the corresponding ground points based on the extrinsic parameter matrix;

[0089] S108: Determine the corresponding Z-axis information based on the three-dimensional coordinates, and compare the Z-axis with the gravity line generated by the RTK rod to determine the deviation between the two. If the deviation is within the set range, store the external parameter matrix.

[0090] This invention, in its embodiments, receives the coordinates of ground points measured by an RTK (Real-Time Kinematic) probe and generates corresponding gravity lines. RTK technology provides high-precision position information, and the measured ground point coordinates are highly accurate. The gravity lines generated by the RTK probe provide a high-precision reference for subsequent verification of the extrinsic parameter matrix. Using this high-precision reference, the accuracy of the camera's extrinsic parameter matrix can be evaluated more accurately, improving the reliability of the extrinsic parameter matrix.

[0091] The solution in this invention determines the corresponding Z-axis information based on the calculated three-dimensional coordinates and compares it with the gravity line generated by the RTK rod to determine the deviation between the two. The Z-axis information reflects the position information of the ground point in the vertical direction, and comparing it with the gravity line can intuitively reflect the accuracy of the extrinsic parameter matrix in the vertical direction. By accurately calculating the deviation, the error of the extrinsic parameter matrix in describing spatial positional relationships can be accurately evaluated, providing a quantitative basis for judging whether the extrinsic parameter matrix meets the requirements.

[0092] More preferably, the attitude detection module includes a tilt sensor.

[0093] The intelligent excavators of this invention typically operate in harsh outdoor environments, such as dusty conditions, strong vibrations, and large temperature fluctuations. Tilt sensors generally possess high anti-interference capabilities and stability, enabling reliable operation in such harsh environments. They are less susceptible to external environmental factors, continuously providing accurate attitude angle information, ensuring the system operates normally even in complex environments, and providing stable support for the intelligent operation of the excavator. Compared to some complex attitude detection devices, tilt sensors are relatively inexpensive. Using tilt sensors as an attitude detection module, while meeting the attitude detection requirements of intelligent excavators, can reduce the overall system cost and improve the system's cost-effectiveness. Furthermore, the installation and maintenance of tilt sensors are relatively simple, requiring no complex equipment or technology, further reducing operating costs and maintenance difficulty. Tilt sensors are modular, facilitating integration with other systems of the intelligent excavator. They can be easily installed in different parts of the excavator, such as the chassis, boom, and stick, forming a complete attitude detection module group. This modular design makes the system structure clearer, facilitating system construction, debugging, and upgrades, and improving the system's scalability and flexibility.

[0094] More preferably, the second attitude detection module is located in the middle of the excavator boom, and the third attitude detection module is located in the middle of the excavator stick.

[0095] In this invention, the boom and stick are the most complex and critical parts of the intelligent excavator during operation, and their attitude changes are crucial to the excavator's working performance and position determination. Placing the attitude detection module in the middle of the boom and stick allows for more accurate measurement of the actual attitude of these two critical parts. Because the middle position is more representative of the overall movement of the components, it avoids the influence of additional interference (such as local deformation caused by changes in end load) that might occur when installed at the ends, thus obtaining more accurate attitude angle information and providing more reliable data support for subsequent calculations of the 3D coordinates of calibration points and the camera extrinsic parameter matrix. The middle of the boom and stick typically has relatively spacious installation space, facilitating the installation and fixation of the attitude detection module. Simultaneously, sensor maintenance and repair are also more convenient in these locations, allowing maintenance personnel easier access to the sensors for inspection, calibration, or replacement, reducing maintenance costs and difficulty, and improving system maintainability.

[0096] The camera extrinsic parameter calibration method in this embodiment of the invention, which incorporates an attitude detection module, acquires attitude angle information through the attitude detection module group and combines it with the vehicle information of the intelligent excavator to accurately determine various vectors, thereby precisely calculating the three-dimensional coordinate information of the calibration point in the gravity coordinate system. Based on this precise three-dimensional coordinate information and the position information of the calibration point in the image, the camera extrinsic parameter matrix determined by the coordinate transformation algorithm has high accuracy, which helps to improve the accuracy of subsequent applications based on camera images.

[0097] Example 2

[0098] Please see Figure 8 , Figure 8 This is a schematic diagram of the camera extrinsic parameter calibration system incorporating an attitude detection module, as disclosed in an embodiment of the present invention. Figure 8 As shown, the camera extrinsic parameter calibration system combined with the attitude detection module may include:

[0099] Image acquisition module 21: used to acquire image information of the current scene through a camera set at the intelligent excavator, and determine the calibration point image and the location information of the calibration point in the image information;

[0100] First acquisition module 22: used to acquire attitude angle information detected by the attitude detection module group set at the intelligent excavator; the attitude detection module group includes a first attitude detection module, a second attitude detection module and a third attitude detection module, wherein the first attitude detection module is set at the vehicle base, the second attitude detection module is set at the excavator boom, and the third attitude detection module is set at the excavator stick.

[0101] The second acquisition module 23 is used to acquire vehicle information of the intelligent excavator to be calibrated. The vehicle information includes the boom length and stick length of the intelligent excavator. The first vector is determined based on the attitude angle information detected by the second attitude detection module and the boom length. The second vector is determined based on the attitude angle information detected by the third attitude detection module and the stick length.

[0102] Determining module 24: is used to determine a third vector based on the first vector and the second vector, the third vector being used to characterize the vector from the boom starting point to the stick end point; and to determine the three-dimensional coordinate information of the calibration point in the gravity coordinate system based on the attitude angle information of the first attitude detection module and the third vector, wherein the boom starting point in the gravity coordinate system is the origin of the gravity coordinate system.

[0103] External parameter calculation module 25: used to determine the external parameter matrix of the camera set at the intelligent excavator based on the three-dimensional coordinate information of the calibration point in the gravity coordinate system, the position information of the calibration point and the coordinate transformation algorithm, and to store the external parameter matrix.

[0104] The camera extrinsic parameter calibration method in this embodiment of the invention, which incorporates an attitude detection module, acquires attitude angle information through the attitude detection module group and combines it with the vehicle information of the intelligent excavator to accurately determine various vectors, thereby precisely calculating the three-dimensional coordinate information of the calibration point in the gravity coordinate system. Based on this precise three-dimensional coordinate information and the position information of the calibration point in the image, the camera extrinsic parameter matrix determined by the coordinate transformation algorithm has high accuracy, which helps to improve the accuracy of subsequent applications based on camera images.

[0105] Example 3

[0106] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present invention. The electronic device can be a computer, a server, etc. Of course, in certain cases, it can also be a mobile phone, tablet computer, monitoring terminal, or other smart device, as well as an image acquisition device with processing capabilities. Figure 9 As shown, the electronic device may include:

[0107] Memory 510 storing executable program code;

[0108] Processor 520 coupled to memory 510;

[0109] The processor 520 calls the executable program code stored in the memory 510 to execute some or all of the steps in the camera extrinsic parameter calibration method combined with the attitude detection module in Embodiment 1.

[0110] This invention discloses a computer-readable storage medium storing a computer program that causes a computer to perform some or all of the steps in the camera extrinsic parameter calibration method incorporating an attitude detection module as described in Embodiment 1.

[0111] This invention also discloses a computer program product, wherein when the computer program product is run on a computer, the computer performs some or all of the steps in the camera extrinsic parameter calibration method incorporating the attitude detection module in Embodiment 1.

[0112] This invention also discloses an application publishing platform, which is used to publish computer program products. When the computer program products are run on a computer, the computer executes some or all of the steps in the camera extrinsic parameter calibration method in conjunction with the attitude detection module in Embodiment 1.

[0113] In various embodiments of the present invention, it should be understood that the sequence number of each process does not necessarily imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0114] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they can be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0115] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0116] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-accessible memory. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several requests to cause a computer device (which can be a personal computer, server, or network device, specifically a processor in the computer device) to execute some or all of the steps of the methods described in the various embodiments of the present invention.

[0117] In the embodiments provided by this invention, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0118] Those skilled in the art will understand that some or all of the steps in the various methods of the embodiments described can be implemented by a program instructing related hardware. This program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0119] The foregoing has provided a detailed description of the camera extrinsic parameter calibration method, system, electronic device, and storage medium incorporating an attitude detection module disclosed in the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for calibrating camera extrinsic parameters in conjunction with an attitude detection module, characterized in that, include: The system uses a camera installed on the intelligent excavator to acquire image information of the current scene and to determine the calibration point image and the location information of the calibration point in the image information. The calibration point is the end point of the bucket pole; The attitude angle information detected by the attitude detection module group set at the intelligent excavator is obtained; the attitude detection module group includes a first attitude detection module, a second attitude detection module and a third attitude detection module, wherein the first attitude detection module is set at the vehicle base, the second attitude detection module is set at the excavator boom and the third attitude detection module is set at the excavator stick. Obtain vehicle information of the intelligent excavator to be calibrated, including the boom length and stick length of the intelligent excavator, and determine a first vector based on the attitude angle information detected by the second attitude detection module and the boom length, and determine a second vector based on the attitude angle information detected by the third attitude detection module and the stick length. A third vector is determined based on the first vector and the second vector, and the third vector is used to characterize the vector from the boom start point to the stick end point; the three-dimensional coordinate information of the calibration point in the gravity coordinate system is determined based on the attitude angle information of the first attitude detection module and the third vector, wherein the boom start point in the gravity coordinate system is the origin of the gravity coordinate system; The extrinsic parameter matrix of the camera installed at the intelligent excavator is determined based on the three-dimensional coordinate information of the calibration point in the gravity coordinate system, the position information of the calibration point, and a coordinate transformation algorithm, and the extrinsic parameter matrix is ​​stored. The determination of the extrinsic parameter matrix of the camera installed at the intelligent excavator based on the three-dimensional coordinate information of the calibration point in the gravity coordinate system, the position information of the calibration point, and a coordinate transformation algorithm includes: The three-dimensional spatial coordinates of the corresponding calibration points of the working mechanism in different working states and the two-dimensional pixel coordinates of the above calibration points on the image plane are obtained, and the one-to-one correspondence between the three-dimensional spatial coordinates and the two-dimensional pixel coordinates is determined. Obtain the intrinsic parameter matrix of the camera, which includes focal length information and principal point coordinates; Initialize the camera's rotation matrix and translation vector; based on the currently estimated rotation matrix, translation vector, and camera intrinsic parameter matrix, project the three-dimensional spatial coordinates onto the corresponding image plane to obtain the reprojection points; The sum of squared reprojection errors of all three-dimensional spatial coordinates is used as the objective function, and an optimization algorithm is used to iteratively update the rotation matrix and translation vector, so that the objective function gradually decreases. When the convergence condition is met, the iteration stops. The rotation matrix and translation vector obtained at this time are the rotation and translation relationship of the camera relative to the world coordinate system. The convergence condition includes the reprojection error being less than a preset value or the number of iterations reaching the upper limit.

2. The camera extrinsic parameter calibration method incorporating an attitude detection module as described in claim 1, characterized in that, The step of acquiring image information of the current scene through a camera installed at the intelligent excavator, and determining the calibration point image and the location information of the calibration point in the image information, includes: By using cameras installed on the intelligent excavator, image information of the excavator's working mechanism at different working positions in the current scene is obtained, and the calibration point images and position information of the calibration points in the image information of the excavator's working mechanism at different working positions are determined.

3. The camera extrinsic parameter calibration method incorporating an attitude detection module as described in claim 2, characterized in that, The method of acquiring image information of the excavator's working mechanism at different working positions in the current scene through a camera installed at the intelligent excavator, and determining the calibration point image and the location information of the calibration point in the image information of the excavator's working mechanism at different working positions, includes: The excavator's working mechanism is controlled according to a pre-stored movement trajectory path so that the excavator's working mechanism is in different working states; The system uses cameras installed on the intelligent excavator to acquire image information of the excavator's working mechanism at different working positions in the current scene. The target detection algorithm is used to identify the image information of the excavator working mechanism in different working positions to determine the calibration point image in the corresponding image information. The two-dimensional coordinate information of the calibration point image is determined according to the center point position of the calibration point image, and the two-dimensional coordinate information is used as the position information of the calibration point; wherein, the calibration point is the end point of the stick.

4. The camera extrinsic parameter calibration method incorporating an attitude detection module as described in claim 1, characterized in that, After determining the extrinsic parameter matrix of the cameras installed at the intelligent excavator, the method further includes: Receive the coordinates of ground points measured by the RTK pole and generate the corresponding gravity lines; The system acquires ground point images captured by the camera and calculates the three-dimensional coordinates of the corresponding ground points based on the extrinsic parameter matrix. The corresponding Z-axis information is determined based on the three-dimensional coordinates, and the Z-axis is compared with the gravity line generated by the RTK rod to determine the deviation between the two. If the deviation is within the set range, the external parameter matrix is ​​stored.

5. The camera extrinsic parameter calibration method incorporating an attitude detection module as described in claim 1, characterized in that, The attitude detection module includes a tilt sensor.

6. The camera extrinsic parameter calibration method incorporating an attitude detection module as described in claim 1, characterized in that, The second attitude detection module is located in the middle of the excavator boom, and the third attitude detection module is located in the middle of the excavator stick.

7. A camera extrinsic parameter calibration system incorporating an attitude detection module, characterized in that, include: Image acquisition module: used to acquire image information of the current scene through a camera set at the intelligent excavator, and to determine the calibration point image and the location information of the calibration point in the image information; The calibration point is the end point of the bucket pole; First acquisition module: used to acquire attitude angle information detected by the attitude detection module group set at the intelligent excavator; the attitude detection module group includes a first attitude detection module, a second attitude detection module and a third attitude detection module, wherein the first attitude detection module is set at the vehicle base, the second attitude detection module is set at the excavator boom, and the third attitude detection module is set at the excavator stick. The second acquisition module is used to acquire vehicle information of the intelligent excavator to be calibrated. The vehicle information includes the boom length and stick length of the intelligent excavator. The first vector is determined based on the attitude angle information detected by the second attitude detection module and the boom length. The second vector is determined based on the attitude angle information detected by the third attitude detection module and the stick length. The determination module is used to determine a third vector based on the first vector and the second vector, wherein the third vector is used to characterize the vector from the boom start point to the stick end point; and to determine the three-dimensional coordinate information of the calibration point in the gravity coordinate system based on the attitude angle information of the first attitude detection module and the third vector, wherein the boom start point in the gravity coordinate system is the origin of the gravity coordinate system. The extrinsic parameter calculation module is used to determine the extrinsic parameter matrix of the camera installed at the intelligent excavator based on the three-dimensional coordinate information of the calibration point in the gravity coordinate system, the position information of the calibration point, and the coordinate transformation algorithm, and to store the extrinsic parameter matrix. The process of determining the extrinsic parameter matrix of the camera installed at the intelligent excavator based on the three-dimensional coordinate information of the calibration point in the gravity coordinate system, the position information of the calibration point, and the coordinate transformation algorithm includes: The three-dimensional spatial coordinates of the corresponding calibration points of the working mechanism in different working states and the two-dimensional pixel coordinates of the above calibration points on the image plane are obtained, and the one-to-one correspondence between the three-dimensional spatial coordinates and the two-dimensional pixel coordinates is determined. Obtain the intrinsic parameter matrix of the camera, which includes focal length information and principal point coordinates; Initialize the camera's rotation matrix and translation vector; based on the currently estimated rotation matrix, translation vector, and camera intrinsic parameter matrix, project the three-dimensional spatial coordinates onto the corresponding image plane to obtain the reprojection points; The sum of squared reprojection errors of all three-dimensional spatial coordinates is used as the objective function, and an optimization algorithm is used to iteratively update the rotation matrix and translation vector, so that the objective function gradually decreases. When the convergence condition is met, the iteration stops. The rotation matrix and translation vector obtained at this time are the rotation and translation relationship of the camera relative to the world coordinate system. The convergence condition includes the reprojection error being less than a preset value or the number of iterations reaching the upper limit.

8. An electronic device, characterized in that, include: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the camera extrinsic parameter calibration method incorporating the attitude detection module as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program causes a computer to perform the camera extrinsic parameter calibration method incorporating an attitude detection module as described in any one of claims 1 to 6.

Images