Engineering machinery calibration method, program product, storage medium and electronic equipment

Abstract

The invention relates to the technical field of engineering machinery, and provides an engineering machinery calibration method, a program product, a storage medium and electronic equipment. The engineering machinery calibration method comprises the following steps: acquiring a multi-view image and prior calibration information of engineering machinery; performing three-dimensional reconstruction on the engineering machinery based on the multi-view image to obtain a three-dimensional model of the engineering machinery; a target space parameter in the three-dimensional model is measured, the target space parameter is converted into a to-be-calibrated space parameter according to prior calibration information, and the prior calibration information comprises information needed by parameter conversion. According to the method, the calibration efficiency is high, manual measurement errors can be avoided, professional measurement equipment is not needed during calibration, the operation is simple, the requirement for calibration personnel is low, the implementation cost is low, and in addition, the method does not involve continuous measurement of the same parameter during calibration, so that accumulated errors can be reduced, and the calibration precision is improved. In addition, the method supports calibration of various spatial parameters, and is high in practicability.

Description

Technical Field

[0001] The present application relates to the technical field of construction machinery, and more specifically, to a calibration method, program product, storage medium, and electronic device for construction machinery. Background Art

[0002] In recent years, digital construction products (such as sensors, controllers, or systems composed of them) have been increasingly widely used in construction machinery. Taking an excavator as an example, after installing an intelligent guidance system, functions such as the output of the tip coordinates and the automatic control of the excavation depth can be supported.

[0003] After digital construction products are installed on construction machinery, they usually need to be calibrated before they can be put into use. Calibration means determining the parameters required for the operation of digital construction products through certain procedures, such as the distance between certain rotating shafts on an excavator, the installation angle of an inclination sensor, etc. In the prior art, the parameters to be calibrated are usually completed through manual measurement, and the efficiency is relatively low. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a calibration method, program product, storage medium, and electronic device for construction machinery to improve at least some of the above technical problems.

[0005] To achieve the above purpose, the present application provides the following technical solutions:

[0006] In a first aspect, the embodiments of the present application provide a calibration method for construction machinery, including: obtaining multi-view images of the construction machinery and prior calibration information; performing three-dimensional reconstruction on the construction machinery based on the multi-view images to obtain a three-dimensional model of the construction machinery; measuring target spatial parameters in the three-dimensional model, and converting the target spatial parameters into parameters to be calibrated according to the prior calibration information; where the prior calibration information includes the information required for parameter conversion.

[0007] The above method uses multi-view images of construction machinery for three-dimensional reconstruction, measures the target spatial parameters in the three-dimensional model, and then uses the prior calibration information to convert them into parameters to be calibrated. This calibration process does not rely on manual parameter measurement, thus significantly improving the calibration efficiency and avoiding manual measurement errors. In addition, the basic equipment required for this method during calibration is only an image acquisition device (for collecting multi-view images), and no professional measurement equipment is needed. Its operation is simple, the requirements for calibration personnel are low, and the implementation cost is also relatively low. In addition, this method can directly calculate each parameter to be calibrated through the three-dimensional model, without involving continuous measurement of the same type of parameter, so the generation of cumulative errors can be reduced, and the calibration accuracy can be improved. In addition, this method supports the calibration of multiple spatial parameters, and has high practicality.

[0008] In an implementation of the first aspect, the prior calibration information includes a reference length, and the reference length represents the physical length between reference positions on the construction machinery; measuring the target space parameters in the three-dimensional model and converting the target space parameters into the space parameters to be calibrated according to the prior calibration information includes: determining the model reference positions corresponding to the reference positions in the three-dimensional model; measuring the first modeled length between the model reference positions, and calculating the conversion scale between the construction machinery and the three-dimensional model according to the reference length and the first modeled length; determining the model target positions in the three-dimensional model, measuring the second modeled length between the model target positions, and converting the second modeled length into the physical length between the target positions according to the conversion scale; where the target positions are the positions corresponding to the model target positions on the construction machinery, and the physical length between the target positions is the length to be calibrated.

[0009] In the above implementation, the length calibration can be completed by providing a reference length for a certain scale conversion, and the calibration process is very simple. And measuring the modeled length only needs to be calculated according to the three-dimensional model in the computer, without the need for manual measurement, so the calibration efficiency is high and does not depend on professional measurement equipment.

[0010] In an implementation of the first aspect, the multi-view images include a spirit level provided on the construction machinery, and the reference positions are specified points on the spirit level.

[0011] In the above implementation, the reference length is provided by setting a spirit level. According to the content below, the spirit level can also be used to provide a horizontal plane or a horizontal line, that is, with the help of this common tool of the spirit level, the functions of length calibration and angle calibration can be completed simultaneously, covering the most important calibration scenarios of the construction machinery.

[0012] In an implementation of the first aspect, the prior calibration information includes the angle readings of the inclination sensors installed on the target components of the construction machinery; measuring the target space parameters in the three-dimensional model and converting the target space parameters into the space parameters to be calibrated according to the prior calibration information includes: determining the axis of the target component model in the three-dimensional model; where the target component model is the part in the three-dimensional model corresponding to the target component; measuring the angle between the axis and a reference plane or a reference line, and calculating the installation angle of the inclination sensor according to the angle readings and the angle; where the installation angle of the inclination sensor is the angle to be calibrated.

[0013] In the above implementation manner, angle calibration can be completed by providing the angle readings of the tilt sensor and performing a certain angle conversion. The calibration process is very simple. Moreover, calculating the angle between the axis of the measured target component model and the reference plane or reference line only requires calculation based on the 3D model in the computer, without the need for manual measurement. Therefore, the calibration efficiency is high and it does not depend on professional measurement equipment.

[0014] In one implementation manner of the first aspect, the multi-view image includes a spirit level disposed on the construction machinery, and the reference plane or the reference line is a horizontal plane or a horizontal line determined according to the spirit level.

[0015] In the above implementation manner, a spirit level is provided to provide a horizontal plane or a horizontal line. According to the foregoing content, the spirit level can also be used to provide a reference length, that is, by means of this common tool of the spirit level, the functions of angle calibration and length calibration can be completed simultaneously, covering the most important calibration scenarios of construction machinery.

[0016] In one implementation manner of the first aspect, the multi-view image includes two spirit levels disposed on the construction machinery, and the two spirit levels are not parallel to each other. The reference plane is the horizontal plane jointly determined by the two spirit levels.

[0017] In the above implementation manner, determining the horizontal plane by providing two spirit levels helps to improve the angle measurement accuracy and reduce the requirements for the parking state of the construction machinery during calibration.

[0018] In one implementation manner of the first aspect, the spirit level is disposed on the central axis plane of the construction machinery.

[0019] In the above implementation manner, disposing the spirit level on the central axis plane of the construction machinery helps to reduce the systematic error during the calibration process.

[0020] In one implementation manner of the first aspect, the multi-view image includes a label disposed on the surface of the construction machinery. The label includes: a pattern with a specific texture for providing additional texture features during the 3D reconstruction process; and / or, a pattern with a unique identification function for marking a specific position in the construction machinery, and the specific position is mapped to a known coordinate in the 3D model after the 3D reconstruction.

[0021] In the above implementation, there are two possible functions of setting labels on the surface of construction machinery: one is to provide additional texture features (compared with the texture features of the construction machinery itself), which helps to enhance the efficiency and accuracy of feature matching during 3D reconstruction, thereby improving the 3D reconstruction effect; the other is to provide a unique identifier to mark specific positions in the construction machinery (including objects such as spirit levels installed on the construction machinery), so that the coordinates of these positions in the 3D model can also be determined after 3D reconstruction, for automated measurement and improved calibration efficiency.

[0022] In one implementation of the first aspect, the label includes a pattern with a unique identification function; the label is set on the spirit level of the construction machinery to mark a specified point on the spirit level. When performing length calibration, the specified point is the reference position on the construction machinery. When performing angle calibration, the coordinates corresponding to the specified point in the 3D model are used to construct a horizontal plane or a horizontal line; and / or, the label is set on the measurement point on the surface of the construction machinery to mark the measurement point. When performing length calibration, the measurement point is the target position in the construction machinery. When performing angle calibration, the measurement point is the axis end point of the target component in the construction machinery.

[0023] In the above implementation, the positions where labels can be set are given when performing length calibration and angle calibration. By setting labels to mark these positions, it is beneficial to realize automatic measurement during the length calibration and angle calibration processes and improve the calibration efficiency.

[0024] In one implementation of the first aspect, after obtaining the 3D model of the construction machinery and before measuring the target space parameters in the 3D model, the method further includes: evaluating the modeling accuracy of the 3D model and determining that the evaluation result meets the expected standard; or, after obtaining the 3D model of the construction machinery, the method further includes: evaluating the modeling accuracy of the 3D model and determining the confidence level of the space parameters to be calibrated according to the evaluation result.

[0025] In the above implementation, the modeling accuracy of the three-dimensional model can be evaluated. The modeling accuracy evaluation includes at least two possible methods: First, after the modeling is completed and before the target space parameters are measured, if the model accuracy is evaluated to meet the expectations, the subsequent calibration process can be executed; otherwise, the subsequent calibration process is not executed. For example, the images can be re-acquired for modeling, thus avoiding the waste of meaningless computing resources. Second, the evaluation is carried out after the modeling is completed, but the evaluation result of the model accuracy does not affect the execution of the subsequent calibration process (that is, the execution order of the evaluation and the calibration is not limited). According to the evaluation result of the model accuracy, the confidence level of the calibration result, that is, the reliability of the calibration result, can be determined. Thus, corresponding operations can be executed based on the confidence level of the calibration result. For example, the calibration result can be recognized, or the user can be prompted to re-acquire images for modeling, and so on. In short, by evaluating the model quality, the accuracy of the calibration result can be improved, or at least the user or the system (referring to the software that executes the calibration method) can have a correct understanding of the accuracy of the calibration result.

[0026] In one implementation of the first aspect, the evaluation of the modeling accuracy of the three-dimensional model includes: calculating a plurality of confidence level indicators representing the modeling accuracy of the three-dimensional model, performing a weighted calculation on the plurality of confidence level indicators, and using the total confidence level of the obtained three-dimensional model as the evaluation result.

[0027] In the above implementation, the modeling accuracy of the three-dimensional model is evaluated by performing a weighted calculation on a plurality of confidence level indicators. Since various factors are considered, the evaluation result is relatively reliable. Moreover, by setting the weights for the weighted calculation, the proportion of different factors in the modeling accuracy evaluation can be flexibly adjusted, further improving the accuracy of the evaluation result.

[0028] In an implementation of the first aspect, the multiple confidence metrics include at least two of the following metrics: Proportion of calibrated images: the proportion of qualified images that meet the three-dimensional reconstruction requirements in the multi-view images; Total number of connection points: the total number of feature points that match successfully between different qualified images; Total number of observed points: the total number of three-dimensional points calculated based on the connection points; Average number of observations per observed point: the ratio of the total number of connection points to the total number of observed points; Average number of observed points per image: the ratio of the total number of observed points to the total number of qualified images; Trajectory length: the total length of the connection lines of the camera centers corresponding to the qualified images; Average trajectory length: the ratio of the trajectory length to the number of trajectory segments; where each trajectory segment is the connection line of two camera centers; Total reprojection error: the sum of all reprojection errors; where the reprojection error is the distance between the projected point and the connection point used to calculate the observed point when the observed point is projected back onto the image plane; Average reprojection error: the ratio of the total reprojection error to the total number of observed points; Reprojection error distribution: the statistical distribution parameters of all reprojection errors; Prior geometric error: the error between the geometric properties measured from the three-dimensional model and the corresponding actual geometric properties in the construction machinery.

[0029] In the above implementation, multiple specific confidence metrics are given, and these metrics characterize the modeling accuracy of the three-dimensional model from different dimensions. Therefore, by weighting two or more of these metrics, the modeling accuracy can be evaluated more accurately from multiple perspectives.

[0030] In an implementation of the first aspect, the multi-view images include images collected from multiple angles around the construction machinery, or the multi-view images include images collected from multiple angles around half of the construction machinery.

[0031] In the above implementation, the multi-view images can be collected by walking around the construction machinery in a full circle, so as to collect images from more perspectives and improve the modeling accuracy; however, sometimes, one side of the construction machinery is in direct sunlight and the other side is in backlight, and the image quality of the backlight side is poor. Therefore, the multi-view images can also be collected by walking around half of the construction machinery. Although the side of the construction machinery where no images are collected cannot be fully modeled in this way, it usually does not affect normal calibration. For the latter method, in order to improve the modeling accuracy, the construction machinery can be walked around multiple half circles repeatedly for image collection, but the radius of each half circle is different, so as to collect more images available for modeling.

[0032] Second aspect, an embodiment of the present application provides a construction machinery calibration device, including: a data acquisition module, configured to acquire multi-view images of the construction machinery and prior calibration information; a three-dimensional reconstruction module, configured to perform three-dimensional reconstruction on the construction machinery based on the multi-view images to obtain a three-dimensional model of the construction machinery; a calibration module, configured to measure target space parameters in the three-dimensional model and convert the target space parameters into to-be-calibrated space parameters according to the prior calibration information; wherein, the prior calibration information includes information required for parameter conversion.

[0033] Third aspect, an embodiment of the present application provides a computer program product, including computer program instructions, which, when read and executed by a processor, execute the method provided in the first aspect or any possible implementation manner of the first aspect.

[0034] Fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer program instructions are stored, which, when read and executed by a processor, execute the method provided in the first aspect or any possible implementation manner of the first aspect.

[0035] Fifth aspect, an embodiment of the present application provides an electronic device, including: a memory and a processor, wherein computer program instructions are stored in the memory, which, when read and executed by the processor, execute the method provided in the first aspect or any possible implementation manner of the first aspect. Description of the Drawings

[0036] To more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0037] Figure 1 Shows the structure of an excavator provided by an embodiment of the present application;

[0038] Figure 2 Shows the flow of a construction machinery calibration method provided by an embodiment of the present application;

[0039] Figure 3 Shows a way of installing a spirit level on an excavator provided by an embodiment of the present application;

[0040] Figure 4 Shows the pattern of a label provided by an embodiment of the present application;

[0041] Figure 5Illustrates a way of setting tags on an excavator provided by an embodiment of the present application;

[0042] Figure 6 Illustrates the functional modules of a calibration device for construction machinery provided by an embodiment of the present application;

[0043] Figure 7 Illustrates the structure of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0044] Next, the technical solutions in the embodiments of the present application will be described with reference to the accompanying drawings in the embodiments of the present application. It should be noted that: similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0045] The term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the phrase "including a..." does not exclude the presence of additional identical elements in the process, method, article or device including the element.

[0046] The terms "first", "second", etc. are only used to distinguish one entity or operation from another entity or operation, and cannot be construed as indicating or implying relative importance, nor can it be construed as requiring or implying any actual relationship or order between these entities or operations.

[0047] Construction machinery refers to construction mechanical equipment, such as excavators, forklifts, cranes, etc. When introducing the calibration solution of the present application, an excavator is mainly taken as an example.

[0048] Digital construction products refer to a series of products that use digital technologies and tools to improve and optimize the construction process. These products combine modern information technologies with traditional construction machinery to achieve intelligent management and control of the construction process. For example, the intelligent guidance system for excavators is a digital construction product. This system calculates the current state of the excavator (such as position, attitude, tip coordinates, etc.) by installing several sensors and antennas on the excavator, and outputs construction guidance information based on the current state of the excavator.

[0049] After digital construction products are installed on construction machinery, they usually need to be calibrated before they can be put into use. Calibration means determining the parameters required for the operation of digital construction products according to certain procedures. For example, the distance between certain rotating shafts on an excavator, the installation angle of an inclination sensor, etc. Based on these parameters, the intelligent guidance system can calculate the state of the excavator.

[0050] Figure 1 Fig. shows the structure of an excavator provided by an embodiment of the present application. Referring to Figure 1 , the excavator 10 includes main components such as a vehicle body 100, a boom 110, an arm 120, a rocker 130 (also called a dog bone), a bucket 140, etc. Each component is rotationally connected by arranging a rotating shaft 150. An inclination sensor 160 is also installed on the arm 120. Among them, the inclination sensor 160 belongs to a part of the intelligent guidance system. In fact, the actual intelligent guidance system may likely include more than one inclination sensor 160. For example, inclination sensors 160 may also be provided on the rocker 130 and the boom 110, Figure 1 which is just a simplified example. It should be noted that Figure 1 the excavator 10 in [[ ]] is just a typical structure, and the actual excavator may not be exactly the same as this structure. For example, there may also be an excavator with a three-section boom.

[0051] The frequently performed parameter calibrations include two types: length calibration and angle calibration. Of course, it is not excluded to calibrate other parameters. Continuing to refer to Figure 1 , length calibration can refer to determining the distance between two specified points on the excavator through technical means. For example, the distance between rotating shafts A and B. In the prior art, length calibration can be completed by manual measurement using a tape measure, but there are the following disadvantages:

[0052] (1) Usually, it requires the cooperation of more than two people to complete, with low efficiency;

[0053] (2) Manual errors are likely to occur during the measurement process;

[0054] (3) There is an accumulated error when continuously measuring the length;

[0055] (4) It is difficult to verify the correctness of the measurement result or the calibration result.

[0056] Angle calibration can refer to determining the installation angle of the inclination sensor. For example, ideally, the inclination sensor is installed parallel to the axis AB, but in fact, there may be a certain angle with this axis, that is, the installation angle. In the prior art, angle calibration can be completed by manually measuring the pitch angle using a total station and calculating in combination with the reading of the inclination sensor itself, but there are the following disadvantages:

[0057] (1) Usually, it requires the cooperation of more than two people to complete, with low efficiency;

[0058] (2) Professional equipment such as total station is required, with relatively high technical threshold and cost;

[0059] (3) Manual errors are likely to occur during the measurement process;

[0060] (4) Cumulative errors exist when continuously measuring angles;

[0061] (5) It is difficult to verify the correctness of the measurement result or the calibration result.

[0062] The construction machinery calibration method provided by the embodiments of the present application (including its possible implementation manners) performs three-dimensional modeling based on multi-view images of construction machinery, and then completes calibration with the aid of the three-dimensional model, which is beneficial to improving the defects existing in the above-mentioned prior art.

[0063] Figure 2 The flowchart of a construction machinery calibration method provided by the embodiments of the present application is shown. This method can be, but is not limited to, executed by an electronic device. Figure 7 The possible structure of the electronic device is shown, and for details, reference can be made to the following description about Figure 7 . Referring to Figure 2 , the construction machinery calibration method includes:

[0064] Step S210: Obtain multi-view images of construction machinery and prior calibration information.

[0065] The multi-view images of construction machinery refer to multiple images of construction machinery collected from different angles. For example, the user can hold a shooting device (such as a mobile phone), walk around the construction machinery for a week, and press the shooting button when walking to different positions to collect these images. Optionally, during the shooting process, the focal length of the camera can be kept unchanged (such as always being 1x), and the construction machinery can always be located at the center of the picture. These measures will be beneficial to the three-dimensional reconstruction in step S220. More multi-view images can be collected. For example, it can be required to collect at least 30 or 40 images (each image has a different view). On the one hand, more images can provide more redundant information, contribute to adjustment calculation, and reduce the influence of errors; on the other hand, it can also better cover the terrain and ground features, avoiding data loss or inaccurate calculation.

[0066] Optionally, although walking around the construction machinery for a week can collect images with more perspectives, thereby improving the model accuracy of the three-dimensional reconstruction in step S220, sometimes, one side of the construction machinery is in the light and the other side is in the backlight, and the image quality of the backlight side is poor. Therefore, it is also possible to only walk around the construction machinery for half a week (for example, walk for half a week on the side with light) to collect multi-view images. Although in this way, the three-dimensional reconstruction of the side of the construction machinery without collected images cannot be completed completely, it usually does not affect normal calibration. For example, referring to Figure 1, the length or angle to be calibrated can be confirmed even when viewed from only one side of the excavator 10. Further, if one only walks half a circle around the construction machinery, the number of images taken may be relatively small. Therefore, in order to improve the accuracy of the 3D reconstruction model in step S220, one can walk around the construction machinery multiple times for half a circle each for image acquisition, with different radii for each half circle.

[0067] Optionally, the user can also shoot videos instead of images, and the algorithm selects multi-view images from the video frames, such as selecting one frame per second as an image of one view, and so on.

[0068] Optionally, to prevent the user from being unfamiliar with how to acquire multi-view images, a demonstration video can also be pre-produced. Before image acquisition, the user can learn the image acquisition methods and requirements by watching the demonstration video. Alternatively, based on Augmented Reality (AR) technology, prompt information can be generated on the preview screen of the user's shooting device, such as marks of some image acquisition positions. The user can walk to the corresponding positions for image acquisition according to this prompt information, and the guiding effect is more intuitive.

[0069] Optionally, it is not necessarily the user who holds the device to acquire multi-view images. For example, multiple fixed shooting devices that surround the center of a dedicated calibration site can be set up, the construction machinery is parked at the center of the site, and then these shooting devices are controlled for image acquisition.

[0070] Before acquiring multi-view images, the construction machinery needs to be parked in a preset posture and kept stationary. The specific posture can be determined according to the calibration requirements. For example, for Figure 1 [[ID=1,4]]the excavator 10, this posture can be described as: the forearm 120 of the excavator 10 is retracted, and the bottom surface of the bucket 140 is close to the ground.

[0071] Optionally, to prevent the user from being unfamiliar with how to adjust the parking posture of the construction machinery, it can also be guided through a pre-produced video or AR technology.

[0072] The device for acquiring multi-view images and the device for executing the calibration method may be the same device or may not be the same device. For the case where they are the same device, for example, using a mobile phone to shoot multi-view images and executing the calibration method on the mobile phone APP, then obtaining images in step S210 can be understood as acquiring images, and the specific acquisition methods have been described above. For the case where they are not the same device, for example, using a mobile phone to shoot multi-view images and then sending the images to a remote server to execute the calibration method, then obtaining images in step S210 can be understood as receiving images or reading images from a storage medium (such as a hard disk, memory).

[0073] In addition, the program for executing the calibration method can be part of a digital construction product or an independent program.

[0074] Regarding the prior calibration information in step S210, its specific meaning will be explained in step S230. For different calibration parameters, the required prior calibration information may also be different. The prior calibration information can be input by the user, and of course, other sources are not excluded.

[0075] Step S220: Perform three-dimensional reconstruction on the construction machinery based on multi-view images to obtain a three-dimensional model of the construction machinery.

[0076] For example, three-dimensional reconstruction based on images can be achieved by performing steps such as initialization, multi-view image feature extraction, matching the extracted features, and BA optimization, or three-dimensional reconstruction can also be directly achieved using software such as COLMAP.

[0077] Step S230: Measure the target space parameters in the three-dimensional model and convert the target space parameters into the space parameters to be calibrated according to the prior calibration information.

[0078] The space parameters refer to geometric parameters in a specific space. For example, one or more parameters such as length, angle, area, volume, etc. The so-called specific space is the model space where the three-dimensional model is located (the entire model space can also be considered as the three-dimensional model) for the target space parameters, and the physical world for the space parameters to be calibrated.

[0079] The target space parameters and the space parameters to be calibrated are corresponding, and there is a conversion relationship between them. For example, if length calibration is currently to be performed, the target space parameter can be the length in the three-dimensional model, the space parameter to be calibrated can be the length in the construction machinery, and there is a conversion relationship between these two lengths. At this time, the calculated space parameter to be calibrated is the result of the length calibration. Another example, if angle calibration is currently to be performed, the target space parameter can be the angle in the three-dimensional model, the space parameter to be calibrated can be the angle in the construction machinery, and there is a conversion relationship between these two angles. At this time, the calculated space parameter to be calibrated is the result of the angle calibration. Another example, if length calibration and angle calibration are currently to be performed, the target space parameters can be the length and angle in the three-dimensional model, the space parameters to be calibrated can be the length and angle in the construction machinery, and there are conversion relationships between these two lengths and these two angles respectively. At this time, the calculated space parameters to be calibrated are the results of the length calibration and the angle calibration.

[0080] The prior calibration information obtained in step S210 includes the information required to convert the target space parameters into the space parameters to be calibrated. For example, when performing length calibration, the prior calibration information includes the information required for length conversion; when performing angle calibration, the prior calibration information includes the information required for angle conversion; when performing both length calibration and angle calibration simultaneously, the prior calibration information includes the information required for both length conversion and angle conversion.

[0081] Optionally, after calculating the space parameters to be calibrated, they can be stored in digital construction products so that they can be put into normal use. Or, the space parameters to be calibrated can also be output so that the user can know, etc.

[0082] Brief summary Figure 2 The method in uses multi-view images of construction machinery for 3D reconstruction, measures the target space parameters in the 3D model, and then uses the prior calibration information to convert them into the space parameters to be calibrated. This calibration process does not rely on manual parameter measurement, thus significantly improving the calibration efficiency and avoiding manual measurement errors. In addition, the basic equipment required for this method during calibration is only an image acquisition device (for acquiring multi-view images), and no professional measurement equipment is needed. Its operation is simple, the requirements for calibration personnel are low, and the implementation cost is relatively low. In addition, this method can directly calculate each space parameter to be calibrated through the 3D model, without involving continuous measurement of the same type of parameter, so it can reduce the generation of cumulative errors and improve the calibration accuracy. In addition, this method supports the calibration of multiple space parameters and has high practicability.

[0083] Next, based on the above embodiments, taking length calibration and angle calibration as examples, the calibration method will be further introduced:

[0084] (1) Length calibration:

[0085] When performing length calibration, the prior calibration information includes the reference length, and the reference length represents the physical length between the reference positions on the construction machinery. Among them, the physical length is the length in the physical world, and there is also the modeling length, that is, the length in the model space where the 3D model is located (which can also be considered as the 3D model).

[0086] The reference positions can be two specified points on the construction machinery (including some additional components set on the construction machinery, such as the spirit level mentioned later), such as the center points of two certain rotating shafts. The reference length can be obtained through manual measurement, or it is possible that the reference length is known (for example, when the construction machinery is manufactured, it has been determined that the length specification between two points is 1m). After obtaining the reference length, it is input to the calibration program so that the calibration program can implement length calibration according to the following steps:

[0087] (1.1) Determine the model reference position corresponding to the reference position in the three-dimensional model.

[0088] After three-dimensional reconstruction, the reference position in the physical world is mapped to the corresponding position in the three-dimensional model, which is called the model reference position. The model reference position can be two points in the three-dimensional model corresponding to the reference position. Determining the model reference position can be achieved by at least one of the following methods:

[0089] Method 1: Manually select the model reference position. For example, if the reference position is the center points of two rotating shafts on an excavator, and the three-dimensional reconstruction is normal, the center points of these two rotating shafts can also be found in the three-dimensional model of the excavator, and they can be manually marked as the model reference position.

[0090] Method 2: If the construction machinery has certain features at the reference position and these features are also retained in the three-dimensional model, the model reference position corresponding to the reference position can be found by performing feature recognition in the three-dimensional model.

[0091] Method 3: The reference position in the construction machinery can be specially marked (for example, using the label mentioned later). The marked points can be found through computer vision algorithms in multi-view images. When performing three-dimensional reconstruction, record the mapping results of the marked points, and the coordinates of the model reference position in the three-dimensional model can be obtained.

[0092] (1.2) Measure the first modeling length between the model reference positions, and calculate the conversion scale between the construction machinery and the three-dimensional model (which is also the conversion scale between the physical world and the model space where the three-dimensional model is located) according to the reference length and the first modeling length.

[0093] Since the three-dimensional model is stored in the electronic device, after determining the model reference position, the first modeling length can be directly measured using a computer program. For example, if the reference length is 1m and the first modeling length is 1000 pix (three-dimensional pixel distance), the conversion scale can be 1 / 1000.

[0094] (1.3) Determine the model target positions in the three-dimensional model, measure the second modeling length between the model target positions, and convert the second modeling length into the physical length between the target positions according to the conversion scale.

[0095] The target positions in the construction machinery can be two specified points on the construction machinery (different from the reference position). The physical length between the target positions is the length to be calibrated, or the space parameter to be calibrated mentioned above. For example Figure 1 point A (the center of rotating shaft A) and point B (the center of rotating shaft B) in are the target positions, and the length of line segment AB is the physical length between the target positions.

[0096] The model target position is the position corresponding to the target position in the 3D model. The length between model target positions is the second modeling length. For example, point A' corresponding to point A and point B' corresponding to point B in the 3D model are model target positions, and the length of line segment A'B' is the second modeling length. Regarding the determination of the model target position and the measurement of the second modeling length, reference can be made to the determination of the model reference position and the measurement of the first modeling length respectively, and no repeated elaboration will be made.

[0097] After obtaining the second modeling length, the length to be calibrated can be obtained by combining the conversion scale calculated in step (1.2). For example, if the length of line segment A'B' is 2000 pix, then the length of line segment AB is 2000×1 / 1000 = 2 m.

[0098] What is introduced above is the calibration of one length. If there are multiple lengths to be calibrated on the same construction machinery, step (1.3) can be repeatedly executed, and the conversion scale does not need to be recalculated.

[0099] In the above length calibration method, length calibration can be completed by providing a reference length and performing certain scale conversions. The calibration process is very simple. Moreover, measuring the first modeling length and the second modeling length only requires calculation based on the 3D model in the computer and does not require manual measurement. Therefore, the calibration efficiency is high and it does not depend on professional measuring equipment.

[0100] In an implementation manner of the length calibration method, a spirit level can be set on the construction machinery and adjusted to be horizontal, such as Figure 3 the spirit level 170 in. For example, the spirit level can be installed on the surface of the construction machinery by means of pasting, etc. For a bubble spirit level, it can be horizontally adjusted according to the bubble therein.

[0101] The reference position for determining the reference length can be the specified points on the spirit level. For example, if the length of the spirit level is 1 m, the two endpoints of its scale can be selected as the specified points. Another example is that labels can be pasted on the spirit level to determine the two specified points. For details, refer to the elaboration on labels later. After selecting the reference position, the reference length can be directly determined using the scale of the spirit level. Of course, the scale can also not be used, and the reference length can be directly measured with another tool. In the latter case, the spirit level can have no scale.

[0102] At least some of the multi-view images collected should contain the spirit level so as to perform 3D reconstruction on the spirit level, thereby supporting the measurement of the first modeling length.

[0103] Optionally, the spirit level can be set on the central axis plane of the construction machinery (allowing a certain deviation, and the actual structure of the construction machinery needs to be considered whether it is convenient to set it here), which helps to reduce the systematic error in the length calibration process. Take Figure 3For example, the central plane is located at the center of the left and right sides of the excavator, and is also located at the center in the front and back sides (in the direction perpendicular to the paper). Figure 3 The spirit level 170 in Figure 3 is not set at the central plane.

[0104] It should be noted that if it is only for length calibration, the spirit level does not need to be adjusted horizontally. Or the spirit level can be not used, and an ordinary ruler can be used instead. Or, without using additional components, the reference position can be directly selected on the surface of the construction machinery. Among the above implementation methods, the reason for choosing the spirit level to provide the reference length is combined with the requirements of the angle calibration method described later. According to the content described later, the spirit level can also be used to provide a horizontal plane or a horizontal line during the angle calibration process. That is, with the help of this common tool, the spirit level, the functions of length calibration and angle calibration can be completed simultaneously, covering the most important calibration scenarios of construction machinery.

[0105] (2) Angle calibration:

[0106] The component on the construction machinery installed with the inclination sensor to be calibrated is called the target component. For example, for Figure 1 the excavator 10 in Figure 1 , the target component is the boom 120. During angle calibration, the prior calibration information includes the angle reading of the inclination sensor on the target component, and this reading can be directly obtained from the inclination sensor and input into the calibration program. The calibration program can achieve angle calibration according to the following steps:

[0107] (2.1) Determine the axis of the target component model in the three-dimensional model.

[0108] The target component model is the part in the three-dimensional model corresponding to the target component. For example, if the target component is the boom, the target component model is the boom model in the three-dimensional model.

[0109] The axis of the target component can refer to the connection line of the rotating shafts at both ends of the target component. For example, if the rotating shafts at both ends of the boom 120 are represented by point A and point B respectively, then the axis of the target component is the line segment AB. Of course, the axis of the target component can also adopt other definition methods, such as specifying two end points on the target component to form an axis, and these two end points do not necessarily correspond to the rotating shafts.

[0110] The axis of the target component corresponds to the axis of the target component model in the three-dimensional model. Given the two end points of the axis of the target component, determine the corresponding points of these two end points in the three-dimensional model (the method refers to determining the model reference position in the previous text), and then the axis of the target component model can be determined. For example, the two ends of the boom model correspond to points A' and B' corresponding to points A and B respectively, then the axis of the target component model is the line segment A'B'.

[0111] (2.2) Measure the angle between the axis of the target component model and the reference plane or reference line, and calculate the installation angle of the inclination sensor based on the angle reading of the inclination sensor and the measured angle.

[0112] The reference plane can be determined according to the calibration requirements. For example, a horizontal plane can be determined as the reference plane, or a vertical plane can be determined as the reference plane, or a plane with a fixed angle to the horizontal plane can be determined as the reference plane. After determining the reference plane, the computer program can measure the angle between the axis of the target component model and the reference plane.

[0113] The reference line is similar to the reference plane. For example, a horizontal line can be determined as the reference line, etc. After determining the reference line, the computer program can measure the angle between the axis of the target component model and the reference line.

[0114] Only one of the reference plane and the reference line needs to be implemented. For example, the reference line can be determined by two points in the three-dimensional model, while the reference plane can be determined by three or more points in the three-dimensional model.

[0115] Taking the case of using the reference plane as an example, if the reference plane is a horizontal plane and the angle reading of the inclination sensor is also a reading relative to the horizontal plane, then subtracting the angle reading of the inclination sensor from the measured angle gives the installation angle of the inclination sensor, which is the angle to be calibrated, or the space parameter to be calibrated in the previous text. For example, the angle between line segment A'B' and the horizontal plane is 84°, and the angle reading of the inclination sensor is 85°, then the installation angle of the inclination sensor is 85 - 84 = 1°.

[0116] If the reference plane is not a horizontal plane, but the angle reading of the inclination sensor is a reading relative to the horizontal plane, then the angle between the reference plane and the horizontal plane needs to be considered to calculate the installation angle of the inclination sensor. If the reference plane is a horizontal plane, but the angle reading of the inclination sensor is not a reading relative to the horizontal plane, it can be analyzed similarly and will not be elaborated in detail.

[0117] The above introduces the calibration of one installation angle. If there are multiple installation angles to be calibrated on the same construction machinery, steps (2.1) and (2.2) can be repeated for calibration.

[0118] In the above angle calibration method, the angle calibration can be completed by providing the angle reading of the inclination sensor and performing certain angle conversions. The calibration process is very simple. And measuring the angle between the axis of the target component model and the reference plane or reference line only needs to be calculated according to the three-dimensional model in the computer, without the need for manual measurement. Therefore, the calibration efficiency is high and it does not depend on professional measurement equipment.

[0119] In an implementation of the angle calibration method, one or more spirit levels can be set on the construction machinery and adjusted to be horizontal, such as Figure 3 the spirit level 170 in []. The reference plane or reference line used in angle calibration can be the horizontal plane or horizontal line determined according to the spirit level. For example, in the case where there is one spirit level set on the construction machinery, a horizontal line can be determined based on the coordinates of two specified points on the spirit level model (the part corresponding to the spirit level in the three-dimensional model).

[0120] At least some of the multi-view images collected should contain the spirit level so as to perform three-dimensional reconstruction of the spirit level, thereby supporting the determination of the horizontal plane or horizontal line based on the spirit level.

[0121] In the above implementation, by setting the spirit level to provide the horizontal plane or horizontal line, according to the previous content, the spirit level can also be used to provide the reference length, that is, by means of this common tool of the spirit level, the functions of angle calibration and length calibration can be completed simultaneously, covering the most important calibration scenarios of construction machinery.

[0122] Furthermore, the inventor found through research that if only one spirit level is set, although it can be adjusted to be horizontal, if the construction machinery is not parked on a horizontal ground (or at least close to a horizontal ground), especially if there are undulations in the direction perpendicular to the spirit level (for example, Figure 3 the direction perpendicular to the paper plane in []), the angle measurement accuracy will be affected.

[0123] Therefore, as an improved solution, two spirit levels can be set on the construction machinery and adjusted to be horizontal respectively. The two spirit levels are not parallel to each other and can be at the same or different heights (relative to the ground). In particular, the two spirit levels can also intersect with each other to form a cross spirit level.

[0124] Based on the spirit level models corresponding to the two spirit levels, a horizontal plane can be constructed. For example, first determine a horizontal line based on the coordinates of two specified points on each spirit level model, and a total of two horizontal lines are obtained. Then, the plane that is parallel to the two horizontal lines respectively and has the minimum sum of the distances from the two horizontal lines is determined as the required horizontal plane.

[0125] At least some of the multi-view images collected should contain the two spirit levels so as to perform three-dimensional reconstruction of the spirit levels, thereby supporting the determination of the horizontal plane based on the spirit levels.

[0126] In this solution, when performing angle calibration, the requirements for the parking conditions of the construction machinery are not high. It can be parked at a position that is not on a horizontal ground (or at least close to a horizontal ground), and relatively accurate angle measurement results can still be obtained, thereby improving the accuracy of the angle calibration results.

[0127] For the solution with two spirit levels, the two spirit levels can respectively provide reference lengths. If length calibration is still required at this time, two conversion scales can be calculated respectively according to the two reference lengths, and then the average value of these two conversion scales can be taken as the conversion scale finally used for length calibration. Alternatively, one of these two conversion scales can be selected as the conversion scale finally used for length calibration, and the other can only serve as a verification function (for example, verifying whether the calculated conversion scale is reasonable).

[0128] It should be noted that the number of spirit levels has no necessary relationship with constructing a horizontal plane or a horizontal line. For example, when two spirit levels are set, it is also possible to determine the horizontal line only according to one spirit level, and the other spirit level only serves as a verification. In addition, if three or more spirit levels are set, although theoretically it is not necessary to use so many spirit levels to construct a horizontal plane or a horizontal line, the extra spirit levels can only serve as a verification.

[0129] Optionally, one or more spirit levels can be set on the central axis plane of the construction machinery (allowing a certain deviation, and it is necessary to consider whether the actual structure of the construction machinery is convenient for setting here), which helps to reduce the systematic error in the angle calibration process. The central axis plane has been introduced during length calibration and will not be repeated here.

[0130] Next, on the basis of the above embodiments, the role of the label will be further introduced:

[0131] In one implementation, labels are provided on the surface of the construction machinery (including some additional components set on the construction machinery, such as spirit levels). The labels can be attached to the construction machinery by pasting or printing before calibration starts. It is even possible that the label is originally part of the construction machinery, for example, manufacturing a construction machinery or a spirit level with a label.

[0132] The pattern of the label can adopt, but is not limited to, Apriltag code, two-dimensional code, etc. Figure 4 Shows the pattern of a label provided by an embodiment of the present application.

[0133] Based on the pattern it contains, a label can provide the following two functions or one of the functions:

[0134] Function 1: The label includes a pattern with a specific texture, which is used to provide additional texture features during the three-dimensional reconstruction process.

[0135] 3D reconstruction is achieved based on the mutual matching of feature points in multi-view images, and the matching feature points in the images correspond to the same point in the actual construction machinery. However, there are often large flat solid-color areas on the surface of actual construction machinery (for example, the sides of the boom of an excavator are painted yellow), resulting in no obvious texture in the collected images in these areas, which is not conducive to the detection and matching of feature points. Therefore, by adding some tags containing texture features, providing additional texture features relative to the texture features of the construction machinery itself, it helps to enhance the efficiency and accuracy of feature matching during 3D reconstruction, thereby improving the 3D reconstruction effect.

[0136] Function 2: The tag includes a pattern with a unique identification function for marking specific positions on the construction machinery, and these specific positions are mapped to known coordinates in the 3D model after 3D reconstruction.

[0137] For example, the pattern of the tag contains a unique ID. Thus, for the position on the surface of the construction machinery where the tag is set, it is equivalent to assigning an ID to this position. During 3D reconstruction, the computer vision algorithm can identify the tag features in the multi-view images to obtain this ID. In the reconstructed 3D model, the feature points corresponding to the tag in the image are mapped to a 3D point, and the coordinates of this 3D point can be calculated, and the ID corresponding to this 3D point is also known. In other words, according to the ID of the tag, the position of each marked point in the 3D model can be determined, so that automated measurement (without manually selecting points in the 3D model) can be performed to achieve highly efficient calibration.

[0138] For example, in Figure 1 a tag with ID = 5 is set at the position corresponding to point A, and a tag with ID = 12 is set at the position corresponding to point B. After 3D reconstruction, according to these two IDs, 5 and 12, the coordinates of points A' and B' can be queried, so as to directly measure the length of line segment A'B' (the second modeling length mentioned above) and use it for length calibration.

[0139] Figure 5 shows a way of setting tags on an excavator provided by an embodiment of the present application. Refer to Figure 5, the white square represents a tag with Function 1, and the black square represents a tag with Function 2. It can be seen that the black squares mainly appear at the shaft positions, including on the spirit level, for marking some points necessary for calibration. The white squares appear at other positions, mainly serving to assist in 3D modeling. It should be noted that since the tags of the black squares also contain patterns, that is, there are texture features, these tags have a high probability of also being able to achieve Function 1 in addition to achieving Function 2. In different implementation methods, only the tags of the white squares can be set on the construction machinery, only the tags of the black squares can be set, or both types of tags can be set.

[0140] Optionally, to avoid users being unfamiliar with how to set tags on the surface of construction machinery, it can also be guided through pre-made videos or AR technology.

[0141] Next, for the tags with Function 2, specifically describe the positions where they can be set in the case of length calibration and angle calibration:

[0142] a. The tag is set on the spirit level for marking a specified point on the spirit level (which can refer to the center position of the tag). Specifically:

[0143] a1. When performing length calibration, the above-mentioned specified point can be used as a reference position to provide a reference length. After being mapped as the model reference position in the 3D model, it can also provide the first modeling length;

[0144] a2. When performing angle calibration, the coordinates corresponding to the above-mentioned specified point in the 3D model can be used to construct a horizontal plane or a horizontal line.

[0145] b. The tag is set on the measurement points on the surface of the construction machinery for marking the measurement points (which can refer to the center position of the tag). The measurement points are, for example, the center points of some shafts. Specifically:

[0146] b1. When performing length calibration, the above-mentioned measurement point is the target position in the construction machinery. After being mapped as the model target position in the 3D model, it is used to provide the second modeling length;

[0147] b2. When performing angle calibration, the above-mentioned measurement point is the axis end point of the target component in the construction machinery. After being mapped as the axis end point of the model target component in the 3D model, it is used to determine the axis of the model target component.

[0148] When implementing the calibration method, tags can be set only on the spirit level, or only at the measurement points on the surface of the construction machinery, or tags can also be set at both of these two places.

[0149] The positions where tags can be set are given above for length calibration and angle calibration. These positions are marked by setting tags, which is beneficial to realizing automatic measurement in the processes of length calibration and angle calibration and improving the calibration efficiency.

[0150] The modeling accuracy of the three-dimensional model represents the quality of the three-dimensional model. If the modeling accuracy of the three-dimensional model is relatively high, the subsequent calibration results will be relatively reliable; if the modeling accuracy of the three-dimensional model is relatively low, the subsequent punctuation results will be relatively unreliable. Next, based on the above embodiments, the evaluation of the modeling accuracy of the three-dimensional model will be further introduced. There are at least the following two solutions:

[0151] Solution 1

[0152] After step S220 and before step S230, the modeling accuracy of the three-dimensional model is evaluated. If the evaluation result meets the expected standard, step S230 is executed; if the evaluation result does not meet the expected standard, step S230 is not executed. For example, a prompt message can be output to instruct the user to re-collect multi-view images and re-perform three-dimensional reconstruction.

[0153] For example, the evaluation result can be a numerical value, and the expected standard can be a threshold. If the larger the numerical value represents the higher the modeling accuracy, then when the numerical value is not less than the threshold, it can be considered that the evaluation result meets the expected standard, and when the numerical value is less than the threshold, it can be considered that the evaluation result does not meet the expected standard.

[0154] The evaluation result meeting the expected standard indicates that the quality of the three-dimensional model is good. Thus, based on such a three-dimensional model, step S230 is executed, and the obtained calibration result (i.e., the calculated spatial parameters to be calibrated) has good reliability, and digital construction products can also work well based on the calibration result. The evaluation result not meeting the expected standard indicates that the quality of the three-dimensional model is poor. Thus, even if step S230 is executed based on such a three-dimensional model, the obtained calibration result has poor reliability, and digital construction products cannot work well based on such a calibration result. Therefore, to avoid wasting computing resources, step S230 can no longer be executed.

[0155] Solution 2

[0156] After step S220, the modeling accuracy of the three-dimensional model is evaluated. Evaluating the modeling accuracy does not affect the execution of step S230, and the two are independent. The execution order of evaluating the modeling accuracy and step S230 is not limited: for example, the modeling accuracy can be evaluated first and then step S230 can be executed; for another example, step S230 can be executed first and then the modeling accuracy can be evaluated; for yet another example, evaluating the modeling accuracy and step S230 can be executed in parallel.

[0157] After step S230 is completed, the confidence level of the calibration result can be given based on the evaluation result of the modeling accuracy, that is, the reliability of the calibration result. Since the calibration method is mainly based on the 3D model, it is reasonable to determine the confidence level of the calibration result based on the evaluation result of the modeling accuracy. For example, if a model confidence level index is used as the evaluation result of the modeling accuracy, this index can be directly used as the confidence level of the calibration result. Another example is that the confidence level of the calibration result can also be determined by combining other factors on the basis of the evaluation result of the modeling accuracy.

[0158] After that, corresponding operations can be performed based on the confidence level of the calibration result: for example, directly outputting this confidence level so that the user can know the reliability of the current calibration result; another example is to judge whether this confidence level is not less than a threshold. If it is not less than the threshold, it means that the confidence level is relatively high and no operation needs to be performed. If it is less than the threshold, it means that the confidence level is relatively low, and a warning message can be output, or the user can be directly prompted to re-collect multi-view images and re-perform 3D reconstruction, etc.

[0159] In summary, whether it is Solution 1 or Solution 2, by evaluating the model quality, the accuracy of the calibration result can be improved (by re-modeling) or at least the user or the system (referring to the software that executes the calibration method) can have a correct understanding of the accuracy of the calibration result, improving the problem in the prior art that it is difficult to verify the correctness of the calibration result.

[0160] In one implementation, multiple confidence level indicators representing the modeling accuracy of the 3D model can be calculated, and these confidence level indicators can be weighted and calculated, and the total confidence level of the obtained 3D model is used as the evaluation result of the modeling accuracy.

[0161] Among them, the confidence level indicators that can be adopted are shown in the examples later. The weights used for weighted calculation can adopt empirical values; or, they can also be obtained through training: regarding the process of calculating the total confidence level based on multiple confidence level indicators as a model, the input of the model is multiple confidence level indicators, and the output is the total confidence level. The parameters of this model are the weights, so the values of the weights can be obtained through training based on some existing labeled samples. During the subsequent use of the model, the values of these weights can also be optimized. Considering the model, the above-mentioned weighted calculation can be not limited to simple weighted summation. For example, the forward propagation process in a multi-layer perceptron (a neural network model) can also be regarded as a weighted calculation.

[0162] In the above implementation, the modeling accuracy of the three-dimensional model is evaluated by weighted calculation of multiple confidence indicators. Since multiple factors are considered, the evaluation results are relatively reliable. Moreover, by setting the weights for weighted calculation, the proportion of different factors in the evaluation of modeling accuracy can be flexibly adjusted, further improving the accuracy of the evaluation results. Of course, it is also possible to directly calculate a single confidence indicator and use it as the evaluation result of the modeling accuracy.

[0163] Next, some possible confidence indicators will be specifically introduced. Two or more of these indicators can be selected to calculate the total confidence of the three-dimensional model.

[0164] (1) Proportion of calibrated images: The proportion of qualified images that meet the requirements of three-dimensional reconstruction among multi-view images.

[0165] Among the multi-view images obtained in step S210, some may not meet the requirements of three-dimensional reconstruction. These images are called unqualified images, and the remaining images are called qualified images. Unqualified images are, for example, images with severe backlighting, blurring, occlusion, or out-of-focus. These images do not participate in three-dimensional reconstruction, and only qualified images participate in three-dimensional reconstruction.

[0166] The larger the value of this indicator, the higher the modeling accuracy.

[0167] (2) Total number of connection points: The total number of feature points that are successfully matched between different qualified images.

[0168] As previously mentioned, three-dimensional reconstruction is achieved based on feature point matching. Feature points that are successfully matched between different qualified images can be called connection points. For example, there are 10 qualified images, and 5 of them have matching feature points that correspond to point A on the construction machinery, that is, there are 5 connection points; another 8 have matching feature points that correspond to point B on the construction machinery, then there are 8 connection points, so the total number of connection points is 5 + 8 = 13. Note that this is only an extremely simplified example and does not represent the real situation. Similar situations apply to the following examples.

[0169] The larger the value of this indicator, the higher the modeling accuracy.

[0170] (3) Total number of observed points: The total number of three-dimensional points calculated based on the connection points.

[0171] A set of mutually matched connection points can calculate a three-dimensional point in the three-dimensional model, which can be called an observed point. Continuing with the example in indicator (2), based on the first 5 connection points, one three-dimensional point can be calculated, and based on the latter 8 connection points, one three-dimensional point can be calculated, so the total number of observed points is 2.

[0172] The larger the value of this indicator, the higher the modeling accuracy.

[0173] (4) Average number of observations at the observation points: Total number of connection points in index (2) / Total number of observation points in index (3).

[0174] The larger the value of this index, the higher the modeling accuracy.

[0175] (5) Average number of observation points per image: Total number of observation points in index (3) / Total number of qualified images.

[0176] The larger the value of this index, the higher the modeling accuracy.

[0177] (6) Trajectory length: Total length of the connecting lines of the camera centers corresponding to the qualified images.

[0178] Each qualified image corresponds to a camera center, and the position of the camera center in the three-dimensional space can be calculated based on the camera parameters. Connect these camera centers with straight lines in the order of image acquisition, and the total length of this connecting line is the trajectory length. For example, if there are 10 qualified images, then there are 10 camera centers, and after connection, a 9-segment broken line is formed, and the total length of this broken line is the trajectory length.

[0179] The closer the value of this index is to a preset optimal value, the higher the modeling accuracy.

[0180] (7) Average trajectory length: Trajectory length / Number of trajectory segments.

[0181] Each segment of the trajectory is defined as the connecting line between two adjacent camera centers, and "adjacent" here can be understood as adjacent in the order of image acquisition. For example, continuing the example in index (6), the average trajectory length is equal to the trajectory length / 9.

[0182] The closer the value of this index is to a preset optimal value, the higher the modeling accuracy. For example, when collecting multi-view images, if the expected walking speed is 1.5 m / s and 2 images are collected per second, then this optimal value can be taken as 0.75 m.

[0183] (8) Total reprojection error: Sum of all reprojection errors.

[0184] The reprojection error can be defined as the distance between the projection point and the connection point used to calculate this observation point when projecting the observation point back to the image plane. The reprojection error is used to measure the matching accuracy between the observation point and the image data, and the total reprojection error is the sum of all reprojection errors.

[0185] The smaller the value of this index (the closer it is to 0), the higher the modeling accuracy.

[0186] (9) Average reprojection error: Total reprojection error in index (8) / Total number of observation points in index (3).

[0187] The smaller the value of this index (the closer it is to 0), the higher the modeling accuracy.

[0188] (10) Reprojection error distribution: Statistical distribution parameters of all reprojection errors.

[0189] For example, the statistical distribution parameter can be the variance or standard deviation of the reprojection error. This index reflects the consistency of the reprojection error.

[0190] The smaller the value of this index (the closer it is to 0), the higher the modeling accuracy.

[0191] (11) Prior geometric error: The error between the geometric properties measured from the 3D model and the corresponding actual geometric properties in construction machinery (including some additional components set on construction machinery, such as a spirit level).

[0192] Among them, the corresponding actual geometric properties in construction machinery are a kind of prior knowledge, and the geometric properties can include one or more of properties such as length, angle (including the sum and difference of angles), radian, parallelism, etc.

[0193] For example, according to the product information or actual measurement of construction machinery, it is known that the length of a certain component is 1m. By measuring the modeled length of this component through the 3D model and converting it into physical length, the result is 1.05m. Then the prior geometric error is 5%.

[0194] Another example, set two spirit levels. The reference length of the first spirit level is 1m, the first modeled length is 1000pix, and the obtained conversion scale is 1 / 1000. The reference length of the second spirit level is 0.5m, and the first modeled length is 495pix. Then, according to 495pix and the conversion scale, the reference length of the second spirit level can be calculated as 0.495m, and there is a 5% prior geometric error from the true reference length.

[0195] That is, in the case of setting multiple spirit levels, the reference lengths provided by some spirit levels can be used as objects for verification. For example, if three spirit levels are set, the average value of the conversion scales of the first two levels can be taken as the conversion scale for calibration length, and then based on this conversion scale, the reference length of the third spirit level is calculated, and the error is calculated with the true reference scale of the third spirit level. For the case of only one spirit level, such verification can also be carried out. For example, set three labels on the spirit level to form multiple reference scales.

[0196] Another example, it is known that a certain component of construction machinery is circular, that is, the roundness value is 1, and this component appears as an ellipse in the 3D model with a roundness value of 0.9. Then the prior geometric error is 10%, and so on.

[0197] In the above implementation manners, a variety of specific confidence indicators are given, and these indicators characterize the modeling accuracy of the 3D model from different dimensions. Therefore, by weighting two or more of these indicators, the modeling accuracy can be evaluated more accurately from multiple perspectives.

[0198] If a numerical value between [0, 1] is used to represent the total confidence of the 3D model, then normalization may be required when calculating the confidence based on the above indicators. For example, for the average trajectory length, the calculation result can be divided by a preset value to normalize it to the range of [0, 1], or the method of scoring in intervals can also be used to normalize it to the range of [0, 1]. In addition, for some of the above indicators, the higher the value, the higher the modeling accuracy, while for others, the lower the value, the higher the modeling accuracy. When calculating the total confidence, the standard can be unified through the sign of the weight (such as unifying to the higher the indicator value, the higher the modeling accuracy).

[0199] Next, a possible calibration process executed on a mobile phone will be described from the perspective of time sequence to facilitate understanding of the solution of this application:

[0200] Step 1: The user opens the APP of the digital construction product (with calibration function) to start the calibration process.

[0201] Step 2: Under video guidance (including AR guidance), the user pastes labels on the excavator, installs a spirit level, adjusts the parking attitude of the excavator, and performs multi-perspective image acquisition through the shooting function of the APP. The order of some of these steps can be exchanged. For example, the excavator can be parked first and then the label can be pasted, or the label can be pasted first and then the excavator can be parked, and so on.

[0202] Step 3: The user inputs the reference length and the readings of the inclination sensor to the APP.

[0203] Step 4: The APP performs 3D reconstruction of the excavator based on the multi-perspective images to obtain a 3D model.

[0204] Step 5: The APP evaluates the modeling accuracy.

[0205] Step 6: The APP performs length and angle calibration based on the 3D model and the information input by the user in Step 3. Since labels are set, efficient automated measurement can be achieved during the calibration process without manual intervention.

[0206] Step 7: The calibration result and the confidence are displayed on the APP interface, where the confidence is calculated based on the evaluation result in Step 5.

[0207] Figure 6 The functional modules of a construction machinery calibration device provided by an embodiment of this application are shown.

[0208] Refer to Figure 6, the construction machinery calibration device 300 includes:

[0209] A data acquisition module 310, configured to acquire multi-view images of the construction machinery and prior calibration information;

[0210] A 3D reconstruction module 320, configured to perform 3D reconstruction on the construction machinery based on the multi-view images to obtain a 3D model of the construction machinery;

[0211] A calibration module 330, configured to measure target space parameters in the 3D model and convert the target space parameters into space parameters to be calibrated according to the prior calibration information; wherein, the prior calibration information includes information required for parameter conversion.

[0212] In an implementation manner of the construction machinery calibration device 300, the prior calibration information includes a reference length, and the reference length represents the physical length between reference positions on the construction machinery; the calibration module 330 measures target space parameters in the 3D model and converts the target space parameters into space parameters to be calibrated according to the prior calibration information, including: determining the model reference positions corresponding to the reference positions in the 3D model; measuring a first modeling length between the model reference positions, and calculating a conversion scale between the construction machinery and the 3D model according to the reference length and the first modeling length; determining model target positions in the 3D model, measuring a second modeling length between the model target positions, and converting the second modeling length into the physical length between the target positions according to the conversion scale; wherein, the target positions are the positions corresponding to the model target positions on the construction machinery, and the physical length between the target positions is the length to be calibrated.

[0213] In an implementation manner of the construction machinery calibration device 300, the multi-view images include a spirit level provided on the construction machinery, and the reference positions are designated points on the spirit level.

[0214] In an implementation manner of the construction machinery calibration device 300, the prior calibration information includes the angle readings of an inclination sensor installed on a target component of the construction machinery; the calibration module 330 measures target space parameters in the 3D model and converts the target space parameters into space parameters to be calibrated according to the prior calibration information, including: determining the axis of the target component model in the 3D model; wherein, the target component model is the part in the 3D model corresponding to the target component; measuring the angle of the axis relative to a reference plane, and calculating the installation angle of the inclination sensor according to the angle readings and the angle; wherein, the installation angle of the inclination sensor is the angle to be calibrated.

[0215] In one implementation of the construction machinery calibration device 300, the multi-view image includes a spirit level disposed on the construction machinery, and the reference plane is a horizontal plane determined according to the spirit level.

[0216] In one implementation of the construction machinery calibration device 300, the multi-view image includes two spirit levels disposed on the construction machinery, and the two spirit levels are not parallel to each other. The reference plane is a horizontal plane jointly determined by the two spirit levels.

[0217] In one implementation of the construction machinery calibration device 300, the spirit level is disposed on the central axis plane of the construction machinery.

[0218] In one implementation of the construction machinery calibration device 300, the multi-view image includes a label disposed on the surface of the construction machinery. The label includes: a pattern with a specific texture for providing additional texture features during the 3D reconstruction process; and / or, a pattern with a unique identification function for marking a specific position in the construction machinery, and the specific position is mapped to a known coordinate in the 3D model after 3D reconstruction.

[0219] In one implementation of the construction machinery calibration device 300, the label includes a pattern with a unique identification function; the label is disposed on the spirit level on the construction machinery for marking a specified point on the spirit level. When performing length calibration, the specified point is a reference position on the construction machinery. When performing angle calibration, the coordinates corresponding to the specified point in the 3D model are used to construct a horizontal plane or a horizontal line; and / or, the label is disposed on a measurement point on the surface of the construction machinery for marking the measurement point. When performing length calibration, the measurement point is a target position in the construction machinery, and when performing angle calibration, the measurement point is an axis end point of a target component in the construction machinery.

[0220] In one implementation of the construction machinery calibration device 300, the device further includes: an accuracy evaluation module for evaluating the modeling accuracy of the 3D model of the construction machinery after the 3D reconstruction module 320 obtains the 3D model of the construction machinery and before the calibration module 330 measures the target space parameters in the 3D model, and determining that the evaluation result meets the expected standard; or, for evaluating the modeling accuracy of the 3D model of the construction machinery after the 3D reconstruction module 320 obtains the 3D model of the construction machinery, and determining the confidence level of the space parameters to be calibrated according to the evaluation result.

[0221] In one implementation of the construction machinery calibration device 300, the evaluation module evaluates the modeling accuracy of the three-dimensional model, including: calculating a plurality of confidence indicators representing the modeling accuracy of the three-dimensional model, and performing a weighted calculation on the plurality of confidence indicators, and using the total confidence of the obtained three-dimensional model as the evaluation result.

[0222] In one implementation of the construction machinery calibration device 300, the plurality of confidence indicators include at least two of the following indicators: proportion of calibrated images: the proportion of qualified images that meet the three-dimensional reconstruction requirements among the multi-view images; total number of connection points: the total number of feature points that match successfully between different qualified images; total number of observed points: the total number of three-dimensional points calculated based on the connection points; average number of observations per observed point: the ratio of the total number of connection points to the total number of observed points; average number of observed points per image: the ratio of the total number of observed points to the total number of qualified images; trajectory length: the total length of the connection lines of the camera centers corresponding to the qualified images; average trajectory length: the ratio of the trajectory length to the number of trajectory segments; where each trajectory segment is the connection line of two camera centers; total reprojection error: the sum of all reprojection errors; where the reprojection error is the distance between the projection point and the connection point used to calculate the observed point when the observed point is projected back to the image plane; average reprojection error: the ratio of the total reprojection error to the total number of observed points; reprojection error distribution: the statistical distribution parameters of all reprojection errors; prior geometric error: the error between the geometric properties measured from the three-dimensional model and the corresponding actual geometric properties in the construction machinery.

[0223] In one implementation of the construction machinery calibration device 300, the multi-view images include images collected from multiple angles around the construction machinery, or the multi-view images include images collected from multiple angles semi-surrounding the construction machinery.

[0224] The construction machinery calibration device 300 provided by the embodiments of the present application can be used to execute the construction machinery calibration method provided by the embodiments of the present application. Its implementation principle and the resulting technical effects have been introduced in the foregoing method embodiments. For a brief description, for the parts not mentioned in the device embodiment, reference can be made to the corresponding content in any of the foregoing method embodiments.

[0225] Figure 7 shows the structure of an electronic device provided by an embodiment of the present application. Refer to Figure 7 , the electronic device 400 includes: a processor 410 and a memory 420. These components are interconnected and communicate with each other through a communication bus 430 and / or other forms of connection mechanisms (not shown).

[0226] Among them, the processor 410 includes one or more (only one is shown in the figure), which can be an integrated circuit chip with the ability to process signals. The above-mentioned processor 410 can be a general-purpose processor, including a central processing unit (CPU), a microcontroller unit (MCU), a network processor (NP), or other conventional processors; it can also be a special-purpose processor, including a graphics processing unit (GPU), a neural-network processing unit (NPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. Moreover, when there are multiple processor 410s, a part of them can be general-purpose processors and another part can be special-purpose processors.

[0227] The memory 420 includes one or more (only one is shown in the figure), which can be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc.

[0228] The processor 410 and other possible components can access the memory 420, read and / or write data therein. In particular, one or more computer program instructions can be stored in the memory 420, and the processor 410 can read and run these computer program instructions to implement the construction machinery calibration method provided by the embodiments of the present application.

[0229] It can be understood that Figure 7 The structure shown is only schematic, and the electronic device 400 can also include more thanFigure 7 more or fewer components shown therein, or having a configuration different from that Figure 7 shown. For example, the electronic device 400 may further include a communication unit, such as a wired and / or wireless communication module, for communicating with other devices. As another example, the electronic device 400 may include a camera for collecting multi-view images required for calibration, and so on.

[0230] Figure 7 Each component shown therein may be implemented by hardware, software, or a combination thereof. The electronic device 700 may be a physical device, such as a PC, a server, a robot, an industrial device, etc., or may be a virtual device, such as a virtual machine, a container, etc. Moreover, the electronic device 700 is not limited to a single device, and may also be a combination of multiple devices or a cluster composed of a large number of devices.

[0231] The embodiments of the present application further provide a computer-readable storage medium, on which computer program instructions are stored. When these computer program instructions are read and run by a processor, they execute the construction machinery calibration method provided by the embodiments of the present application. For example, the computer-readable storage medium may be implemented as Figure 7 the memory 420 in the electronic device 400 shown therein, or may be implemented as a medium independent of the device, such as a mobile hard disk, a USB flash drive, an optical disc, etc.

[0232] The embodiments of the present application further provide a computer program product, which includes computer program instructions. When these computer program instructions are read and run by a processor, they execute the construction machinery calibration method provided by the embodiments of the present application. For example, these computer program instructions may be stored Figure 7 inside the memory 420 in the electronic device 400 shown therein, or may be stored in a medium independent of the device, such as a mobile hard disk, a USB flash drive, an optical disc, etc.

[0233] The above description is only for the embodiments of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A calibration method for construction machinery, characterized in that, Including: Obtaining multi-view images of construction machinery and prior calibration information; Performing three-dimensional reconstruction on the construction machinery based on the multi-view images to obtain a three-dimensional model of the construction machinery; Measuring target space parameters in the three-dimensional model and converting the target space parameters into space parameters to be calibrated according to the prior calibration information; wherein, the prior calibration information includes information required for parameter conversion.

2. The calibration method for construction machinery according to claim 1, wherein The prior calibration information includes a reference length, and the reference length represents the physical length between reference positions on the construction machinery; The measuring the target space parameters in the three-dimensional model and converting the target space parameters into space parameters to be calibrated according to the prior calibration information includes: Determining the model reference positions corresponding to the reference positions in the three-dimensional model; Measuring a first modeling length between the model reference positions and calculating a conversion scale between the construction machinery and the three-dimensional model according to the reference length and the first modeling length; Determining model target positions in the three-dimensional model, measuring a second modeling length between the model target positions, and converting the second modeling length into the physical length between the target positions according to the conversion scale; wherein, the target positions are the positions corresponding to the model target positions on the construction machinery, and the physical length between the target positions is the length to be calibrated.

3. The calibration method for construction machinery according to claim 2, wherein, The multi-view images include a spirit level provided on the construction machinery, and the reference positions are specified points on the spirit level.

4. The construction machinery calibration method according to claim 1, characterized in that The prior calibration information includes the angle readings of an inclination sensor mounted on a target component of the construction machinery; The measuring the target space parameters in the three-dimensional model and converting the target space parameters into space parameters to be calibrated according to the prior calibration information includes: Determining the axis of a target component model in the three-dimensional model; wherein, the target component model is the part in the three-dimensional model corresponding to the target component; Measuring the angle between the axis and a reference plane or a reference line and calculating the installation angle of the inclination sensor according to the angle readings and the angle; wherein, the installation angle of the inclination sensor is the angle to be calibrated.

5. The calibration method for construction machinery according to claim 4, wherein, The multi-view images include a spirit level provided on the construction machinery, and the reference plane or the reference line is a horizontal plane or a horizontal line determined according to the spirit level.

6. The calibration method for construction machinery according to claim 5, characterized in that, The multi-view images include two spirit levels provided on the construction machinery, and the two spirit levels are not parallel to each other, and the reference plane is the horizontal plane jointly determined by the two spirit levels.

7. The calibration method for construction machinery according to any one of claims 3, 5, and 6, characterized in that, The spirit level is provided on the central plane of the construction machinery.

8. The calibration method for construction machinery according to claim 1, wherein The multi-view images include labels provided on the surface of the construction machinery, and the labels include: Patterns with specific textures, which are used to provide additional texture features during the three-dimensional reconstruction process; and / or, Patterns with a unique identification function, which are used to mark specific positions in the construction machinery, and the specific positions are mapped to known coordinates in the three-dimensional model after the three-dimensional reconstruction.

9. The calibration method for construction machinery according to claim 8, wherein The label includes a pattern with a unique identification function; The label is set on the spirit level on the construction machinery and is used to mark a specified point on the spirit level. When performing length calibration, the specified point is the reference position on the construction machinery. When performing angle calibration, the coordinates corresponding to the specified point in the three-dimensional model are used to construct a horizontal plane or a horizontal line; and / or, The label is set on the measurement point on the surface of the construction machinery and is used to mark the measurement point. When performing length calibration, the measurement point is the target position in the construction machinery. When performing angle calibration, the measurement point is the axis end point of the target component in the construction machinery.

10. The calibration method for construction machinery according to claim 1, characterized in that, After obtaining the three-dimensional model of the construction machinery and before measuring the target space parameters in the three-dimensional model, the method further includes: Evaluating the modeling accuracy of the three-dimensional model and determining that the evaluation result meets the expected standard; or, After obtaining the three-dimensional model of the construction machinery, the method further includes: Evaluating the modeling accuracy of the three-dimensional model and determining the confidence level of the space parameters to be calibrated according to the evaluation result.

11. The calibration method for construction machinery according to claim 10, wherein, The evaluating the modeling accuracy of the three-dimensional model includes: Calculating a plurality of confidence level indicators characterizing the modeling accuracy of the three-dimensional model, performing a weighted calculation on the plurality of confidence level indicators, and using the total confidence level of the obtained three-dimensional model as the evaluation result.

12. The calibration method for construction machinery according to claim 11, wherein The plurality of confidence level indicators include at least two of the following indicators: Proportion of calibrated images: the proportion of qualified images that meet the three-dimensional reconstruction requirements in the multi-view images; Total number of connection points: the total number of feature points that match successfully between different qualified images; Total number of observed points: the total number of three-dimensional points calculated based on the connection points; Average number of observations of the observed points: the ratio of the total number of connection points to the total number of observed points; Average number of observed points per image: the ratio of the total number of observed points to the total number of qualified images; Track length: the total length of the connection lines of the camera centers corresponding to the qualified images; Average track length: the ratio of the track length to the number of track segments; wherein, each track segment is the connection line between two adjacent camera centers; Total reprojection error: the sum of all reprojection errors; wherein, the reprojection error is the distance between the projection point and the connection point used to calculate the observed point when projecting the observed point back to the image plane; Average reprojection error: the ratio of the total reprojection error to the total number of observed points; Reprojection error distribution: the statistical distribution parameters of all reprojection errors; Prior geometric error: the error between the geometric attributes measured from the three-dimensional model and the corresponding actual geometric attributes in the construction machinery.

13. The construction machinery calibration method according to claim 1, characterized in that The multi-view images include images collected from multiple angles around the construction machinery, or the multi-view images include images collected from multiple angles semi-surrounding the construction machinery.

14. A computer program product, characterized in that, Including computer program instructions, when the computer program instructions are read and run by a processor, the method according to any one of claims 1-13 is executed.

15. A computer-readable storage medium, characterized in that, Computer program instructions are stored on the computer-readable storage medium, and when the computer program instructions are read and run by a processor, the method according to any one of claims 1-13 is executed.

16. An electronic device, characterized in that, Including: A memory and a processor, wherein computer program instructions are stored in the memory, and when the computer program instructions are read and run by the processor, the method according to any one of claims 1-12 is executed.

Images