A method, device, equipment and medium for calibrating a visual sensor based on a fork

Abstract

The present application discloses a method, device, equipment and medium for calibrating a visual sensor based on a fork, and relates to the field of sensor calibration, including: modeling the fork based on the current point cloud data collected by the visual sensor fixed at the fork root to obtain the initial fork center point; determining the single calibration parameters using the relationship between the initial fork center point and the coordinate system of the visual sensor to perform a single calibration on the visual sensor; adjusting the current calibration model according to the covariance between the predicted fork state determined by the current calibration model and the actual fork state determined by the visual sensor after a single calibration, and if the adjusted calibration model tends to be stable, calculating the real-time calibration parameters through the adjusted calibration model to perform real-time calibration on the visual sensor. The present application adjusts the calibration model through the visual sensor after a single calibration, so as to perform real-time calibration on the visual sensor using the adjusted calibration model, thereby improving the calibration efficiency of the visual sensor.

Description

Technical Field

[0001] The present application relates to the field of sensor calibration, and in particular to a fork-based visual sensor calibration method, device, equipment and medium. Background Art

[0002] The load or vibration generated by the AGV (Automated Guided Vehicle) fork during operation will cause the relative position of the visual sensor and the fork to change, which will easily cause large errors when the fork performs visual tasks, affecting the success rate of the fork operation. Currently, the visual sensor is usually calibrated by specific tooling calibration or QR code calibration; among them, for specific tooling calibration, it is easy to incur high costs because special tooling is required; for QR code calibration, because the QR code is easily damaged, it requires frequent manual maintenance. Summary of the invention

[0003] In view of this, the purpose of this application is to provide a fork-based visual sensor calibration method, device, equipment and medium, which can adjust the calibration model through the visual sensor after a single calibration, so as to use the adjusted calibration model to calibrate the visual sensor in real time, thereby improving the calibration efficiency of the visual sensor. The specific scheme is as follows:

[0004] In a first aspect, the present application provides a fork-based visual sensor calibration method, comprising:

[0005] Modeling the fork based on current point cloud data collected by a visual sensor fixed at the fork root to obtain an initial fork center point;

[0006] Determine a single calibration parameter by using the relationship between the initial fork center point and the target coordinate system established based on the visual sensor to perform a single calibration on the visual sensor;

[0007] The current calibration model is adjusted according to the target state error covariance between the predicted fork state determined by the current calibration model and the actual fork state determined by the visual sensor after a single calibration. If the adjusted calibration model tends to be stable, the real-time calibration parameters are calculated by the adjusted calibration model to perform real-time calibration on the visual sensor; wherein the fork state includes position information and speed information of the center point of the fork.

[0008] Optionally, the fork is modeled based on the current point cloud data collected by a visual sensor fixed at the fork root to obtain an initial fork center point, including:

[0009] Acquire current point cloud data collected by a visual sensor fixed at the fork root of the fork, and determine target point cloud data of the fork from the current point cloud data according to the relative position relationship between the visual sensor and the fork;

[0010] The fork is modeled based on the target point cloud data and the priori parameters of the fork, and an initial fork center point is determined according to the modeled fork.

[0011] Optionally, the determining of a single calibration parameter by using the relationship between the initial fork center point and a target coordinate system established based on the visual sensor includes:

[0012] Determine the deflection angle of the initial fork center point relative to the longitudinal coordinate axis in the target coordinate system to obtain a corresponding rotation matrix; the target coordinate system is a coordinate system established based on the visual sensor;

[0013] Determine the deflection angle of the initial fork center point relative to the transverse coordinate axis in the target coordinate system to obtain a corresponding transfer matrix;

[0014] A single calibration parameter is determined based on the rotation matrix and the transfer matrix.

[0015] Optionally, the current calibration model is a model constructed using a target detection algorithm based on deep learning.

[0016] Optionally, the current calibration model is a model constructed using a Kalman filter algorithm;

[0017] Accordingly, the target state error covariance between the predicted fork state determined by the current calibration model and the actual fork state determined by the visual sensor after a single calibration is used to adjust the current calibration model, including:

[0018] Determine the prior fork state at the current moment based on the fork state at the previous moment through the current calibration model, and determine the prior state error covariance at the current moment based on the state error covariance at the previous moment and the process noise covariance;

[0019] Determine the Kalman gain based on the prior state error covariance and the measurement noise covariance, and determine the predicted fork state at the current moment by using the Kalman gain, the prior fork state and the actual fork state at the current moment; the actual fork state is the fork state determined according to the visual sensor after a single calibration;

[0020] The target state error covariance at the current moment between the predicted fork state and the actual fork state is determined according to the Kalman gain and the prior state error covariance, so as to adjust the current calibration model.

[0021] Optionally, the fork state at the initial moment is the fork state corresponding to the initial fork center point; and the state error covariance at the initial moment is a preset state error covariance.

[0022] Optionally, the adjusting the current calibration model includes:

[0023] adjusting the process noise covariance and the measurement noise covariance in the current calibration model according to the target state error covariance;

[0024] If the adjusted process noise covariance and the measurement noise covariance tend to converge, it is determined that the adjusted calibration model tends to be stable.

[0025] In a second aspect, the present application provides a fork-based visual sensor calibration device, comprising:

[0026] A center point acquisition module, used to model the fork based on current point cloud data collected by a visual sensor fixed at the fork root, so as to obtain an initial fork center point;

[0027] A single calibration module, used to determine a single calibration parameter by using the relationship between the initial fork center point and the target coordinate system established based on the visual sensor, so as to perform a single calibration on the visual sensor;

[0028] A real-time calibration module is used to adjust the current calibration model according to the target state error covariance between the predicted fork state determined by the current calibration model and the actual fork state determined by the visual sensor after a single calibration. If the adjusted calibration model tends to be stable, the real-time calibration parameters are calculated by the adjusted calibration model to perform real-time calibration on the visual sensor; wherein the fork state includes the position information and speed information of the fork center point.

[0029] In a third aspect, the present application provides an electronic device, including:

[0030] Memory, used to store computer programs;

[0031] A processor is used to execute the computer program to implement the aforementioned fork-based visual sensor calibration method.

[0032] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned fork-based visual sensor calibration method.

[0033] The present application models the fork based on current point cloud data collected by a visual sensor fixed at the root of the fork to obtain an initial fork center point; determines a single calibration parameter using the relationship between the initial fork center point and a target coordinate system established based on the visual sensor to perform a single calibration on the visual sensor; adjusts the current calibration model based on a target state error covariance between a predicted fork state determined by a current calibration model and an actual fork state determined by the visual sensor after a single calibration; if the adjusted calibration model tends to be stable, calculates real-time calibration parameters using the adjusted calibration model to perform real-time calibration on the visual sensor; wherein the fork state includes position information and speed information of the fork center point.

[0034] As can be seen from the above, the present application models the fork through the current point cloud data collected by the visual sensor to determine the center point of the fork based on the modeled fork, and thus uses the relationship between the center point of the fork and the coordinate system of the visual sensor to perform a single calibration on the visual sensor, which not only avoids the problem of the need for special tooling when calibrating the visual sensor using specific tooling, thereby reducing costs, but also avoids the problem of the need for manual maintenance of the QR code when calibrating the visual sensor using a QR code, thereby reducing manpower consumption; further, the present application determines the actual fork state through the visual sensor after a single calibration, and determines the predicted fork state through the current calibration model, thereby adaptively adjusting the current calibration model according to the error covariance between the predicted fork state and the actual fork state, so as to automatically perform real-time calibration on the visual sensor through the adjusted calibration model that tends to be stable, thereby solving the problem of the need to recalibrate the visual sensor when errors occur in long-term operation, and improving the success rate of fork operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0036] Figure 1 A flow chart of a fork-based visual sensor calibration method disclosed in this application;

[0037] Figure 2 A single calibration flow chart of a visual sensor disclosed in this application;

[0038] Figure 3 A real-time calibration flow chart of a visual sensor disclosed in this application;

[0039] Figure 4 This is a schematic diagram of the structure of a fork-based visual sensor calibration device disclosed in this application;

[0040] Figure 5 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0042] The load or vibration generated by the AGV fork during operation will cause the relative position of the visual sensor and the fork to change, which will easily cause large errors when the fork performs visual tasks, affecting the success rate of the fork operation. To this end, the present application provides a fork-based visual sensor calibration method, which adjusts the calibration model through a single-calibrated visual sensor, so as to use the adjusted calibration model to perform real-time calibration on the visual sensor, thereby improving the calibration efficiency of the visual sensor.

[0043] See also Figure 1 As shown, the embodiment of the present application discloses a fork-based visual sensor calibration method, comprising:

[0044] Step S11, modeling the fork based on current point cloud data collected by a visual sensor fixed at the fork root to obtain an initial fork center point.

[0045] The fork-based visual sensor calibration method disclosed in this embodiment is applied to an industrial computer used for image processing. Specifically, the industrial computer obtains the visual sensor calibration task issued by the scheduling system, and obtains the current point cloud data collected by the visual sensor fixed at the fork root based on the visual sensor calibration task, so as to model the fork based on the current point cloud data, and determine the initial fork center point according to the modeled fork. Among them, the visual sensor includes but is not limited to a 3D (three-dimensiona) camera or a binocular depth camera; and the visual sensor is fixed at the fork root through a fork root visual sensor fixing device.

[0046] Furthermore, in order to obtain the initial fork center point, the current point cloud data collected by the visual sensor fixed at the fork root can be firstly obtained by using an industrial computer for image processing, and the target point cloud data of the fork can be determined from the current point cloud data according to the relative position relationship between the visual sensor and the fork; then the fork is modeled based on the target point cloud data and the prior parameters of the fork, and the initial fork center point is determined according to the modeled fork. The prior parameters include but are not limited to the shape and size of the fork.

[0047] It is understandable that, since the above-mentioned visual sensor is fixed at the root of the fork, the current point cloud data collected by the visual sensor includes the point cloud data of the fork itself and the point cloud data of other objects within the field of view of the visual sensor. In order to model the fork, it is necessary to identify the point cloud data of the fork from the above-mentioned current point cloud data, that is, the target point cloud data; and since the above-mentioned target point cloud data may be only part of the point cloud data of the fork, not all of the point cloud data of the fork, it is necessary to further model the fork based on the prior parameters such as the shape and size of the fork combined with the target point cloud data of the fork to obtain the modeled fork. Among them, the method of modeling the fork includes but is not limited to a grid reconstruction method and a surface fitting method.

[0048] Furthermore, after the fork is modeled, the initial fork center point can be determined by calculating the center of gravity, geometric center or other suitable center point of the fork model.

[0049] Step S12: determining a single calibration parameter by using the relationship between the initial fork center point and the target coordinate system established based on the visual sensor, so as to perform a single calibration on the visual sensor.

[0050] In this embodiment, after obtaining the initial fork center point, the deflection angle of the initial fork center point relative to the longitudinal coordinate axis in the target coordinate system can be determined to obtain the corresponding rotation matrix; correspondingly, the deflection angle of the initial fork center point relative to the transverse coordinate axis in the target coordinate system can be determined to obtain the corresponding transfer matrix; and the above rotation matrix and transfer matrix are used as single calibration parameters. The above target coordinate system is a coordinate system established based on a visual sensor.

[0051] Furthermore, the process of performing a single calibration on the visual sensor using the single calibration parameters includes: multiplying the current point cloud data collected by the visual sensor using the above rotation matrix to correct the longitudinal coordinate axis so that it is parallel to the fork; correspondingly, multiplying the current point cloud data collected by the visual sensor using the above transfer matrix to correct the transverse coordinate axis so that it is perpendicular to the fork, thereby completing a single calibration of the visual sensor.

[0052] Step S13, adjusting the current calibration model according to the target state error covariance between the predicted fork state determined by the current calibration model and the actual fork state determined by the visual sensor after a single calibration, and if the adjusted calibration model tends to be stable, calculating the real-time calibration parameters through the adjusted calibration model to perform real-time calibration on the visual sensor; wherein the fork state includes the position information and speed information of the fork center point.

[0053] In this embodiment, the actual fork state is determined by a visual sensor after a single calibration, and the predicted fork state is determined by the current calibration model; wherein the fork state includes the position information and speed information of the center point of the fork. Then, the target state error covariance between the predicted fork state and the actual fork state is calculated, and the current calibration model is adaptively adjusted according to the target state error covariance, so as to automatically calculate the real-time calibration parameters using the adjusted calibration model that tends to be stable, so as to perform real-time calibration on the visual sensor according to the real-time calibration parameters. The above calibration model can be a model constructed using a target detection algorithm based on deep learning, or a model constructed using a Kalman filter algorithm.

[0054] In a specific embodiment, when the calibration model is a model constructed using a Kalman filter algorithm, the adjustment of the current calibration model may include: determining the prior fork state at the current moment based on the fork state at the previous moment through the current calibration model, and determining the prior state error covariance at the current moment based on the state error covariance at the previous moment and the process noise covariance; then determining the Kalman gain based on the prior state error covariance at the current moment and the measurement noise covariance, and determining the predicted fork state at the current moment using the Kalman gain, the prior fork state at the current moment and the actual fork state at the current moment; wherein the actual fork state is the fork state determined according to the visual sensor after a single calibration; further, determining the target state error covariance at the current moment between the predicted fork state and the actual fork state based on the Kalman gain and the prior state error covariance at the current moment, so as to adjust the current calibration model using the target state error covariance at the current moment.

[0055] Among them, the fork state at the previous moment is the fork state corresponding to the initial fork center point at the initial moment, and is the predicted fork state at other moments; the state error covariance at the previous moment is the preset state error covariance at the initial moment, and is the target state error covariance at other moments.

[0056] When the calibration model is a model constructed using the Kalman filter algorithm, the calibration model can be expressed as:

[0057] ;

[0058] in, is the prior fork state at time k; F is the state transfer matrix; is the fork state at time k-1; B is the control matrix; is the acceleration at time k-1; is the Gaussian noise at time k; is the prior state error covariance at time k; is the state error covariance at time k-1; T represents the transpose of the matrix; Q is the process noise covariance; K is the Kalman gain; H is the observation transfer matrix, and R is the measurement noise covariance; is the predicted fork state at time k; is the actual fork state at time k; is the target state error covariance at time k; I is the unit matrix.

[0059] After obtaining the target state error covariance at the current moment, the current calibration model is adjusted using the target state error covariance at the current moment, and it is determined whether the adjusted calibration model tends to be stable. If the adjusted calibration model does not tend to be stable, the adjusted calibration model is determined as the new current calibration model, and the process jumps again to step S13 to adjust the current calibration model based on the target state error covariance between the predicted fork state determined by the current calibration model and the actual fork state determined by the visual sensor after a single calibration; if the adjusted calibration model tends to be stable, the real-time calibration parameters are calculated using the adjusted calibration model to perform real-time calibration on the visual sensor using the real-time calibration parameters.

[0060] Among them, the current calibration model is adjusted using the target state error covariance at the current moment. Specifically, the process noise covariance and measurement noise covariance in the current calibration model are adjusted using the target state error covariance at the current moment. If the adjusted process noise covariance and measurement noise covariance tend to converge, it is determined that the adjusted calibration model tends to be stable.

[0061] When the adjusted calibration model tends to be stable, the rotation matrix and the transfer matrix can be calculated in real time through the adjusted calibration model tends to be stable to obtain real-time calibration parameters, so as to perform real-time calibration on the visual sensor using the real-time calibration parameters.

[0062] As can be seen from the above, the present application models the fork through the current point cloud data collected by the visual sensor to determine the center point of the fork based on the modeled fork, and thus uses the relationship between the center point of the fork and the coordinate system of the visual sensor to perform a single calibration on the visual sensor, which not only avoids the problem of the need for special tooling when calibrating the visual sensor using specific tooling, thereby reducing costs, but also avoids the problem of the need for manual maintenance of the QR code when calibrating the visual sensor using a QR code, thereby reducing manpower consumption; further, the present application determines the actual fork state through the visual sensor after a single calibration, and determines the predicted fork state through the current calibration model, thereby adaptively adjusting the current calibration model according to the error covariance between the predicted fork state and the actual fork state, so as to automatically perform real-time calibration on the visual sensor through the adjusted calibration model that tends to be stable, thereby solving the problem of the need to recalibrate the visual sensor when errors occur in long-term operation, and improving the success rate of fork operations.

[0063] See also Figure 2 and Figure 3 As shown, the embodiment of the present application discloses a fork-based visual sensor calibration method, comprising:

[0064] During the single calibration of the visual sensor, the industrial computer obtains the visual sensor calibration task issued by the scheduling system, and obtains the current point cloud data collected by the visual sensor fixed at the fork root based on the visual sensor calibration task, and then performs point cloud filtering on the current point cloud data according to the relative position relationship between the visual sensor and the fork to obtain the target point cloud data of the fork, and then models the fork based on the target point cloud data and prior parameters such as the size of the fork, so as to determine the initial fork center point according to the modeled fork. Further, the deflection angle of the initial fork center point relative to the longitudinal coordinate axis in the target coordinate system of the visual sensor is calculated to obtain the corresponding rotation matrix, and the deflection angle of the initial fork center point relative to the transverse coordinate axis in the target coordinate system of the visual sensor is calculated to obtain the corresponding transfer matrix, and then the single calibration parameters are obtained based on the rotation matrix and the transfer matrix, so as to perform a single calibration on the visual sensor using the single calibration parameters.

[0065] In the process of real-time calibration of visual sensors based on real-time tracking algorithms (such as Kalman filter algorithms), the prior fork state at the current moment is determined based on the fork state at the previous moment through the current calibration model, and the prior state error covariance at the current moment is determined based on the state error covariance at the previous moment and the process noise covariance; then the Kalman gain is determined based on the prior state error covariance at the current moment and the measurement noise covariance, and the predicted fork state at the current moment is determined using the Kalman gain, the prior fork state at the current moment, and the actual fork state at the current moment determined by the visual sensor after a single calibration; further, the target state error covariance at the current moment between the predicted fork state and the actual fork state is determined based on the Kalman gain and the prior state error covariance at the current moment. Among them, the fork state at the initial moment is the fork state corresponding to the initial fork center point, and the state error covariance at the initial moment is the preset state error covariance.

[0066] After obtaining the target state error covariance at the current moment, the process noise covariance and the measurement noise covariance in the current calibration model are adjusted using the target state error covariance at the current moment, and then it is determined whether the adjusted process noise covariance and the measurement noise covariance tend to converge. If the adjusted process noise covariance and the measurement noise covariance do not tend to converge, the adjusted calibration model determined based on the adjusted process noise covariance and the measurement noise covariance is determined as the new current calibration model, and the step of determining the prior fork state at the current moment based on the fork state at the previous moment through the current calibration model is jumped again. If the adjusted process noise covariance and the measurement noise covariance tend to converge, it is determined that the adjusted calibration model determined based on the adjusted process noise covariance and the measurement noise covariance tends to be stable, and the adjusted calibration model that tends to be stable is used to calculate the real-time calibration parameters, so as to perform real-time calibration on the visual sensor according to the real-time calibration parameters.

[0067] As can be seen from the above, the present application models the fork through the current point cloud data collected by the visual sensor to determine the center point of the fork based on the modeled fork, and thus uses the relationship between the center point of the fork and the coordinate system of the visual sensor to perform a single calibration on the visual sensor, which not only avoids the problem of the need for special tooling when calibrating the visual sensor using specific tooling, thereby reducing costs, but also avoids the problem of the need for manual maintenance of the QR code when calibrating the visual sensor using a QR code, thereby reducing manpower consumption; further, the present application determines the actual fork state through the visual sensor after a single calibration, and determines the predicted fork state through the current calibration model, thereby adaptively adjusting the current calibration model according to the error covariance between the predicted fork state and the actual fork state, so as to automatically perform real-time calibration on the visual sensor through the adjusted calibration model that tends to be stable, thereby solving the problem of the need to recalibrate the visual sensor when errors occur in long-term operation, and improving the success rate of fork operations.

[0068] See also Figure 4 As shown, the embodiment of the present application also discloses a fork-based visual sensor calibration device, including:

[0069] A center point acquisition module 11 is used to model the fork based on the current point cloud data collected by the visual sensor fixed at the fork root to obtain the initial fork center point;

[0070] A single calibration module 12, used to determine a single calibration parameter by using the relationship between the initial fork center point and the target coordinate system established based on the visual sensor, so as to perform a single calibration on the visual sensor;

[0071] The real-time calibration module 13 is used to adjust the current calibration model according to the target state error covariance between the predicted fork state determined by the current calibration model and the actual fork state determined by the visual sensor after a single calibration. If the adjusted calibration model tends to be stable, the real-time calibration parameters are calculated by the adjusted calibration model to perform real-time calibration on the visual sensor; wherein the fork state includes the position information and speed information of the fork center point.

[0072] As can be seen from the above, the present application models the fork through the current point cloud data collected by the visual sensor to determine the center point of the fork based on the modeled fork, and thus uses the relationship between the center point of the fork and the coordinate system of the visual sensor to perform a single calibration on the visual sensor, which not only avoids the problem of the need for special tooling when calibrating the visual sensor using specific tooling, thereby reducing costs, but also avoids the problem of the need for manual maintenance of the QR code when calibrating the visual sensor using a QR code, thereby reducing manpower consumption; further, the present application determines the actual fork state through the visual sensor after a single calibration, and determines the predicted fork state through the current calibration model, thereby adaptively adjusting the current calibration model according to the error covariance between the predicted fork state and the actual fork state, so as to automatically perform real-time calibration on the visual sensor through the adjusted calibration model that tends to be stable, thereby solving the problem of the need to recalibrate the visual sensor when errors occur in long-term operation, and improving the success rate of fork operations.

[0073] In some specific implementations, the center point acquisition module 11 includes:

[0074] A data determination unit, used to obtain current point cloud data collected by a visual sensor fixed at the fork root of the fork, and determine target point cloud data of the fork from the current point cloud data according to the relative position relationship between the visual sensor and the fork;

[0075] A center point determination unit is used to model the fork based on the target point cloud data and the priori parameters of the fork, and determine an initial fork center point according to the modeled fork.

[0076] In some specific embodiments, the single calibration module 12 includes:

[0077] a first matrix determination unit, used to determine the deflection angle of the initial fork center point relative to the longitudinal coordinate axis in the target coordinate system to obtain a corresponding rotation matrix; the target coordinate system is a coordinate system established based on the visual sensor;

[0078] a second matrix determination unit, used to determine a deflection angle of the initial fork center point relative to a transverse coordinate axis in the target coordinate system to obtain a corresponding transfer matrix;

[0079] A parameter determination unit is used to determine a single calibration parameter based on the rotation matrix and the transfer matrix.

[0080] In some specific embodiments, the current calibration model is a model constructed using a target detection algorithm based on deep learning.

[0081] In some specific embodiments, the current calibration model is a model constructed using a Kalman filter algorithm;

[0082] Accordingly, the real-time calibration module 13 includes:

[0083] A covariance determination unit, used to determine the prior fork state at the current moment based on the fork state at the previous moment through the current calibration model, and determine the prior state error covariance at the current moment according to the state error covariance at the previous moment and the process noise covariance;

[0084] a state determination unit, for determining a Kalman gain based on the prior state error covariance and the measurement noise covariance, and determining a predicted fork state at a current moment by using the Kalman gain, the prior fork state and the actual fork state at a current moment; the actual fork state is the fork state determined according to the visual sensor after a single calibration;

[0085] The model adjustment submodule is used to determine the target state error covariance between the predicted fork state and the actual fork state at the current moment according to the Kalman gain and the prior state error covariance, so as to adjust the current calibration model.

[0086] In some specific implementations, the fork state at the initial moment is the fork state corresponding to the initial fork center point; and the state error covariance at the initial moment is a preset state error covariance.

[0087] In some specific implementations, the model adjustment submodule includes:

[0088] A covariance adjustment unit, configured to adjust the process noise covariance and the measurement noise covariance in the current calibration model according to the target state error covariance;

[0089] A determination unit is used to determine that the adjusted calibration model tends to be stable if the adjusted process noise covariance and the measurement noise covariance tend to converge.

[0090] Furthermore, the present application also discloses an electronic device. Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0091] Figure 5The present invention provides a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the fork-based visual sensor calibration method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0092] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0093] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0094] The operating system 221 is used to manage and control the hardware devices and computer program 222 on the electronic device 20, and can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program that can be used to complete the fork-based visual sensor calibration method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include computer programs that can be used to complete other specific tasks.

[0095] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the fork-based visual sensor calibration method disclosed above is implemented. The specific steps of the method can refer to the corresponding contents disclosed in the above embodiments, and will not be repeated here.

[0096] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0097] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0098] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0099] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0100] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A fork-based visual sensor calibration method, characterized in that: include: Modeling the fork based on current point cloud data collected by a visual sensor fixed at the fork root to obtain an initial fork center point; Determine a single calibration parameter by using the relationship between the initial fork center point and the target coordinate system established based on the visual sensor to perform a single calibration on the visual sensor; The current calibration model is adjusted according to the target state error covariance between the predicted fork state determined by the current calibration model and the actual fork state determined by the visual sensor after a single calibration. If the adjusted calibration model tends to be stable, the real-time calibration parameters are calculated by the adjusted calibration model to perform real-time calibration on the visual sensor; wherein the fork state includes position information and speed information of the center point of the fork.

2. The fork-based visual sensor calibration method according to claim 1, characterized in that: The fork is modeled based on the current point cloud data collected by the visual sensor fixed at the fork root to obtain the initial fork center point, including: Acquire current point cloud data collected by a visual sensor fixed at the fork root of the fork, and determine target point cloud data of the fork from the current point cloud data according to the relative position relationship between the visual sensor and the fork; The fork is modeled based on the target point cloud data and the priori parameters of the fork, and an initial fork center point is determined according to the modeled fork.

3. The fork-based visual sensor calibration method according to claim 1, characterized in that: The method of determining a single calibration parameter by using the relationship between the initial fork center point and the target coordinate system established based on the visual sensor includes: Determine the deflection angle of the initial fork center point relative to the longitudinal coordinate axis in the target coordinate system to obtain a corresponding rotation matrix; the target coordinate system is a coordinate system established based on the visual sensor; Determine the deflection angle of the initial fork center point relative to the transverse coordinate axis in the target coordinate system to obtain a corresponding transfer matrix; A single calibration parameter is determined based on the rotation matrix and the transfer matrix.

4. The fork-based visual sensor calibration method according to any one of claims 1 to 3, characterized in that: The current calibration model is a model constructed using a target detection algorithm based on deep learning.

5. The fork-based visual sensor calibration method according to any one of claims 1 to 3, characterized in that: The current calibration model is a model constructed using a Kalman filter algorithm; Accordingly, the target state error covariance between the predicted fork state determined by the current calibration model and the actual fork state determined by the visual sensor after a single calibration is used to adjust the current calibration model, including: Determine the prior fork state at the current moment based on the fork state at the previous moment through the current calibration model, and determine the prior state error covariance at the current moment based on the state error covariance at the previous moment and the process noise covariance; Determine the Kalman gain based on the prior state error covariance and the measurement noise covariance, and determine the predicted fork state at the current moment by using the Kalman gain, the prior fork state and the actual fork state at the current moment; the actual fork state is the fork state determined according to the visual sensor after a single calibration; The target state error covariance at the current moment between the predicted fork state and the actual fork state is determined according to the Kalman gain and the prior state error covariance, so as to adjust the current calibration model.

6. The fork-based visual sensor calibration method according to claim 5, characterized in that: The fork state at the initial moment is the fork state corresponding to the initial fork center point; and the state error covariance at the initial moment is a preset state error covariance.

7. The fork-based visual sensor calibration method according to claim 5, characterized in that: The adjusting the current calibration model includes: adjusting the process noise covariance and the measurement noise covariance in the current calibration model according to the target state error covariance; If the adjusted process noise covariance and the measurement noise covariance tend to converge, it is determined that the adjusted calibration model tends to be stable.

8. A visual sensor calibration device based on a fork, characterized in that: include: A center point acquisition module, used to model the fork based on current point cloud data collected by a visual sensor fixed at the fork root, so as to obtain an initial fork center point; A single calibration module, used to determine a single calibration parameter by using the relationship between the initial fork center point and the target coordinate system established based on the visual sensor, so as to perform a single calibration on the visual sensor; A real-time calibration module is used to adjust the current calibration model according to the target state error covariance between the predicted fork state determined by the current calibration model and the actual fork state determined by the visual sensor after a single calibration. If the adjusted calibration model tends to be stable, the real-time calibration parameters are calculated by the adjusted calibration model to perform real-time calibration on the visual sensor; wherein the fork state includes the position information and speed information of the fork center point.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the fork-based visual sensor calibration method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, which, when executed by a processor, implements the fork-based visual sensor calibration method as described in any one of claims 1 to 7.

Images