Method, device and storage medium for calibrating a speckle camera

Abstract

This application discloses a calibration method, device, and storage medium for a speckle camera. The calibration method includes: acquiring the angle deviation of the speckle camera under preset operating conditions; performing coordinate correction processing on the initial point cloud acquired by the speckle camera under the current operating conditions based on the angle deviation to obtain corrected coordinates; and performing extrinsic parameter calibration processing on the speckle camera based on the corrected coordinates of multiple initial point clouds to obtain the camera extrinsic parameters. The above scheme, by calibrating the extrinsic parameters of the speckle camera using the coordinate-corrected point cloud, can improve its data accuracy.

Description

Technical Field

[0001] This application relates to the field of robot calibration technology, and in particular to a calibration method, device and storage medium for a speckle camera. Background Technology

[0002] A speckle camera is a 3D imaging device based on structured light technology. It can actively project a speckle pattern using a speckle projector and capture the reflected light spots using a camera to achieve 3D measurement of a target object. Speckle cameras are commonly used in applications such as robot navigation and positioning, industrial inspection and measurement, and virtual reality and augmented reality.

[0003] However, speckle cameras are affected by actual working conditions. For example, different temperatures and / or different object distances can cause varying degrees of drift, leading to a decrease in the accuracy of target detection and recognition. Taking the scenario of robot (AGV) navigation and positioning as an example, if drift occurs when the AGV docks with a pallet rack to perform a handling task, the identified handling position of the pallet rack will deviate from the actual position.

[0004] Therefore, there is an urgent need for a calibration method for speckle cameras that can improve their data accuracy. Summary of the Invention

[0005] This application provides at least one method, apparatus, device, and computer-readable storage medium for calibrating a speckle camera.

[0006] The first aspect of this application provides a method for calibrating a speckle camera, comprising: acquiring the angle deviation of the speckle camera under a preset working condition; performing coordinate correction processing on the initial point cloud acquired by the speckle camera under the current working condition based on the angle deviation to obtain calibrated coordinates; and performing extrinsic parameter calibration processing on the speckle camera based on the calibrated coordinates of multiple initial point clouds to obtain the camera extrinsic parameters of the speckle camera.

[0007] In one embodiment, obtaining the angle deviation of the speckle camera under preset conditions includes: determining the target point cloud plane corresponding to each preset condition based on the planar point cloud information collected by the speckle camera on the preset plane under various preset conditions; and determining the angle deviation between the normal vector of the target point cloud plane and the acquisition direction of the speckle camera under each preset condition based on the target point cloud plane, wherein the acquisition direction is perpendicular to the preset plane.

[0008] In one embodiment, determining the target point cloud plane corresponding to each preset working condition based on the planar point cloud information acquired by the speckle camera on the preset plane under various preset working conditions includes: acquiring multiple frames of planar point cloud information acquired by the speckle camera on the preset plane under various preset working conditions; and performing planar fitting processing on the multiple frames of planar point cloud information to obtain multiple target point cloud planes.

[0009] In one embodiment, the step of performing coordinate correction processing on the initial point cloud acquired by the speckle camera under the current working condition based on the angle deviation includes: obtaining the initial coordinates of the currently traversed point cloud in the initial point cloud; determining at least one target working condition from multiple preset working conditions based on the current working condition and preset filtering conditions; correcting the initial coordinates based on the angle deviation of the at least one target working condition to obtain the corrected coordinates.

[0010] In one embodiment, the target operating condition includes a preset object distance and a preset temperature, and the current operating condition includes a current object distance and a current temperature. Correcting the initial coordinates based on the angle deviation of at least one target operating condition to obtain the corrected coordinates includes: responding to the presence of multiple target operating conditions, determining the angle deviation weight of each target operating condition based on the preset object distance and the preset temperature, as well as the current object distance and the current temperature; performing a weighted summation of the angle deviations of each target operating condition based on the angle deviation weights to obtain a target angle deviation; and correcting the initial coordinates based on the target angle deviation to obtain the corrected coordinates.

[0011] In one embodiment, after performing extrinsic parameter calibration on the speckle camera based on the calibration coordinates of multiple initial point clouds to obtain the camera extrinsic parameters of the speckle camera, the method further includes: acquiring a point cloud of a target to be identified; performing coordinate transformation on the point cloud of the target to be identified based on the camera extrinsic parameters to obtain a target point cloud; and determining the pose information of the target to be identified based on the target point cloud.

[0012] In one embodiment, the speckle camera is mounted on a forklift, which includes a first type of forklift and a second type of forklift. The first type of forklift has a fixed connecting plate on its fork teeth, and the plane of the fixed plate is perpendicular to the fork tooth's picking direction. The second type of forklift does not have the fixed plate on its fork teeth. After performing extrinsic parameter calibration on the speckle camera based on the correction coordinates of multiple initial point clouds to obtain the camera extrinsic parameters, the method further includes: in response to the forklift being the second type of forklift, determining the pose information of the target to be identified based on the camera extrinsic parameters and the collected point cloud of the target to be identified; in response to the forklift being the first type of forklift, determining the pose information of the target to be identified based on the camera extrinsic parameters, the collected point cloud of the fixed plate, and the point cloud of the target to be identified.

[0013] In one embodiment, determining the pose information of the target to be identified based on the camera extrinsic parameters, the acquired fixed plate point cloud, and the point cloud to be identified includes: performing planar fitting processing on the fixed plate point cloud to obtain a fixed plate plane; correcting the plane angle of the fixed plate plane based on the camera extrinsic parameters to obtain a corrected angle; performing coordinate correction processing on the fixed plate point cloud and the point cloud to be identified based on the corrected angle to obtain a corrected fixed plate point cloud and a corrected point cloud to be identified; performing coordinate transformation processing on the corrected fixed plate point cloud and the corrected point cloud to be identified based on the camera extrinsic parameters to obtain a target fixed plate point cloud and a target point cloud to be identified; and determining the pose information of the target to be identified based on the target fixed plate point cloud and the target point cloud to be identified.

[0014] A second aspect of this application provides a calibration device for a speckle camera, comprising: an acquisition module for acquiring the angle deviation of the speckle camera under a preset working condition; a calibration module for performing coordinate calibration processing on the initial point cloud acquired by the speckle camera under the current working condition based on the angle deviation to obtain calibrated coordinates; and a calibration module for performing extrinsic parameter calibration processing on the speckle camera based on the calibrated coordinates of multiple initial point clouds to obtain the camera extrinsic parameters of the speckle camera.

[0015] A third aspect of this application provides an electronic device, including a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the above-described speckle camera correction method.

[0016] The fourth aspect of this application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the above-described correction method for a speckle camera.

[0017] The above scheme, by obtaining the angle deviation of the speckle camera under preset working conditions, can find the corresponding angle deviation of the speckle camera under the current working conditions. Based on the angle deviation under the current working conditions, coordinate correction processing is performed on the initial point cloud acquired by the speckle camera under the current working conditions to obtain the coordinate-corrected point cloud and its corrected coordinates. Based on the corrected coordinates of multiple coordinate-corrected initial point clouds, extrinsic parameter calibration processing of the speckle camera is performed to obtain the accurate extrinsic parameters of the calibrated speckle camera. Using these extrinsic parameters, the point cloud data subsequently acquired by the speckle camera is projected onto a specified coordinate system, which can further locate the pose of the target object according to the point cloud recognition and detection method. This avoids the drift deviation problem of the speckle camera, improves the data accuracy of the speckle camera, and enables accurate target recognition and detection.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0020] Figure 1 This is a schematic flowchart of an exemplary embodiment of the speckle camera correction method of this application;

[0021] Figure 2 This is an exemplary scene diagram illustrating the angle deviation acquisition method in the speckle camera correction method of this application.

[0022] Figure 3 This is an exemplary forklift structure diagram in the speckle camera correction method of this application;

[0023] Figure 4 This is another exemplary forklift structure diagram in the speckle camera correction method of this application;

[0024] Figure 5 This is a block diagram illustrating a correction device for a speckle camera, as shown in an exemplary embodiment of this application;

[0025] Figure 6 This is a schematic diagram of the structure of an embodiment of the electronic device of this application;

[0026] Figure 7 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0027] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0028] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0029] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0030] To facilitate understanding, one of the applicable scenarios of this application will be illustrated by example.

[0031] Speckle cameras are affected by actual working conditions during use. For example, when a speckle camera photographs a planar object at different distances, the obtained angle of the object's plane will drift to varying degrees. Similarly, the angle of the object's plane will drift to varying degrees when photographed at different temperatures. This drift reduces the accuracy of target detection and recognition based on the point cloud data acquired by the speckle camera. Taking robot (AGV) navigation and localization as an example, when an AGV uses a monocular speckle camera to dock with a pallet rack to perform a transport task, drift will cause the identified transport position of the pallet rack to deviate from its actual position (for example, the AGV forklift may misalign the target object).

[0032] Please see Figure 1 , Figure 1 This is a schematic flowchart of an exemplary embodiment of the speckle camera correction method of this application. Specifically, it may include the following steps:

[0033] Step S110: Obtain the angle deviation of the speckle camera under preset operating conditions.

[0034] Preset operating conditions refer to pre-defined operating conditions. The operating conditions in this application are mainly set for scenarios where speckle cameras are prone to drift problems. For example, the preset operating conditions in this application are mainly set for the temperature and / or object distance when the speckle camera is working.

[0035] For example, point cloud data collected by a speckle camera on a specified object under certain preset working conditions (a certain temperature and a certain object distance) can be obtained, and the recognition pose of the specified object can be determined by recognizing these point cloud data; then the recognition pose is compared with the real pose of the specified object to obtain the angle deviation between the recognition pose and the real pose of the specified object, which is equivalent to obtaining the angle deviation of the speckle camera in recognizing the specified object under the scene of that temperature and that object distance.

[0036] Furthermore, the temperature and / or object distance of the speckle camera can be changed, and the angle deviation of the speckle camera under various temperature and / or object distance scenarios can be collected and statistically analyzed to obtain the angle deviation of the speckle camera under various preset operating conditions. The collected angle deviations under various preset operating conditions can be stored in the speckle camera or a device that has a communication connection with the speckle camera, so as to obtain the corresponding angle deviations in practical applications.

[0037] Step S120: Based on the angle deviation, perform coordinate correction processing on the initial point cloud acquired by the speckle camera under the current working conditions to obtain the corrected coordinates.

[0038] Based on the steps described above, after obtaining the angle deviations under various preset working conditions, in actual applications, the corresponding angle deviations can be found according to the current working condition of the speckle camera, and the coordinates of the initial point cloud collected by the speckle camera under the current working condition can be corrected to obtain the corrected coordinates.

[0039] For example, taking the scenario of an AGV forklift docking with a pallet and transporting goods as an example, the initial point cloud refers to the point cloud data collected by the AGV forklift during the docking process. In general scenarios, the AGV forklift needs to determine the position of the pallet through point cloud recognition technology based on the point cloud data collected during the docking process, which can provide the AGV forklift with an accurate docking method to accurately pick up the goods.

[0040] Step S130: Perform extrinsic parameter calibration on the speckle camera based on the calibration coordinates of multiple initial point clouds to obtain the camera extrinsic parameters of the speckle camera.

[0041] Among them, the speckle camera possesses extrinsic parameters. Extrinsic parameters typically refer to the transformations between coordinates in the world coordinate system and coordinates in the camera coordinate system. Extrinsic parameters include rotation matrices and translation vectors. Knowledge of extrinsic parameters is well-known in this field and will not be elaborated upon here.

[0042] This application primarily focuses on calibrating the extrinsic parameters of the speckle camera to optimize its accuracy, thereby improving the subsequent accuracy of the speckle camera in recognizing point clouds. In practical applications, the speckle camera can transform the point cloud in the world coordinate system to the point cloud in the camera coordinate system using extrinsic parameters, and then determine relevant information of the point cloud (such as point cloud pose) through point cloud recognition and other methods.

[0043] For example, methods for calibrating extrinsic parameters can refer to relevant methods in the art, which will not be elaborated here. For instance, methods for calibrating extrinsic parameters may include, but are not limited to, solving for camera extrinsic parameters by minimizing reprojection error, such as referring to the PnP (Perspective-n-Point) algorithm, obtaining the coordinates of multiple points in the world coordinate system and the camera coordinate system respectively, and then iteratively solving for camera extrinsic parameters, etc., which will not be elaborated here.

[0044] Understandably, in this application, the speckle camera corrects the coordinates of the acquired point cloud, and uses the corrected point cloud coordinates to calibrate the extrinsic parameters, thereby improving the accuracy of the extrinsic parameters. This makes the point cloud recognition process after the point cloud coordinates are converted based on the speckle camera's extrinsic parameters more accurate.

[0045] As can be seen, this application obtains the angle deviation of the speckle camera under preset working conditions, and can then find the corresponding angle deviation of the speckle camera under the current working conditions. Based on the angle deviation under the current working conditions, coordinate correction processing is performed on the initial point cloud acquired by the speckle camera under the current working conditions to obtain the coordinate-corrected point cloud and its corrected coordinates. Based on the corrected coordinates of multiple coordinate-corrected initial point clouds, extrinsic parameter calibration processing is performed on the speckle camera to obtain the accurate extrinsic parameters of the calibrated camera. Using these extrinsic parameters, the point cloud data subsequently acquired by the speckle camera is projected onto a specified coordinate system, which further enables the positioning of the target object according to the point cloud recognition and detection method. This avoids the drift deviation problem of the speckle camera, improves the data accuracy of the speckle camera, and achieves accurate target recognition and detection.

[0046] Based on the above embodiments, this application embodiment describes the steps for obtaining the angle deviation of a speckle camera under preset operating conditions. Specifically, the method of this embodiment includes the following steps:

[0047] Based on the planar point cloud information acquired by the speckle camera from the preset plane under various preset working conditions, the target point cloud plane corresponding to each preset working condition is determined; based on the target point cloud plane, the angular deviation between the normal vector of the target point cloud plane and the acquisition direction of the speckle camera under each preset working condition is determined, with the acquisition direction perpendicular to the preset plane.

[0048] Based on the foregoing embodiments, this embodiment mainly describes a method for obtaining angle deviations under various preset working conditions using a speckle camera. See also... Figure 2 As shown, Figure 2 This is an exemplary scene diagram illustrating the angle deviation acquisition method in the speckle camera correction method of this application.

[0049] For example, the speckle camera can be mounted on a guide rail via a sliding block, allowing the object distance of the speckle camera to be changed by sliding the block along the rail. A baffle can be positioned in the acquisition direction of the speckle camera to represent a preset plane (this baffle also represents the plane of the pallet that the speckle camera faces when docking with the AGV forklift), and the acquisition direction of the speckle camera (its movement direction on the guide rail) is perpendicular to the preset plane. That is, the direction from the camera to the baffle is the positive x-axis, the direction from the camera towards the ground upwards is the positive z-axis, and the direction facing the flat plate to the left is the positive y-axis.

[0050] This process is an offline calibration method. In practice, the camera is mounted on a guide rail, and both the camera and the guide rail are turned on. A heater can be used to control the camera's temperature, and a temperature measuring device can also be used to measure the camera's temperature. By slowly moving the camera on the guide rail using a slider, point clouds of the baffle can be acquired for each frame under various operating conditions (different temperatures, different object distances). Specifically, this could involve acquiring point clouds of the baffle collected by the camera at different object distances at the same temperature, and / or acquiring point clouds of the baffle collected by the camera at different temperatures at the same object distance, and / or acquiring point clouds of the baffle collected by the camera at different temperatures and different object distances; the specific method is not limited here.

[0051] Furthermore, after acquiring the baffle point cloud, the baffle point cloud information (planar point cloud information) collected under different working conditions is subjected to planar fitting processing to obtain the target point cloud plane under each working condition (which is equivalent to the recognition pose in the previous example). The direction information of the target point cloud plane under each working condition is compared with the actual direction information of the baffle to obtain the angle deviation of the speckle camera under each working condition.

[0052] Specifically, the orientation information of the target point cloud plane can be represented by the normal vector of the point cloud plane. Since the acquisition direction of the speckle camera (the direction of movement on the guide rail) is perpendicular to the preset plane (baffle), the acquisition direction of the speckle camera can be used as the actual orientation information of the baffle. Therefore, in the specific implementation process, the angular deviation between the normal vector of the point cloud plane and the acquisition direction under each working condition can be determined as the angular deviation of the speckle camera under each working condition.

[0053] Based on the above embodiments, this application embodiment describes the steps for determining the target point cloud plane corresponding to each preset working condition based on the planar point cloud information acquired by the speckle camera from a preset plane under various preset working conditions. Specifically, the method of this embodiment includes the following steps:

[0054] Acquire multiple frames of planar point cloud information captured by a speckle camera on a preset plane under various preset working conditions; perform planar fitting processing on the multiple frames of planar point cloud information to obtain multiple target point cloud planes.

[0055] The above embodiments will be described in conjunction with the following: Figure 2 This method acquires multiple frames of planar point cloud information from a speckle camera under various preset working conditions on a preset plane. Then, planar fitting processing can be performed on the point clouds in the multiple frames of planar point cloud information to obtain the target point cloud plane corresponding to the multiple frames of planar point cloud information.

[0056] For example, during the acquisition of the preset planar point cloud, the speckle camera may also acquire point clouds from other parts. Therefore, in specific implementations, point cloud filtering can be performed based on the guide rail structure, prior range, etc., to obtain point clouds belonging to the baffle. For instance, methods including but not limited to Euclidean clustering can be used to select the largest point cloud cluster block passing through the guide rail within the range, remove noise points, and thus obtain baffle point clouds at different distances.

[0057] Furthermore, by performing least-squares plane fitting on the baffle point cloud using software including but not limited to Ceres optimization, the plane equation of the target point cloud plane can be obtained, such as: ax + by + cz + d = 0. Based on the plane equation of each target point cloud plane, we obtain the angular deviation of that plane, such as: θ = atan2(b, a), which is a formula in the C++ programming language. Simultaneously, the object distance d corresponding to each target point cloud plane can be recorded. plane =fabs(d) and temperature t, from which a set of point pairs (d) corresponding to each target point cloud plane can be constructed. plane ,t,θ).

[0058] In subsequent specific application scenarios, the corresponding deviation angle can be found in the point pairs acquired based on the current working condition of the speckle camera, and this deviation angle can be used to correct the point cloud coordinates of the point cloud acquired under the current working condition.

[0059] Based on the above embodiments, this application embodiment describes the steps for performing coordinate correction processing on the initial point cloud acquired by the speckle camera under the current operating conditions according to the angle deviation. Specifically, the method of this embodiment includes the following steps:

[0060] Obtain the initial coordinates of the currently traversed point cloud in the initial point cloud; determine at least one target working condition from multiple preset working conditions based on the current working condition and preset filtering conditions; correct the initial coordinates based on the angle deviation of at least one target working condition to obtain the corrected coordinates.

[0061] Illustrated in combination with the foregoing embodiments, during the process of correcting the point cloud coordinates, for each point (x, y, z) in the original point cloud, at least one target working condition can be determined from the point pairs of the preset working conditions based on the preset screening conditions with the current working condition as the reference, and then the initial coordinates can be corrected according to the angular deviations of at least one target working condition to obtain the corrected coordinates. Among them, the preset screening conditions refer to the conditions for screening according to the temperature and / or object distance when the speckle camera collects the point cloud.

[0062] Exemplarily, for the point cloud (x, y, z), the object distance of the speckle camera at this time is x, and the temperature at this time is t. Thus, according to the preset screening conditions, at least one (e.g., 4) target working condition points closest to the x value can be queried from the point pairs of the preset working conditions. For example, the preset screening conditions can be d plane0 <x<d plane1 , t0<t<t1, that is, select a target working condition point (d plane0 , t0, θ plane0 ) when the object distance is d 00 and the temperature is t0, select a target working condition point (d plane0 , t1, θ plane0 ) when the object distance is d 01 and the temperature is t1, select a target working condition point (d plane1 , t0, θ plane1 ) when the object distance is d 10 , and select a target working condition point (d plane1 , t1, θ plane1 [[ID=二十九]] when the object distance is d 11 ). That is, the angular deviations θ 00 , θ 01 , θ 10 , θ 11 corresponding to the four target working conditions closest to the working condition of the currently traversed point cloud (x, y, z) are obtained.

[0063] Furthermore, the final angular deviation θ can be determined according to the angular deviations of multiple target working conditions through a preset calculation method (such as bilinear interpolation, weighted summation, average value calculation, etc.) to correct the initial coordinates (x, y, z) to obtain the corrected coordinates. For example, the mathematical expression of the corrected point cloud coordinates can be exemplified as:

[0064] x modify =x + ytan(θ)

[0065]

[0066] z modify =z

[0067] Then, the rotation and translation matrix (camera extrinsic parameters) is calibrated using the corrected point cloud coordinates to obtain the camera extrinsic parameters (x, y, z, roll, pitch, yaw) in the vehicle coordinate system. In fact, the camera extrinsic parameters obtained by calibration according to the above example of this application also include angle information compared with the extrinsic parameters obtained by traditional calibration methods.

[0068] Based on the above embodiments, this application embodiment describes the steps of correcting the initial coordinates according to the angle deviation of at least one target working condition, including the current object distance and the current temperature, to obtain the corrected coordinates. The target working conditions include a preset object distance and a preset temperature. Specifically, the method of this embodiment includes the following steps:

[0069] Since there are multiple target working conditions, the angle deviation weight of each target working condition is determined based on the preset object distance and preset temperature of each target working condition, as well as the current object distance and current temperature. The angle deviations of each target working condition are weighted and summed according to the angle deviation weight to obtain the target angle deviation. The initial coordinates are corrected according to the target angle deviation to obtain the corrected coordinates.

[0070] In conjunction with the foregoing embodiments, in the specific implementation of this application, one or more target working conditions can be determined from preset working conditions based on flexibly set preset screening conditions, and then the angle deviation used for coordinate correction can be determined based on the angle deviation of one or more target working conditions.

[0071] For example, if the coordinates of the currently traversed point cloud are (x, y, z), and there are multiple target working conditions (taking the four target working conditions in the previous example as an example), then the angle deviation weight of each target working condition can be determined based on the preset object distance and preset temperature of each target working condition, as well as the current object distance and current temperature. Then, the angle deviations of each target working condition can be weighted and summed according to the angle deviation weights to obtain the target angle deviation. The mathematical expression of the target angle deviation can be:

[0072]

[0073]

[0074] Another example is that the target working condition can be selected from multiple preset working conditions as the one that is closest to the currently traversed point cloud, and the angle deviation of the target working condition can be directly used to correct the coordinates of the currently traversed point cloud. This will not be elaborated here.

[0075] Based on the above embodiments, this application embodiment describes the steps after performing extrinsic parameter calibration on the speckle camera according to the correction coordinates of multiple initial point clouds to obtain the camera extrinsic parameters of the speckle camera. Specifically, the method of this embodiment includes the following steps:

[0076] Acquire the point cloud of the target to be identified; perform coordinate transformation on the point cloud of the target to be identified based on the camera extrinsic parameters to obtain the target point cloud; determine the pose information of the target to be identified based on the target point cloud.

[0077] Based on the foregoing embodiments, after obtaining the extrinsic parameters of the speckle camera, the speckle camera can be used to collect point clouds, and the collected point clouds can be analyzed and identified to obtain point cloud-related information.

[0078] For example, in the pallet rack recognition process of an AGV forklift, the pallet rack is equivalent to a target to be recognized. The pallet rack recognition is initiated when the forklift moves to the recognition point in front of the pallet rack. A speckle camera is used to acquire the point cloud of the target to be recognized; then, coordinate transformation is performed on the point cloud based on the camera's extrinsic parameters to obtain the target point cloud; and the initial pose information of the target to be recognized is determined based on the target point cloud.

[0079] The method for determining the pose information of the target to be identified based on the target point cloud can refer to relevant processing methods in this field, and will not be elaborated here. For example, the geometric center of the target to be identified can be determined by point cloud clustering, and the pose of the pallet legs can be determined by calculating the normal vector; or the pose can be determined by deep learning methods such as point cloud template matching, etc., without limitation. After the target pose is identified, the forklift can move to pick up the pallet shelf and perform tasks such as goods handling.

[0080] Based on the above embodiments, this application embodiment describes the steps after calibrating the speckle camera's extrinsic parameters according to the correction coordinates of multiple initial point clouds to obtain the camera's extrinsic parameters. The speckle camera is mounted on a forklift, which includes a first type of forklift and a second type of forklift. The first type of forklift has a fixed connecting plate on its fork teeth, and the plane of the connecting plate is perpendicular to the fork tooth's picking direction. The second type of forklift does not have a fixed connecting plate on its fork teeth. Specifically, the method of this embodiment includes the following steps:

[0081] In response to the forklift being a type 2 forklift, the pose information of the target to be identified is determined based on the camera extrinsic parameters and the collected point cloud of the target to be identified; in response to the forklift being a type 1 forklift, the pose information of the target to be identified is determined based on the camera extrinsic parameters, the collected point cloud of the fixed plate, and the point cloud of the target to be identified.

[0082] Referring to the foregoing embodiments, in scenarios where conditions permit, the mobile device equipped with the speckle camera can also be modified accordingly. For example, taking an AGV forklift as an example, a fixedly connected baffle (fixed plate) can be installed on the forklift's forks. See also... Figure 3 As shown, Figure 3This is an exemplary forklift structure diagram of the speckle camera correction method of this application. When the target object picked up by the forklift is only a two-legged shelf, a large retaining plate (retaining baffle) can be added to the end of the two fork teeth. Alternatively, refer to... Figure 4 As shown, Figure 4 This is another exemplary forklift structure diagram in the speckle camera correction method of this application. When the target object picked up by the forklift includes a three-legged rack, a small fixing plate can be added to the end of each of the two fork teeth (to prevent the middle leg of the three-legged rack from contacting the forklift during movement). Figure 3 (See diagram showing baffle conflict). By installing baffles on the fork teeth, the forklift can have more reference objects when performing point cloud recognition during pallet docking, thereby improving the accuracy of point cloud recognition.

[0083] For example, for ease of explanation, a forklift with a connecting plate on the fork teeth is referred to as a first type forklift. Figure 3 and Figure 4 The forklifts shown all belong to the first type of forklift. Forklifts without a fixing plate on the fork teeth are called the second type of forklift (common AGV forklifts). The speckle camera calibration method of this application can have different specific implementation processes for different types of forklifts.

[0084] For example, in scenarios where forklifts have already been manufactured and there is no opportunity to modify their hardware (i.e., the forklift is a Type II forklift), the offline calibration scheme described in the previous embodiments can be used. In scenarios where pallet rack recognition is performed, the camera drift angle (angle deviation) of the speckle camera at various object distances and temperatures has been determined in advance through offline calibration. Therefore, the point cloud collected by the speckle camera under the current operating conditions (different temperatures and different object distances) can be corrected by using a forklift-as-moving recognition method. This also avoids the camera drift problem and achieves good correction results.

[0085] For example, if the forklift is a Type 1 forklift, in the pallet rack recognition process, after the forklift recognizes the initial pose of the pallet rack, the secondary recognition point of the forklift can be calculated based on the initial pose, and the forklift can be controlled to move to the secondary recognition point to perform secondary recognition of the pallet rack. The moving distance of the forklift during the secondary recognition process is approximately the distance between the initial position of the pallet rack and the speckle camera minus the distance between the fixed plate and the camera (the specific distance requirement can be set as needed and is not limited here), which means that the fixed plate should be as close as possible to the plane of the pallet rack. When the forklift moves to a position where the fixed plate and the pallet plane are roughly level, the forklift can pause its movement and start secondary recognition; or, in scenarios where efficiency is paramount, a method of performing secondary recognition while moving can be chosen. For example, a secondary recognition area can be pre-defined within a certain area before and after the fixed plate. When the forklift moves, as soon as it enters the secondary recognition area, the secondary recognition process can be started, and secondary recognition can be performed using a method of moving while recognizing. In the secondary recognition area, the forklift's moving speed can be reduced (or made lower than a preset speed threshold), and slow-speed recognition can be adopted while moving in the secondary recognition area.

[0086] Furthermore, during the secondary recognition process, the camera can acquire the fixed plate point cloud and the target point cloud of the stack. Then, similarly following the method described in the previous embodiment, the plane angle of the fixed plate point cloud plane (fixed plate plane) can be determined using the fixed plate point cloud. Based on the difference between the plane angle of the fixed plate and the yaw parameter of the speckle camera, the true angle of the fixed plate is determined. This true angle is then used as the current angle deviation of the speckle camera to correct the coordinates of each point cloud, achieving real-time calibration and avoiding angle drift caused by installation errors of the speckle camera. Subsequently, the corrected point cloud is transformed to a specified coordinate system based on the camera extrinsic parameters, and the pose information of the target to be identified is determined through point cloud recognition.

[0087] Based on the above embodiments, this application embodiment describes the steps for determining the pose information of the target to be identified based on camera extrinsic parameters, the collected point cloud of the fixed plate, and the point cloud to be identified. Specifically, the method of this embodiment includes the following steps:

[0088] Plane fitting is performed on the fixed plate point cloud to obtain the fixed plate plane; the plane angle of the fixed plate plane is corrected according to the camera extrinsic parameters to obtain the corrected angle; coordinate correction is performed on the fixed plate point cloud and the point cloud to be identified according to the corrected angle to obtain the corrected fixed plate point cloud and the corrected point cloud to be identified; coordinate transformation is performed on the corrected fixed plate point cloud and the corrected point cloud to be identified according to the camera extrinsic parameters to obtain the target fixed plate point cloud and the target point cloud to be identified; the pose information of the target to be identified is determined based on the target fixed plate point cloud and the target point cloud to be identified.

[0089] In conjunction with the foregoing embodiments, this embodiment provides an example of the online calibration process in the forklift identification of pallet racks.

[0090] During the identification process using the fixed plate, the fixed plate area can be cropped from the acquired point cloud based on the forklift structure and the position of the fixed plate (e.g., Figure 3 A whole baffle or Figure 4 The point clouds of the two baffles in the middle are used to obtain the point cloud of the fixed plate, such as (x p ,y p ,z p Similarly, methods including but not limited to Ceres can be used to perform least-squares fitting of the plane to obtain the plane angle θ of the fixed plate. The plane angle of the fixed plate is then corrected based on the speckle camera extrinsic parameter yaw to obtain the corrected angle (true angle) θ. true Its mathematical expression can be:

[0091] θ true =θ-yaw

[0092] θ true This refers to the angular drift at a fixed distance of the speckle camera, excluding installation errors. The point cloud of the object to be identified (a three-legged pallet or a two-legged shelf) is extracted (i.e., the point cloud to be identified is obtained). For the object to be identified and the baffle, the steps of the aforementioned embodiments can be used, with θ... true Point cloud coordinates are corrected to account for angular drift (angular deviation), resulting in corrected fixed plate point clouds and corrected point clouds to be identified. For example, for the fixed plate point cloud (x... p ,y p ,z p The mathematical expression for the correction method of ) can be:

[0093] x modify =x p +y p tan(θtrue)

[0094]

[0095] z modify =z p

[0096] Furthermore, based on the camera extrinsic parameters (x, y, z, roll, pitch, yaw), the corrected fixed plate point cloud and the corrected point cloud to be identified can be transformed into the vehicle coordinate system. The lateral coordinate y of the center point of the fixed plate point cloud block in the vehicle coordinate system is then calculated. board Theoretically, the fixed plate is perfectly symmetrical with respect to the vehicle body, that is, theoretically y board It is 0. But in reality, due to various factors such as vehicle wear and tear, y boardIt may not be zero. To improve recognition accuracy, at this point, we can choose to subtract y from the y-value of the point cloud to be recognized and the y-value of the fixed plate point cloud. board This is to achieve real-time calibration.

[0097] Subsequently, the target pose of the object to be identified (pallet rack) can be determined by combining the point cloud coordinates calibrated in the vehicle coordinate system with methods including but not limited to point cloud recognition in the aforementioned example. The forklift then moves to the target position and picks up the pallet rack.

[0098] Based on the above embodiments, it should also be noted that, during the offline calibration process of the aforementioned examples, under constant temperature conditions, the scheme can be adaptively adjusted to only correct the speckle camera angle drift at different object distances. Similarly, under constant object distance conditions, the scheme can be adaptively adjusted to only correct the speckle camera angle drift at different temperatures, which will not be elaborated here.

[0099] In summary, the speckle camera correction method of this application can correct the angular drift of the speckle camera by fitting a point cloud to a plane using a guide rail at different distances and temperatures. Online angular drift correction of the speckle camera can also be performed by modifying the fork tooth design. Furthermore, by designing a secondary recognition scheme when the speckle camera is at the same distance from the pallet shelf and when it is at the same distance from the fixed plate plane, the impact of operating conditions such as temperature and object distance on the speckle camera's recognition on the pallet shelf is reduced.

[0100] Since a speckle camera is a black box to the user, users can only obtain the point cloud data acquired by the camera, but cannot obtain the image information inside the camera. The method in this application only processes the point cloud, and can be fully applied to perform angle drift correction on a speckle camera whose internal structure is completely unknown.

[0101] It should be further noted that the entity executing the speckle camera calibration method can be the speckle camera calibration device. For example, the speckle camera calibration method can be executed by a terminal device, server, or other processing device. The terminal device can be a user equipment (UE), computer, mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. In some possible implementations, the speckle camera calibration method can be implemented by a processor calling computer-readable instructions stored in memory.

[0102] Figure 5 This is a block diagram illustrating a speckle camera correction device according to an exemplary embodiment of this application. Figure 5As shown, the exemplary speckle camera correction device 500 includes: an acquisition module 510, a correction module 520, and a calibration module 530. Specifically:

[0103] The acquisition module 510 is used to acquire the angle deviation of the speckle camera under preset working conditions.

[0104] The correction module 520 is used to perform coordinate correction processing on the initial point cloud acquired by the speckle camera under the current working conditions based on the angle deviation, so as to obtain the corrected coordinates.

[0105] The calibration module 530 is used to perform extrinsic parameter calibration on the speckle camera based on the calibration coordinates of multiple initial point clouds, so as to obtain the camera extrinsic parameters of the speckle camera.

[0106] In this exemplary speckle camera calibration device, by acquiring the angle deviation of the speckle camera under preset working conditions, the corresponding angle deviation of the speckle camera under the current working conditions can be found. Based on the angle deviation under the current working conditions, coordinate correction processing is performed on the initial point cloud acquired by the speckle camera under the current working conditions to obtain the coordinate-corrected point cloud and its corrected coordinates. Based on the corrected coordinates of multiple coordinate-corrected initial point clouds, extrinsic parameter calibration processing is performed on the speckle camera to obtain the accurate extrinsic parameters of the calibrated camera. Using these extrinsic parameters, the point cloud data subsequently acquired by the speckle camera is projected onto a specified coordinate system, enabling further positioning of the target object's pose using a point cloud recognition and detection method. This avoids the drift deviation problem of the speckle camera, improves the data accuracy of the speckle camera, and achieves accurate target recognition and detection.

[0107] It should be noted that the apparatus and method provided in the above embodiments belong to the same concept, and the specific ways in which each module and unit performs operations have been described in detail in the method embodiments, and will not be repeated here. In practical applications, the apparatus provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above, and this is not a limitation.

[0108] The functions of each module can be found in the implementation examples of the speckle camera correction method, and will not be repeated here.

[0109] Please see Figure 6 , Figure 6This is a schematic diagram of the structure of an embodiment of the electronic device of this application. The electronic device 100 includes a memory 101 and a processor 102. The processor 102 is used to execute program instructions stored in the memory 101 to implement the steps in any of the above-described embodiments of the speckle camera correction method. In a specific implementation scenario, the electronic device 100 may include, but is not limited to, a microcomputer or a server. In addition, the electronic device 100 may also include mobile devices such as laptops and tablets, which are not limited here.

[0110] Specifically, processor 102 controls itself and memory 101 to implement the steps in any of the above-described speckle camera correction method embodiments. Processor 102 can also be referred to as a CPU (Central Processing Unit). Processor 102 may be an integrated circuit chip with signal processing capabilities. Processor 102 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 102 can be implemented using integrated circuit chips.

[0111] In this exemplary electronic device, by acquiring the angle deviation of the speckle camera under preset operating conditions, the angle deviation of the speckle camera under the current operating conditions can be found. Based on the angle deviation under the current operating conditions, coordinate correction processing is performed on the initial point cloud acquired by the speckle camera under the current operating conditions to obtain the coordinate-corrected point cloud and its corrected coordinates. Based on the corrected coordinates of multiple coordinate-corrected initial point clouds, extrinsic parameter calibration processing is performed on the speckle camera to obtain the accurate extrinsic parameters of the calibrated camera. Using these extrinsic parameters, the point cloud data subsequently acquired by the speckle camera is projected onto a specified coordinate system, enabling further positioning of the target object's pose using a point cloud recognition and detection method. This avoids the drift deviation problem of the speckle camera, improves the data accuracy of the speckle camera, and achieves accurate target recognition and detection.

[0112] Please see Figure 7 , Figure 7 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 110 stores program instructions 111 that can be executed by a processor. The program instructions 111 are used to implement the steps in any of the above-described embodiments of the speckle camera correction method.

[0113] In this exemplary storage medium, by running the program instructions within the storage medium, the angle deviation of the speckle camera under preset operating conditions can be obtained. The corresponding angle deviation of the speckle camera under the current operating condition can then be located. Based on the angle deviation under the current operating condition, the initial point cloud acquired by the speckle camera under the current operating condition is subjected to coordinate correction processing to obtain the coordinate-corrected point cloud and its corrected coordinates. Based on the corrected coordinates of multiple coordinate-corrected initial point clouds, the speckle camera's extrinsic parameters are calibrated to obtain the accurate extrinsic parameters of the calibrated speckle camera. Using these extrinsic parameters, the point cloud data subsequently acquired by the speckle camera is projected onto a specified coordinate system, enabling further positioning of the target object's pose using point cloud recognition and detection methods. This avoids the drift deviation problem of the speckle camera, improves the data accuracy of the speckle camera, and achieves accurate target recognition and detection.

[0114] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0115] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0116] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0117] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for correcting a speckle camera, characterized in that, The method includes: Obtain the angle deviation of the speckle camera under preset operating conditions; Based on the angle deviation, the initial point cloud acquired by the speckle camera under the current operating conditions is subjected to coordinate correction processing to obtain the corrected coordinates; The step of performing coordinate correction processing on the initial point cloud acquired by the speckle camera under the current working condition based on the angle deviation includes: obtaining the initial coordinates of the currently traversed point cloud in the initial point cloud; determining at least one target working condition from multiple preset working conditions based on the current working condition and preset filtering conditions; correcting the initial coordinates based on the angle deviation of at least one target working condition to obtain the corrected coordinates; The target operating condition includes a preset object distance and a preset temperature, and the current operating condition includes a current object distance and a current temperature. Correcting the initial coordinates based on the angle deviation of at least one target operating condition to obtain the corrected coordinates includes: responding to the presence of multiple target operating conditions, determining the angle deviation weight for each target operating condition based on the preset object distance and the preset temperature, as well as the current object distance and the current temperature; performing a weighted summation of the angle deviations of each target operating condition based on the angle deviation weights to obtain the target angle deviation; and correcting the initial coordinates based on the target angle deviation to obtain the corrected coordinates. The speckle camera's extrinsic parameters are obtained by performing extrinsic parameter calibration on the correction coordinates of multiple initial point clouds.

2. The method according to claim 1, characterized in that, The step of obtaining the angle deviation of the speckle camera under preset operating conditions includes: Based on the planar point cloud information collected by the speckle camera on the preset plane under various preset working conditions, the target point cloud plane corresponding to each preset working condition is determined respectively. Based on the target point cloud plane, determine the angular deviation between the normal vector of the target point cloud plane and the acquisition direction of the speckle camera under each preset working condition, wherein the acquisition direction is perpendicular to the preset plane.

3. The method according to claim 2, characterized in that, The step of determining the target point cloud plane corresponding to each preset working condition based on the planar point cloud information acquired by the speckle camera from a preset plane under various preset working conditions includes: Acquire multiple frames of planar point cloud information of the preset plane under various preset working conditions using the speckle camera; Planar fitting processing was performed on multiple frames of planar point cloud information to obtain multiple target point cloud planes.

4. The method according to claim 1, characterized in that, After performing extrinsic parameter calibration on the speckle camera based on the correction coordinates of multiple initial point clouds to obtain the camera extrinsic parameters, the method further includes: Obtain the point cloud of the target to be identified; The target point cloud is obtained by performing coordinate transformation on the point cloud to be identified based on the camera extrinsic parameters. The pose information of the target to be identified is determined based on the target point cloud.

5. The method according to claim 1, characterized in that, The speckle camera is mounted on a forklift, which includes a first type of forklift and a second type of forklift. The first type of forklift has a fixed connecting plate on its fork teeth, and the plane of the fixed plate is perpendicular to the fork-taking direction of the fork teeth. The second type of forklift does not have the fixed plate on its fork teeth. After the extrinsic parameter calibration of the speckle camera is performed based on the correction coordinates of multiple initial point clouds to obtain the camera extrinsic parameters of the speckle camera, the method further includes: In response to the forklift being the second type of forklift, the pose information of the target to be identified is determined based on the camera extrinsic parameters and the collected point cloud of the target to be identified; In response to the forklift being the first type of forklift, the pose information of the target to be identified is determined based on the camera extrinsic parameters, the point cloud of the fixed plate acquired, and the point cloud to be identified.

6. The method according to claim 5, characterized in that, The step of determining the pose information of the target to be identified based on the camera extrinsic parameters, the collected point cloud of the fixed plate, and the point cloud to be identified includes: The point cloud of the fixed plate is subjected to plane fitting processing to obtain the plane of the fixed plate; The plane angle of the fixed plate is corrected based on the camera's extrinsic parameters to obtain the corrected angle; The fixed plate point cloud and the point cloud to be identified are subjected to coordinate correction processing according to the correction angle to obtain the corrected fixed plate point cloud and the corrected point cloud to be identified. Based on the camera extrinsic parameters, coordinate transformation is performed on the corrected fixed plate point cloud and the corrected point cloud to be identified to obtain the target fixed plate point cloud and the target point cloud to be identified. The pose information of the target to be identified is determined based on the point cloud of the target fixed plate and the point cloud of the target to be identified.

7. An electronic device, characterized in that, The method includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 6.

8. A computer-readable storage medium having program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they implement the method described in any one of claims 1 to 6.

Images