Risk positioning method, electronic equipment, storage medium and program product

Abstract

The invention provides a risk positioning method, electronic equipment, a storage medium and a program product, and relates to the technical field of machine vision, and the method comprises the steps: obtaining a calibration external parameter and a first pose of a calibration plate in a fixed coordinate system in response to the completion of calibration of the external parameter of a camera; based on a plurality of groups of external parameter calibration data, calibration external parameters and the first pose collected during camera external parameter calibration, checking the preset check item, and determining a check result corresponding to the preset check item; the preset inspection item comprises at least one of Euler angle type inspection, camera precision inspection and robot absolute precision inspection in the external parameter calibration data, and the external parameter calibration data comprises a flange pose and a coordinate value of a mark point on the calibration plate in a camera coordinate system; and if the check result is that the check is not passed, determining a corresponding risk positioning suggestion according to the check result. The risk positioning efficiency can be effectively improved.

Description

Technical Field

[0001] This disclosure relates to the field of machine vision technology, and in particular to a risk location method, electronic device, storage medium, and program product. Background Technology

[0002] In recent years, machine vision technology has been increasingly widely used. For example, it can be used to identify and grasp objects. However, in the initial deployment and continuous operation of machine vision-guided services, various problems with inaccurate grasping often arise.

[0003] Currently, for various problems that are difficult to pinpoint, on-site engineers need to rely on their experience to gradually investigate the reasons for the uncertainty. However, the complex on-site situation leads to low efficiency in risk identification. Summary of the Invention

[0004] This disclosure provides a risk location method, electronic device, storage medium, and program product to address the current problem of low efficiency in risk location due to the inability to accurately identify various discovered issues.

[0005] Firstly, this disclosure provides a risk positioning method, including:

[0006] In response to the completion of camera extrinsic calibration, the calibration extrinsic parameters and the first pose of the calibration plate in the fixed coordinate system are obtained;

[0007] Based on multiple sets of extrinsic calibration data, calibration extrinsic parameters and first pose collected during camera extrinsic calibration, the preset check items are checked and the check results corresponding to the preset check items are determined. The preset check items include at least one of Euler angle type check, camera accuracy check and robot absolute accuracy check in the extrinsic calibration data. The extrinsic calibration data includes the flange pose and the coordinate values ​​of the marker points on the calibration plate in the camera coordinate system.

[0008] If the inspection result is that the inspection fails, then the corresponding risk positioning recommendations will be determined based on the inspection results.

[0009] Secondly, this disclosure provides a risk positioning device, comprising:

[0010] The acquisition module is used to acquire the calibration extrinsic parameters and the first pose of the calibration plate in the fixed coordinate system in response to the completion of camera extrinsic parameter calibration;

[0011] The first processing module is used to check preset check items based on multiple sets of extrinsic calibration data, calibration extrinsic parameters and first pose collected during camera extrinsic calibration, and to determine the check results corresponding to the preset check items. The preset check items include at least one of Euler angle type check, camera accuracy check and robot absolute accuracy check in the extrinsic calibration data. The extrinsic calibration data includes the flange pose and the coordinate values ​​of the marker points on the calibration plate in the camera coordinate system.

[0012] The determination module is used to determine the corresponding risk positioning recommendations based on the inspection result if the inspection result is "inspection fails".

[0013] Thirdly, this disclosure provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0014] The memory stores the instructions that the computer executes;

[0015] The processor executes computer execution instructions stored in memory to implement the risk location method as described in the first aspect of this disclosure.

[0016] Fourthly, this disclosure provides a computer-readable storage medium storing computer program instructions, which, when executed, implement the risk location method as described in the first aspect of this disclosure.

[0017] Fifthly, this disclosure provides a computer program product, including a computer program that, when executed, implements the risk location method as described in the first aspect of this disclosure.

[0018] The risk location method, electronic device, storage medium, and program product provided in this disclosure acquire the calibration extrinsic parameters and the first pose of the calibration board in a fixed coordinate system in response to the completion of camera extrinsic parameter calibration. Based on multiple sets of extrinsic parameter calibration data, calibration extrinsic parameters, and the first pose collected during camera extrinsic parameter calibration, preset check items are checked to determine the corresponding check results. The preset check items include at least one of Euler angle type check, camera accuracy check, and robot absolute accuracy check in the extrinsic parameter calibration data. The extrinsic parameter calibration data includes the flange pose and the coordinate values ​​of the marker points on the calibration board in the camera coordinate system. If the check result is a failure, a corresponding risk location suggestion is determined based on the check result. This disclosure automatically checks the preset check items based on multiple sets of extrinsic parameter calibration data, calibration extrinsic parameters, and the first pose collected during camera extrinsic parameter calibration, determines the corresponding check results, and provides corresponding risk location suggestions when the check result is a failure. This allows for problem repair based on the risk location suggestions, effectively guiding risk location and improving the efficiency of risk location and resolution. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A schematic diagram of the EIH scenario provided for related technologies;

[0021] Figure 2 A schematic diagram of an ETH scenario provided for related technologies;

[0022] Figure 3 This is a schematic diagram illustrating an application scenario provided by an embodiment of the present disclosure;

[0023] Figure 4 A flowchart illustrating a risk location method provided in an embodiment of this disclosure;

[0024] Figure 5 A schematic diagram of the external parameter calibration report provided for embodiments of this disclosure. Figure 1 ;

[0025] Figure 6 A schematic diagram of the external parameter calibration report provided for embodiments of this disclosure. Figure 2 ;

[0026] Figure 7 A schematic diagram of the external parameter calibration report provided for embodiments of this disclosure. Figure 3 ;

[0027] Figure 8 A schematic diagram of the external parameter calibration report provided for embodiments of this disclosure. Figure 4 ;

[0028] Figure 9 A schematic diagram of the external parameter calibration report provided for embodiments of this disclosure. Figure 5 ;

[0029] Figure 10 A flowchart of a risk location method provided in another embodiment of this disclosure;

[0030] Figure 11 A schematic diagram of an input interface provided according to an embodiment of this disclosure;

[0031] Figure 12 A schematic diagram illustrating the horizontal tolerance provided in an embodiment of this disclosure;

[0032] Figure 13 A schematic diagram of the inspection logic of an error analysis tool provided in an embodiment of this disclosure;

[0033] Figure 14 A schematic diagram of a crawling scenario provided in the embodiments of this disclosure. Figure 1 ;

[0034] Figure 15 A schematic diagram of the external parameter calibration report provided for embodiments of this disclosure. Figure 6 ;

[0035] Figure 16 This is an illustration of the generation of troubleshooting guidance information provided in the embodiments of this disclosure. Figure 1 ;

[0036] Figure 17 This is an illustration of the generation of troubleshooting guidance information provided in the embodiments of this disclosure. Figure 2 ;

[0037] Figure 18 A schematic diagram of a crawling scenario provided in the embodiments of this disclosure. Figure 2 ;

[0038] Figure 19 A schematic diagram of the external parameter calibration report provided for embodiments of this disclosure. Figure 7 ;

[0039] Figure 20 This is an illustration of the generation of troubleshooting guidance information provided in the embodiments of this disclosure. Figure 3 ;

[0040] Figure 21 A schematic diagram illustrating updated troubleshooting guidance information provided in an embodiment of this disclosure;

[0041] Figure 22 This is a schematic diagram of the structure of a risk positioning device provided in an embodiment of the present disclosure;

[0042] Figure 23 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0044] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0045] To better understand this disclosure, some technical terms and general data involved in the embodiments of this disclosure will be briefly described first, as follows:

[0046] I. Some technical terms:

[0047] In an Eye to Hand (ETH) scenario, the camera and robotic arm are separated. The robotic arm's working position and working plane are fixed. The camera is placed in a reasonable position (such as a ceiling or beam) to identify the plane on which the robotic arm will work. During the robotic arm's operation, the camera does not move with the robotic arm.

[0048] The Eye in Hand (EIH) scenario involves a camera mounted at the end of a robotic arm, which moves along with the arm during operation.

[0049] II. Some general data:

[0050] 1. Common forms of expression:

[0051] The pose value T = [t, q], where t represents the translation amount and q represents the quaternion.

[0052] The coordinate values ​​[x, y, z] represent the translation amounts along the x, y, and z axes in the current coordinate system.

[0053] 2. Information in the configuration file:

[0054] (1) Nominal dimensions of the calibration plate: This dimension allows the generation of coordinate values ​​for the nominal points (i.e., marker points) on the calibration plate; it can be used to evaluate the accuracy of camera measurement dimensions, and also to calculate the coordinate values ​​of the calibration plate within the camera (e.g., using...). express).

[0055] (2) Euler angle type of robotic arm posture (convention): When the external parameter calculation error is too large, this information can be used to search for Euler angle type.

[0056] 3. The external parameter calibration process will collect N sets of external parameter calibration data. Each set of external parameter calibration data may include:

[0057] (1) Flange (i.e., the end effector of the robotic arm) position (e.g., using...) The data format is [t, q). Since the calculation of extrinsic parameters requires certain numerical constraints, the flange pose includes both the flange attitude with relative movement and the flange attitude with relative rotation.

[0058] (2) The coordinates of the marker points on the calibration board acquired by the camera (i.e., the data of the marker points in the camera coordinate system), in the format [x,y,z]. Based on these coordinates, the coordinates of the calibration board in the camera can be obtained. The data format is [t, q].

[0059] 4. Two results can be obtained through the extrinsic parameter algorithm:

[0060] (1) External reference For the EIH scenario, the extrinsic parameter represents the relationship between the camera and the end effector of the robotic arm; for the ETH scenario, the extrinsic parameter represents the relationship between the camera and the robotic arm base.

[0061] (2) The pose of the calibration plate in a fixed coordinate system (e.g., using...) (This pose represents the relationship between the calibration plate and the robotic arm base in the EIH scenario; in the ETH scenario, it represents the relationship between the calibration plate and the robotic arm end effector. For example,) Figure 1 A schematic diagram of the EIH scenario provided for related technologies, such as... Figure 1 As shown, 101 is the camera under test. It shows that in the EIH scenario, the extrinsic parameters represent the relationship between the camera and the end effector of the robotic arm, as well as the pose of the calibration plate in the fixed coordinate system. Figure 2 A schematic diagram of an ETH scenario provided for related technologies, such as... Figure 2 As shown, 201 is the camera under test. It shows that in the ETH scenario, the extrinsic parameters represent the relationship between the camera and the robotic arm base, as well as the pose of the calibration plate in the fixed coordinate system.

[0062] 5. Based on external parameters The pose of the calibration plate in a fixed coordinate system can be obtained individually for each set of data in the N sets of extrinsic parameter calibration data (e.g., using...). (This indicates that) due to the existence of systematic errors, With the above-mentioned There will be slight differences.

[0063] 6. Euler angle types are as follows:

[0064] z1-x'-z2″(intrinsic rotations) or (or)z2-x-z1(extrinsic rotations)

[0065] x1-y'-x2″(intrinsic rotations)or x2-y-x1(extrinsic rotations)

[0066] y1-z'-y2″(intrinsic rotations)or y2-z-y1(extrinsic rotations)

[0067] z1-y'-z2″(intrinsic rotations)or z2-y-z1(extrinsic rotations)

[0068] x1-z'-x2″(intrinsic rotations)or x2-z-x1(extrinsic rotations)

[0069] y1-x'-y2″(intrinsic rotations)or y2-x-y1(extrinsic rotations)

[0070] x-y'-z″(intrinsic rotations)or z-y-x(extrinsic rotations)

[0071] y-z'-x″(intrinsic rotations)or x-z-y(extrinsic rotations)

[0072] z-x'-y″(intrinsic rotations)or y-x-z(extrinsic rotations)

[0073] x-z'-y″(intrinsic rotations)or y-z-x(extrinsic rotations)

[0074] z-y'-x″(intrinsic rotations)or x-y-z(extrinsic rotations)

[0075] y-x'-z″(intrinsic rotations)or z-x-y(extrinsic rotations)

[0076] Wherein, intrinsic rotations = rotated axis, and extrinsic rotations = static / fixed axis. Intrinsic rotations include yaw, pitch, and roll. In the embodiments of this disclosure, the Euler angle type is, for example, XYZ_Static, that is, the above xyz (extrinsic rotations), which represents the rotation angle (α, β, γ) and the rotation order (x->y->z), which is extrinsic rotation.

[0077] Currently, during the initial deployment and continuous operation of machine vision-guided services, various inaccuracies are frequently encountered. Specifically, after camera extrinsic calibration, if serious deployment obstacles are found, on-site engineers need to rely on their experience to troubleshoot and fix the problems step by step. After fixing these issues, deploying the grasping process may again result in inaccuracies, requiring on-site engineers to rely on their experience to troubleshoot the causes. However, due to the complexity of on-site situations, on-site engineers often spend a significant amount of time troubleshooting, leading to low risk identification efficiency and consequently impacting delivery efficiency.

[0078] Based on the aforementioned problems, and considering that extrinsic parameters can reveal certain data characteristics for various uncertainties, this disclosure provides a risk location method, electronic device, storage medium, and program product. Upon completion of camera extrinsic parameter calibration, the calibration extrinsic parameters and the pose of the calibration board in a fixed coordinate system are acquired. Based on multiple sets of extrinsic parameter calibration data collected during camera calibration, the acquired calibration extrinsic parameters, and the pose, preset check items are checked. If the check result is a failure, corresponding risk location suggestions are determined based on the check result, effectively improving the efficiency of locating and resolving problems that severely hinder deployment. Furthermore, if uncertainties exist during deployment, troubleshooting guidance information is generated to guide risk location, effectively improving the efficiency of locating uncertainties during deployment.

[0079] The following section provides examples illustrating the application scenarios of the solutions provided in this disclosure.

[0080] Figure 3 This is a schematic diagram illustrating an application scenario provided by an embodiment of this disclosure. For example... Figure 3As shown, this application scenario takes the ETH scenario as an example. This scenario can include: a robotic arm 301, a calibration plate 302 connected to the end of the robotic arm 301, a camera 303, and an electronic device 304 communicatively connected to the camera 303. In response to the completion of the extrinsic parameter calibration of the camera 303, the electronic device 304 acquires the calibration extrinsic parameters and the pose of the calibration plate 302 in a fixed coordinate system. Based on multiple sets of extrinsic parameter calibration data collected during the calibration of the camera 303, the acquired calibration extrinsic parameters, and the pose, the electronic device 304 checks preset check items and determines the check results corresponding to the preset check items. If the check result is a failure, the corresponding risk positioning suggestion is determined based on the check result.

[0081] It should be noted that, Figure 3 This is merely a schematic diagram illustrating one application scenario provided by an embodiment of this disclosure; this embodiment does not necessarily represent an application scenario. Figure 3 The included equipment is not limited, nor is it restricted. Figure 3 The positional relationships between the devices are defined.

[0082] Next, the risk identification method will be introduced through specific examples.

[0083] Figure 4 This is a flowchart illustrating a risk localization method according to an embodiment of the present disclosure. The risk localization method can be executed by software and / or hardware devices; for example, the hardware device can be a risk localization apparatus, which can be an electronic device or a processing chip within an electronic device. Figure 4 As shown, the method in this embodiment includes:

[0084] S401. In response to the completion of camera extrinsic calibration, acquire the calibration extrinsic parameters and the first pose of the calibration plate in the fixed coordinate system.

[0085] In this disclosure embodiment, by way of example, reference is made to Figure 1 and Figure 2 After the camera extrinsic calibration is completed, the calibration extrinsic parameters can be obtained. The first pose of the calibration plate in the fixed coordinate system

[0086] S402. Based on multiple sets of extrinsic parameter calibration data, calibration extrinsic parameters and first pose collected during camera extrinsic parameter calibration, check the preset check items and determine the check results corresponding to the preset check items; the preset check items include at least one of Euler angle type check, camera accuracy check and robot absolute accuracy check in the extrinsic parameter calibration data, and the extrinsic parameter calibration data includes the flange pose and the coordinate values ​​of the marker points on the calibration plate in the camera coordinate system.

[0087] In this step, multiple sets of extrinsic parameter calibration data collected during camera extrinsic parameter calibration are used. Each set of extrinsic parameter calibration data includes the flange pose and the coordinate values ​​of the marker points on the calibration plate in the camera coordinate system. Preset check items are determined based on the errors and their sources present in the grasping scene, including at least one of Euler angle type check, camera accuracy check, and robot absolute accuracy check. Checking the preset check items can effectively improve risk localization efficiency. For example, the preset check items may include Euler angle type check, camera accuracy check, and robot absolute accuracy check. Each preset check item can be checked based on the multiple sets of extrinsic parameter calibration data collected during camera extrinsic parameter calibration, the calibration extrinsic parameters, and the first pose, to determine the check result corresponding to each preset check item.

[0088] Optionally, the inspection results for Euler angle type inspection can include inspection passed and inspection failed; the inspection results for camera accuracy inspection can include inspection passed, inspection failed, and risk exists; the inspection results for robot absolute accuracy inspection can include inspection passed and risk exists.

[0089] For each preset check item, the specific method of checking the preset check item based on multiple sets of extrinsic calibration data, calibration extrinsic parameters and first pose collected during camera extrinsic calibration can be found in the subsequent embodiments, and will not be repeated here.

[0090] S403. If the inspection result is that the inspection fails, then the corresponding risk positioning recommendations shall be determined based on the inspection result.

[0091] In this step, if the inspection result for a preset inspection item is "failure," a corresponding risk positioning recommendation can be determined based on the data obtained during the inspection of the preset inspection item. For details on how to determine the corresponding risk positioning recommendation, please refer to subsequent embodiments; it will not be repeated here.

[0092] The risk location method provided in this disclosure, in response to the completion of camera extrinsic parameter calibration, acquires the calibration extrinsic parameters and the first pose of the calibration board in a fixed coordinate system; based on multiple sets of extrinsic parameter calibration data, calibration extrinsic parameters, and the first pose collected during camera extrinsic parameter calibration, it checks preset inspection items and determines the corresponding inspection results; the preset inspection items include at least one of Euler angle type check, camera accuracy check, and robot absolute accuracy check in the extrinsic parameter calibration data, and the extrinsic parameter calibration data includes the flange pose and the coordinate values ​​of the marker points on the calibration board in the camera coordinate system; if the inspection result is a failure, a corresponding risk location suggestion is determined based on the inspection result. In this disclosure, for preset inspection items, based on multiple sets of extrinsic parameter calibration data, calibration extrinsic parameters, and the first pose collected during camera extrinsic parameter calibration, the preset inspection items are automatically checked, and the corresponding inspection results are determined. If the inspection result is a failure, a corresponding risk location suggestion is provided to correct the problem, effectively guiding risk location and improving the efficiency of risk location and resolution.

[0093] The following section provides a detailed description of risk identification methods for different preset inspection items.

[0094] I. Euler Angle Type Check:

[0095] For example, the input information for performing Euler angle type checking may include:

[0096] (1) Information in the configuration file: coordinates of the nominal point on the calibration board; Euler angle type of the robot arm posture (convention).

[0097] (2) N sets of external parameter calibration data collected in the external parameter calibration process: flange pose; coordinate values ​​of the marker points on the calibration plate collected by the camera.

[0098] When the Euler angle type check is completed, the output information may include: the extrinsic parameter calibration data after the flange pose is corrected; the extrinsic parameters obtained after the flange pose is corrected; the correct Euler angle type; and the Euler angle type check result.

[0099] Based on the above embodiments, in one possible implementation, the flange pose includes relative rotation data. Based on multiple sets of extrinsic calibration data, calibration extrinsic parameters, and the first pose collected during camera extrinsic calibration, preset check items are checked to determine the check result corresponding to the preset check items. This may include: determining whether to check the Euler angle type based on multiple sets of extrinsic calibration data, calibration extrinsic parameters, and the first pose collected during camera extrinsic calibration; if so, based on the flange pose, multiple groups containing a preset number of relative rotation data are obtained, and the multiple groups include a first group of reference pose data used to determine the relative rotation; Euler angle type search is performed according to the first group to determine the optimal Euler angle type; if the current Euler angle type is different from the optimal Euler angle type, the check result corresponding to the Euler angle type check is determined to be a check failure.

[0100] In this embodiment, it can be determined whether to check the Euler angle type based on multiple sets of extrinsic parameter calibration data, calibration extrinsic parameters, and the first pose collected during camera extrinsic parameter calibration. Optionally, determining whether to check the Euler angle type based on multiple sets of extrinsic parameter calibration data, calibration extrinsic parameters, and the first pose collected during camera extrinsic parameter calibration can include: for each set of extrinsic parameter calibration data, obtaining the second pose of the calibration board in a fixed coordinate system corresponding to the extrinsic parameter calibration data according to the extrinsic parameter calibration data and calibration extrinsic parameters; obtaining the maximum error magnitude based on the difference between the first pose and each second pose; if the maximum error magnitude is greater than a first threshold, then it is determined to perform an Euler angle type check; if the maximum error magnitude is less than or equal to the first threshold, then it is determined not to perform an Euler angle type check.

[0101] For example, based on the coordinate values ​​of the nominal points on the calibration plate and the N sets of extrinsic parameter calibration data collected in the extrinsic parameter calibration process, the calibration extrinsic parameters are obtained. The first pose of the calibration plate in the fixed coordinate system Based on the N sets of external parameter calibration data and calibration external parameters collected during the external parameter calibration process, the second pose of each calibration plate in the fixed coordinate system is obtained. by Based on this, obtain all The error magnitude can be determined, thus allowing us to determine the maximum error magnitude (e.g., using error). max (This is an example of a threshold value of 0.02m.) If the error... max If the value is greater than 0.02m (where m represents meters), then the current system has a large error and an Euler angle check is needed; if the error is greater than 0.02m, then the system has a large error and an Euler angle check is needed. max If the value is ≤0.02m, then Euler angle type check will not be performed.

[0102] After determining the Euler angle type, the relative rotation data in the flange pose can be further grouped. For example, with a preset group size of 4, each group consists of 4 relative rotation data points, resulting in multiple groups. Each group contains a first group of reference pose data used to determine the relative rotation. The Euler angle type can then be searched based on this first group to determine the optimal Euler angle type.

[0103] Optionally, performing an Euler angle type search based on the first group to determine the optimal Euler angle type may include: for each relative rotation data in the first group, converting the relative rotation data into a first Euler angle value using the current Euler angle type; expanding the first Euler angle value according to the Euler angle configuration properties to obtain multiple expanded second Euler angle values; combining each second Euler angle value with each of the multiple preset Euler angle types to obtain multiple combinations; obtaining the first extrinsic parameter corresponding to each of the multiple combinations; for each of the multiple combinations, obtaining the first residual corresponding to the combination based on the first extrinsic parameter; and determining the Euler angle type corresponding to the combination with the smallest first residual among the first residuals as the optimal Euler angle type.

[0104] For example, for each relative rotation data (i.e., flange rotation q) in the first group, q is converted into a first Euler angle value (e.g., represented by [e0,e1,e2]0) using the current Euler angle type. Based on the original Euler angle configuration properties, the first Euler angle value is expanded to obtain multiple expanded second Euler angle values ​​(i.e., Euler angle alternatives). Assuming there are k expanded second Euler angle values, these k values ​​might be: [e0,e1,e2]0, [e0,e1,e2]1, [e0,e1,e2]2, ..., [e0,e1,e2] k Each second Euler angle value is combined with each of the multiple preset Euler angle types to obtain multiple combinations, and the first extrinsic parameter corresponding to each combination is obtained. For each combination, based on the first extrinsic parameter, the second pose of the calibration plate in the fixed coordinate system can be obtained, and thus the first residual corresponding to each combination can be obtained based on the second pose of the calibration plate in the fixed coordinate system. The Euler angle type corresponding to the combination with the smallest first residual is determined as the optimal Euler angle type (e.g., using the convention). new express).

[0105] After determining the optimal Euler angle type, if the current Euler angle type of the system is different from the optimal Euler angle type, the check result corresponding to the Euler angle type check is determined to be a check failure, the optimal Euler angle type (i.e. the correct Euler angle type) is saved, and the corresponding risk positioning suggestion is output.

[0106] In another possible implementation, for each of the multiple groups, an Euler angle value search can be performed based on the group and the optimal Euler angle type to determine the optimal Euler angle value corresponding to each group; the relative rotation data can be updated based on the optimal Euler angle type and the optimal Euler angle value corresponding to each group to obtain updated extrinsic parameter calibration data; if the flange pose is different from the flange pose in the updated extrinsic parameter calibration data, the check result corresponding to the Euler angle type check is determined to be a check failure.

[0107] In this embodiment, Euler angle value search can be performed for each of the multiple groups to determine the optimal Euler angle value corresponding to each group. Optionally, for each of the multiple groups, Euler angle value search is performed based on the group and the optimal Euler angle type to determine the optimal Euler angle value corresponding to each group. This can include: for each of the multiple groups, converting each relative rotation data in the group into a third Euler angle value using the current Euler angle type; expanding the third Euler angle value according to the Euler angle configuration properties to obtain multiple expanded fourth Euler angle values; combining each fourth Euler angle value with the optimal Euler angle type to obtain multiple combinations; obtaining the second extrinsic parameter corresponding to each of the multiple combinations; for each of the multiple combinations, obtaining the second residual corresponding to the combination based on the second extrinsic parameter; and for each group, determining the fourth Euler angle value corresponding to the second extrinsic parameter with the smallest difference from the calibration extrinsic parameter as the optimal Euler angle value for that group.

[0108] For example, for each of the multiple groups, obtain each relative rotation data (i.e., flange rotation q) in the group, and convert q into a third Euler angle value using the current Euler angle type. Based on the original Euler angle configuration properties, expand the third Euler angle value to obtain multiple expanded fourth Euler angle values ​​(i.e., Euler angle alternatives). Combine each fourth Euler angle value with the optimal Euler angle type to obtain multiple combinations, and obtain the second extrinsic parameter corresponding to each combination. For each combination, based on the second extrinsic parameter corresponding to the combination, the second pose of the calibration plate in the fixed coordinate system corresponding to the combination can be obtained, thus obtaining the second residual corresponding to the combination based on the second pose of the calibration plate in the fixed coordinate system corresponding to each combination. For each group, determine the fourth Euler angle value corresponding to the second extrinsic parameter with the smallest difference from the calibration extrinsic parameter as the optimal Euler angle value for the group (e.g., represented by [e0,e1,e2]*).

[0109] After determining the optimal Euler angle values ​​for each group, the relative rotation data can be updated based on the optimal Euler angle type and the optimal Euler angle values ​​for each group to obtain updated extrinsic parameter calibration data, i.e., extrinsic parameter calibration data after the flange pose has been corrected. Updated extrinsic parameters can also be obtained from the updated extrinsic parameter calibration data, i.e., extrinsic parameters obtained after the flange pose has been corrected. If the flange pose differs from the flange pose in the updated extrinsic parameter calibration data, the check result corresponding to the Euler angle type check is determined to be a failure. The extrinsic parameter calibration data and the extrinsic parameters obtained after the flange pose has been corrected are saved, and corresponding risk positioning suggestions are output.

[0110] It is understandable that if, during the process of checking the Euler angle type, the current Euler angle type is the same as the optimal Euler angle type, and the current flange pose is the same as the flange pose in the updated extrinsic parameter calibration data, then the result of the Euler angle type check can be determined as a pass.

[0111] Optionally, if the Euler angle type check fails, the corresponding risk positioning recommendations are determined based on the check result, including replacing the current Euler angle type with the optimal Euler angle type and / or replacing the calibration extrinsic parameters with the updated extrinsic parameters, which are obtained based on the updated extrinsic parameter calibration data.

[0112] For example, if the check fails because the current Euler angle type is different from the optimal Euler angle type, the corresponding risk positioning suggestion is to replace the current Euler angle type with the optimal Euler angle type. If the check fails because the current flange pose is different from the flange pose in the updated extrinsic parameter calibration data, the corresponding risk positioning suggestion is to replace the flange pose in the current extrinsic parameter calibration data with the flange pose in the updated extrinsic parameter calibration data, and replace the calibration extrinsic parameters with the updated extrinsic parameters. Referring to the above embodiments, the updated extrinsic parameters are obtained based on the updated extrinsic parameter calibration data. For details on how to output the risk positioning suggestion on the interface, please refer to subsequent embodiments.

[0113] II. Camera accuracy check:

[0114] For example, when performing a camera accuracy check, the corresponding input information may include:

[0115] (1) Information in the configuration file: coordinate values ​​of the nominal points on the calibration board.

[0116] (2) N sets of external parameter calibration data collected in the external parameter calibration process: flange pose; coordinate values ​​of the marker points on the calibration plate collected by the camera.

[0117] (3) Calibrate external parameters

[0118] When the camera accuracy check is completed, the output information obtained may include the check results of the camera accuracy check.

[0119] Based on the above embodiments, optionally, based on multiple sets of extrinsic parameter calibration data, calibration extrinsic parameters, and the first pose acquired during camera extrinsic parameter calibration, pre-defined check items are checked to determine the check results corresponding to the pre-defined check items. This may include: obtaining the scalar error of the camera corresponding to each position of the calibration plate in the extrinsic parameter calibration data and the nominal size of the calibration plate, as well as the coordinate values ​​of the calibration plate in the camera coordinate system, according to the extrinsic parameter calibration data and the nominal size of the calibration plate; determining the reference position and other positions based on the coordinate values ​​of the calibration plate in the camera coordinate system, where other positions are the positions of the calibration plate other than the reference position; determining whether the main error source in the vertical axis direction is the camera based on the second pose of the reference position and the second poses of other positions, where the second pose is the pose of the calibration plate in the fixed coordinate system corresponding to the extrinsic parameter calibration data obtained based on the extrinsic parameter calibration data and the calibration extrinsic parameters; if the main error source in the vertical axis direction is the camera, then checking the vertical axis coordinates and other positions in the second poses of other positions. The translation error of the second pose relative to the reference position is fitted, and the vertical axis error and vertical axis error ratio corresponding to other positions are determined based on the fitting result; or, if the main error source in the vertical axis direction is not the camera, the vertical axis error and vertical axis error ratio corresponding to other positions are determined based on the scalar error and the vertical axis difference between other positions and the reference position; if the horizontal and vertical axis error ratios corresponding to any other position are greater than the second threshold, or the vertical axis error ratios corresponding to any other position are greater than the third threshold, the camera accuracy check result is determined to be a failure. The horizontal and vertical axis error ratios are obtained based on the distances between other positions and the reference position and the scalar error, or, if the main error source in the vertical axis direction is the camera, they are determined based on the distances between other positions and the reference position, the deviations between other positions and the reference position, and the scalar error.

[0120] For example, the horizontal axis is the x-axis, the vertical axis is the y-axis, and the vertical axis is the z-axis. When obtaining the scalar error of the camera corresponding to the calibration plate at each position in the extrinsic calibration data, the standard distance between two adjacent markers on the calibration plate (e.g., denoted by d1) can be obtained based on the coordinate values ​​of the nominal points on the calibration plate; the distance between two adjacent markers measured by the camera (e.g., denoted by d2) can be obtained based on the coordinate values ​​of the markers on the calibration plate in the extrinsic calibration data; based on d1 and d2, the scalar error of the camera corresponding to the calibration plate at each position in the extrinsic calibration data (e.g., denoted by ratio) can be obtained as: ratio = |d1 - d2| / d1.

[0121] Based on the coordinates of the nominal points on the calibration plate and the coordinates of the marker points on the calibration plate in the extrinsic parameter calibration data, the coordinates of the calibration plate at each position in the extrinsic parameter calibration data in the camera coordinate system can be obtained. Based on the coordinates of the calibration plate at each position in the camera coordinate system, the reference position and other positions are determined. The other positions are the locations of the calibration plate other than the reference position. For example, assuming the j-th extrinsic parameter calibration data is located at the center of the field of view, this position can be selected as the reference position. The distances (e.g., denoted by d3) from other positions along the horizontal and vertical axis planes (i.e., the xoy plane) to this reference position can be obtained, along with the error estimates (i.e., horizontal and vertical axis errors, e.g., denoted by xy) in the horizontal and vertical axis directions (i.e., the xoy direction) of these other positions. error (represented as: xy) error =ratio*d3.

[0122] Based on the second pose at the reference position and the second poses at other positions, determine whether the main source of error in the vertical direction is the camera. The second pose is determined based on the extrinsic parameter calibration data and calibration extrinsic parameters. The acquired extrinsic calibration data corresponds to the pose of the calibration plate in a fixed coordinate system, i.e.

[0123] Further, optionally, determining whether the main error source in the vertical direction is the camera based on the second pose at the reference position and the second poses at other positions may include: obtaining the correlation between the vertical coordinates and the translation error in the vertical direction based on the vertical coordinates and the translation error in the vertical direction in the second poses at other positions; obtaining the significance of the correlation based on the correlation; if the significance is less than a fifth threshold, then the main error source in the vertical direction is determined to be the camera; if the significance is greater than or equal to the fifth threshold, then the main error source in the vertical direction is determined not to be the camera.

[0124] For example, the translation error (i.e., translation error, denoted by δt) between the second pose at other positions and the second pose at the reference position is obtained: δt = [δx, y, δz]. The z-axis value of the translation of the second pose at each position is extracted and analyzed with the translation error δt. Generally, the z-axis error of the robotic arm does not follow a pattern, while the camera's z-axis error is related to the distance from the marker point to the camera, δz ~ z. 2 Therefore, the correlation between the z-axis value and δz can be obtained, and the significance of the correlation (e.g., represented by p) can be determined based on the correlation. The fifth threshold can be, for example, e. -4 If p <e -4 If the significance is strong, then the main source of error in the vertical direction is determined to be the camera; if p ≥ e -4 If so, then the main source of error in the vertical direction is not the camera.

[0125] If the camera is determined to be the primary source of error in the vertical direction, a quadratic curve f(z) can be fitted based on the z-axis values ​​and δz mentioned above. For data set i and data set j composed of z-axis values ​​and δz, the camera's z-axis error (i.e., vertical axis error, for example, using z...) error (represented as: z) error =|f(z) i )-f(z j |, the corresponding vertical axis error ratio (e.g., using ratio) z (represented as: ratio) z =z error / |z i -z j |

[0126] If the main source of error in the vertical direction is not the camera, then according to empirical formulas, in the absence of time variation, the camera's z-axis error is approximately three times the planar inspection error. Therefore, the camera's z-axis error is:

[0127] z error =ratio*3*|z i -z j | The corresponding vertical axis error ratio is: ratio z = ratio * 3.

[0128] Based on the distances from other positions to the reference position and the scalar error, obtain the horizontal and vertical axis error ratios (i.e., the xy-direction error ratios) for any of the other positions. Alternatively, if the camera is the primary source of error in the vertical direction, obtain the horizontal and vertical axis error ratios for any of the other positions based on the distances from other positions to the reference position, the deviations from the reference position, and the scalar error. A second threshold is, for example, 5%, and a third threshold is, for example, 15%. If the horizontal and vertical axis error ratios for any of the other positions are greater than 5%, or the vertical axis error ratios for any of the other positions are greater than 15%, then the camera accuracy check result is determined to be a failure.

[0129] Optionally, if the ratio of horizontal to vertical axis error corresponding to any other position is greater than the fourth threshold, or the ratio of vertical axis error corresponding to any other position is greater than the fourth threshold by a preset multiple, then the inspection result corresponding to the camera accuracy check is determined to be risky, the fourth threshold is less than the second threshold, and the fourth threshold by a preset multiple is less than the third threshold.

[0130] For example, assuming the fourth threshold is represented by 'a', and the preset multiple is, for example, 3 times, if the horizontal and vertical axis error ratios corresponding to any other position are greater than 'a', or the vertical axis error ratios corresponding to any other position are greater than 3*a, then the camera accuracy check result is determined to be risky.

[0131]

[0132] Apart from the two scenarios mentioned above where the camera accuracy check result is either "failed" or "risky," for all other cases, the camera accuracy check result is determined to be "passed."

[0133] Optionally, if the camera accuracy check fails, the corresponding risk positioning recommendations are determined based on the check result, including checking whether there is a problem with the calibration plate and / or performing manual processing; if the camera accuracy check indicates that there is a risk, the corresponding risk positioning recommendations are determined based on the check result, including checking whether there is a problem with the calibration plate, correcting the camera's internal parameters, and replacing the camera, at least one of these.

[0134] It is understandable that corresponding risk positioning recommendations can be determined based on the inspection results of different camera accuracy checks. For details on how to output these risk positioning recommendations on the interface, please refer to the following embodiments.

[0135] III. Robot Absolute Precision Inspection:

[0136] For example, when performing an absolute accuracy check on a robot, the corresponding input information may include:

[0137] (1) Information in the configuration file: coordinate values ​​of the nominal points on the calibration board.

[0138] (2) N sets of external parameter calibration data collected in the external parameter calibration process: flange pose; coordinate values ​​of the marker points on the calibration plate collected by the camera.

[0139] (3) Calibrate external parameters

[0140] (4) Data obtained during camera accuracy inspection: horizontal and vertical axis errors and vertical axis errors of the camera.

[0141] When the absolute precision check of the robot is completed, the output information obtained can include the inspection results of the robot's absolute precision check.

[0142] Based on the above embodiments, optionally, based on multiple sets of extrinsic parameter calibration data, calibration extrinsic parameters, and the first pose collected during camera extrinsic parameter calibration, the preset inspection items are checked to determine the inspection results corresponding to the preset inspection items. This may include: obtaining the robot arm errors corresponding to other positions based on the translation errors of other positions relative to the reference position and the camera errors. The camera errors include the horizontal and vertical axis errors and the vertical axis errors corresponding to other positions. The horizontal and vertical axis errors are obtained based on the distances of other positions to the reference position and the scalar errors. The reference position and other positions are determined based on the coordinate values ​​of the calibration board in the camera coordinate system. The other positions are the positions of the calibration board other than the reference position. The maximum robot arm error among multiple robot arm errors is determined. The maximum distance among the distances of other positions to the reference position is determined. If the ratio of the maximum robot arm error to the maximum distance is greater than a sixth threshold, the inspection result corresponding to the robot absolute accuracy inspection is determined to be risky.

[0143] For example, based on camera accuracy checks, the error at each other location relative to the reference location is: error camera =[xy error , z error Here, the horizontal and vertical axis errors are simplified to x-direction errors. Assuming the j-th extrinsic parameter calibration data is located at the center of the field of view, this position is selected as the reference position, and the values ​​at other positions are obtained... The error value of the translation from the reference position (e.g., expressed as δt) system (Represented). Due to the systematic error δt system Mainly caused by camera error camera And robotic arm error (e.g., represented by error) robot This is a configuration that allows us to obtain the robotic arm error at each position: error. robot =max(||δt) system ||-||error camera Since ||,0), the maximum robotic arm error can be determined (e.g., using error). robot_max (Representation). Obtain the distance from other locations to the reference location (e.g., using dist). i (represented by), thus the maximum distance can be determined (e.g., using dist). max (Indicated). The sixth threshold, for example, is 0.01. If error... robot_max / dist max If the value is greater than 0.01, the robotic arm is considered to have a serious error, and the inspection result corresponding to the robot's absolute precision check is determined to be risky. robot_max / dist max If the value is ≤0.01, then the inspection result corresponding to the robot's absolute precision inspection is determined to be a pass.

[0144] Optionally, if the inspection result of the robot's absolute accuracy check indicates that there is a risk, then the corresponding risk positioning recommendations are determined based on the inspection result, including checking the robot's accuracy and / or checking whether the robot's zero point is accurate.

[0145] It is understandable that if the inspection result of the robot's absolute accuracy check indicates a risk, corresponding risk identification suggestions can be determined. For details on how to output these risk identification suggestions on the interface, please refer to the following embodiments.

[0146] Based on the above embodiments, optionally, after determining the corresponding risk positioning recommendations based on the inspection results, the risk positioning method provided in this disclosure embodiment may further include: generating an external parameter calibration report, the external parameter calibration report including preset inspection items, inspection results corresponding to the preset inspection items, and risk positioning recommendations determined for the inspection results when the inspection results are not passed; and outputting the external parameter calibration report.

[0147] For example, Figure 5 A schematic diagram of the external parameter calibration report provided for embodiments of this disclosure. Figure 1 ,like Figure 5 As shown, the preset check items include Euler angle type check, camera accuracy check, and robot absolute accuracy check. Figure 5 The report shows the preset check items and their corresponding check results included in the external parameter calibration report. Figure 6 A schematic diagram of the external parameter calibration report provided for embodiments of this disclosure. Figure 2 ,like Figure 6 As shown, when the camera accuracy check result indicates a risk (i.e., the camera error value is greater than a lenient threshold), the external parameter calibration report includes preset check items, the check results corresponding to the preset check items, and risk positioning suggestions for the camera accuracy check results. Figure 7 A schematic diagram of the external parameter calibration report provided for embodiments of this disclosure. Figure 3 ,like Figure 7 As shown, if the camera accuracy check fails (camera error value, for example, greater than 5mm), the external parameter calibration report includes preset check items, the corresponding check results for the preset check items, and risk positioning suggestions for the camera accuracy check results. Figure 8 A schematic diagram of the external parameter calibration report provided for embodiments of this disclosure. Figure 4 ,like Figure 8As shown, when the Euler angle type check fails, the extrinsic parameter calibration report includes preset check items, the check results corresponding to the preset check items, and risk positioning suggestions for the Euler angle type check results. Among them, the virtual control "Replace with better extrinsic parameters" is used to replace the current extrinsic parameters, the flange pose in the extrinsic parameter calibration data, and the Euler angle type in the configuration file with better parameter values ​​based on the data obtained during the Euler angle type check. Figure 9 A schematic diagram of the external parameter calibration report provided for embodiments of this disclosure. Figure 5 ,like Figure 9 As shown, if the inspection result of the robot's absolute accuracy inspection indicates a risk, the external parameter calibration report includes preset inspection items, the inspection results corresponding to the preset inspection items, and risk positioning suggestions for the robot's absolute accuracy inspection.

[0148] Based on the above embodiments, Figure 10 A flowchart illustrating a risk location method provided in another embodiment of this disclosure. Figure 10 As shown, the method in this embodiment of the disclosure may include:

[0149] S1001, In response to the completion of camera extrinsic calibration, acquire the calibration extrinsic parameters and the first pose of the calibration plate in the fixed coordinate system.

[0150] For a detailed description of this step, please refer to [link / reference]. Figure 4 The relevant description of S401 in the illustrated embodiment will not be repeated here.

[0151] S1002. Based on multiple sets of extrinsic calibration data, calibration extrinsic parameters and first pose collected during camera extrinsic calibration, check the preset check items and determine the check results corresponding to the preset check items; the preset check items include at least one of Euler angle type check, camera accuracy check and robot absolute accuracy check in the extrinsic calibration data, and the extrinsic calibration data includes the flange pose and the coordinate values ​​of the marker points on the calibration plate in the camera coordinate system.

[0152] For a detailed description of this step, please refer to [link / reference]. Figure 4 The relevant description of S402 in the illustrated embodiment will not be repeated here.

[0153] S1003. If the inspection result is that the inspection fails, then the corresponding risk positioning recommendations shall be determined based on the inspection result.

[0154] For a detailed description of this step, please refer to [link / reference]. Figure 4 The relevant description of S403 in the illustrated embodiment will not be repeated here.

[0155] S1004. Generate an external parameter calibration report. The external parameter calibration report includes preset inspection items, the inspection results corresponding to the preset inspection items, and risk positioning suggestions for the inspection results if the inspection results are not passed; output the external parameter calibration report.

[0156] For a detailed description of this step, please refer to the relevant description in the above embodiments. If a serious problem hindering deployment is found, the on-site engineer can fix the problem based on the external parameter calibration report and then deploy the crawling project.

[0157] S1005. If there is a problem of inaccuracy during deployment, the system responds to the user's input operation and obtains the input target information. The target information includes the nominal size of the calibration board, multiple sets of external parameter calibration data, calibration external parameters, multiple sets of external parameter verification data, relevant information of the capture scenario, and accuracy requirement information.

[0158] Steps S1005 to S1007 can be understood as the execution steps of the error analysis tool. For example, Figure 11 This is a schematic diagram of an input interface provided in an embodiment of the present disclosure, as shown below. Figure 11 The diagram shows the input information that needs to be obtained, which is user input; specifically, the description of each piece of input information is as follows:

[0159] (1) Path to the external parameters folder (i.e., Extri Folder). The external parameters folder includes the nominal dimensions of the calibration board in the configuration file, N sets of external parameter calibration data collected in the external parameter calibration process, and calibration external parameters.

[0160] (2) The path to the extrinsic parameter validation data folder (i.e., the Validate Folder). The extrinsic parameter validation data in this folder can be used to verify whether there are serious errors in the extrinsic parameters. This folder includes the nominal dimensions of the calibration board in the configuration file and multiple sets (e.g., 3 sets) of additional extrinsic parameter calibration data collected. For details on how to collect multiple sets of extrinsic parameter calibration data, please refer to the subsequent embodiments, which will not be repeated here.

[0161] (3) Grasp Pose Relation is the relationship between the grasp pose and the grasp pose when grasping error occurs. The available options include close positions, obvious movement in the xyz axis direction, obvious rotation, and obvious rotation and movement in the xyz axis direction.

[0162] (4) Whether the image position is consistent. In the EIH scenario, whether the photo position when creating the point cloud template is consistent with the photo position when the capture error occurs (i.e., Photo Position Consistency) can be used to determine whether the camera's photo position has changed. If it is not an EIH scenario, this item can be omitted. The available options include yes or no.

[0163] (5) Is Point Cloud Quality Normal? This can be used to determine whether the image quality is poor. Available options include yes or no.

[0164] (6) Maximum allowable error of rotation posture is the maximum allowable error when the grasping posture is rotated (i.e., Max Error of Rot Poses). It can be used to determine whether the rotation error is greater than the maximum allowable error. It can obtain a value in meters (m) input by the user, such as 0.002m.

[0165] (7) The maximum allowable error in the horizontal direction is the maximum allowable error in the horizontal direction under the camera coordinate system (i.e., MaxError Along XY). It can be used to determine whether the translation error is greater than the maximum allowable error. It can obtain a value in m units input by the user, such as 0.002m.

[0166] (8) The horizontal movement distance is the horizontal movement distance in the camera coordinate system (i.e., Dist AlongXY), which can be obtained as a value in m input by the user, such as 0.500m. Figure 12 This is a schematic diagram of the horizontal tolerance provided in an embodiment of the present disclosure, as shown below. Figure 12 As shown, the horizontal tolerance is 2mm@500mm, which means that with point B as the reference, the workpiece moves 500mm from B to A and 500mm from B to C, and the allowable tolerance is 2mm, where mm represents millimeters.

[0167] (9) The maximum allowable error in the vertical direction is the maximum allowable error in the vertical direction in the camera coordinate system (i.e., MaxError Along Z). It can be obtained as a value in meters (m) input by the user, such as 0.002m.

[0168] (10) The vertical distance is the distance moved vertically in the robot arm coordinate system (i.e., DistAlong Z). It can obtain a value in meters (m) input by the user, such as 0.500m.

[0169] S1006. Based on the target information, generate investigation guidance information, which is used to guide risk identification.

[0170] In this step, after obtaining the input target information, troubleshooting guidance information can be generated based on the target information. For example, the specific troubleshooting guidance information may include common primary solutions shown in Table 1 and common secondary solutions shown in Table 2; wherein, the common secondary solutions shown in Table 2 are measures that cannot be directly determined by external parameter files and need to be manually investigated.

[0171] Table 1

[0172]

[0173]

[0174] Table 2

[0175] Troubleshooting Guide Information Options Specific solutions corresponding to the options TCP_Error Check the accuracy of the tool center point (TCP) or fixture. Project_Logic_Or_Parameters Adjust project properties; examine key steps, such as matching and output results, to ensure they are correct. Cloud_Quality Improve point cloud quality Robot_Repeatability Check the repeatability of the robotic arm

[0176] For details on how to generate investigation guidance information based on target information, please refer to the subsequent embodiments, which will not be repeated here.

[0177] S1007, Output troubleshooting guidance information.

[0178] In this step, after generating the troubleshooting guidance information, the troubleshooting guidance information can be output to facilitate on-site engineers in fixing the problem based on the troubleshooting guidance information.

[0179] The risk positioning method provided in this embodiment of the present disclosure, in response to the completion of camera extrinsic parameter calibration, acquires the calibration extrinsic parameters and the first pose of the calibration board in a fixed coordinate system; based on multiple sets of extrinsic parameter calibration data, calibration extrinsic parameters, and the first pose collected during camera extrinsic parameter calibration, checks are performed on preset check items to determine the check results corresponding to the preset check items; the preset check items include at least one of Euler angle type check, camera accuracy check, and robot absolute accuracy check in the extrinsic parameter calibration data, and the extrinsic parameter calibration data includes the flange pose and the coordinate values ​​of the marker points on the calibration board in the camera coordinate system; if the check result is a failure, a corresponding risk positioning suggestion is determined based on the check result; an extrinsic parameter calibration report is generated and output; subsequently, if there is a problem of inaccurate grasping during deployment, in response to the user's input operation, the input target information is acquired, the target information includes the nominal size of the calibration board, multiple sets of extrinsic parameter calibration data, calibration extrinsic parameters, multiple sets of extrinsic parameter verification data, relevant information of the grasping scene, and accuracy requirement information; based on the target information, investigation guidance information for guiding risk positioning is generated and output. In this embodiment, preset check items are automatically checked, and the check results corresponding to the preset check items are determined. If the check result is that the check fails, corresponding risk positioning suggestions are provided, and an external parameter calibration report is generated and output. The problem can be repaired based on the risk positioning suggestions in the external parameter calibration report, which can effectively guide risk positioning and improve the efficiency of risk positioning and resolution. If there are problems that cannot be accurately identified during the deployment process, troubleshooting guidance information is generated to guide risk positioning, thereby improving the positioning efficiency of problems that cannot be accurately identified during the deployment process.

[0180] Based on the above embodiments, optionally, for the ETH scenario, the requirements for collecting extrinsic parameter validation data in the Validate Folder may include:

[0181] (1) Fix the calibration plate to the end of the robotic arm.

[0182] (2) Open the calibration interface.

[0183] (3) Collect calibration plate data at the current location.

[0184] (4) Move the robotic arm to be close to the initial calibration posture (i.e. the posture at the beginning of the external parameter calibration), and rotate the end of the robotic arm 90° around the z-axis. If the camera cannot capture the calibration plate, translate the robotic arm so that the calibration plate is located in the middle of the field of view.

[0185] (5) Acquire calibration board images to collect calibration board data at the current location.

[0186] (6) Move the robotic arm to be close to the initial calibration posture, and rotate the end of the robotic arm by -90° around the z-axis. If the camera cannot capture the calibration plate, translate the robotic arm so that the calibration plate is located in the middle of the field of view.

[0187] (7) Acquire images of the calibration board.

[0188] (8) Save the collected data.

[0189] Optionally, for EIH scenarios, the requirements for collecting external parameter validation data in the Validate Folder may include:

[0190] (1) Place the calibration plate on a flat surface, in a position close to the calibration position (i.e., the position when calibrating external parameters); do not move the calibration plate during the following steps.

[0191] (2) Open the calibration interface.

[0192] (3) Collect calibration plate data at the current location.

[0193] (4) Move the robotic arm to be close to the initial calibration posture, and rotate the end of the robotic arm 90° around the z-axis. If the camera cannot capture the calibration plate, translate the robotic arm so that the calibration plate is located in the middle of the field of view.

[0194] (5) Acquire images of the calibration board.

[0195] (6) Move the robotic arm to be close to the initial calibration posture, and rotate the end of the robotic arm by -90° around the z-axis. If the camera cannot capture the calibration plate, translate the robotic arm so that the calibration plate is located in the middle of the field of view.

[0196] (7) Acquire images of the calibration board.

[0197] (8) Save the collected data.

[0198] Among the requirements for the acquired calibration board images are: the angle difference of the images should be as large as possible, and the calibration board should be located in the center of the field of view as much as possible.

[0199] Based on the above embodiments, Figure 13 A schematic diagram of the inspection logic of an error analysis tool provided in an embodiment of this disclosure is shown below. Figure 13 As shown, for reference Figure 11Tables 1 and 2, based on the obtained input information 1301, 1302, 1303, 1304, and 1305, can generate corresponding troubleshooting guidance information. For example, if the user inputs "Yes" for information 1301, the corresponding troubleshooting guidance information generated will be "Check robot accuracy (Robot_Absolute_Accuracy)". For information 1305, considering that the robot arm's evaluation may be inaccurate (due to differences in calibration and gripping space), if there are no robot arm-related checks in the main inspection items, checks will also be performed in the secondary inspection items, generating relevant troubleshooting guidance information.

[0200] Based on the above embodiments, in one possible implementation, if there is a relative translation between the grasping posture and the initial grasping posture, step S1006 may further include: obtaining the maximum residual and the average residual corresponding to the calibration extrinsic parameters based on multiple sets of extrinsic parameter calibration data and calibration extrinsic parameters; obtaining the maximum error ratio of the camera in the horizontal and vertical axes; and obtaining the maximum error ratio of the camera in the vertical axis based on whether the main error source in the vertical axis is the camera, the second pose of the reference position, and the second poses of other positions. The reference position and other positions are determined based on the coordinate values ​​of the calibration plate in the camera coordinate system. The coordinates of the calibration plate in the camera coordinate system are obtained based on the external parameter calibration data and the nominal dimensions of the calibration plate. The maximum error ratio of the robotic arm is obtained based on the maximum robotic arm error and the distances from other positions to the reference position. The maximum robotic arm error is obtained based on the translational errors of other positions relative to the reference position and the camera error. The camera error includes the horizontal and vertical axis errors and the vertical axis error corresponding to other positions. The horizontal and vertical axis errors are obtained based on the distances from other positions to the reference position and scalar errors. The vertical axis error is determined based on whether the main error source in the vertical direction is the camera or the second pose of the reference position. The second poses of other positions are obtained; the maximum systematic error in the horizontal and vertical directions is determined as the sum of the maximum error ratio of the camera in the horizontal and vertical directions and the maximum error ratio of the robotic arm; the maximum systematic error in the vertical direction is determined as the sum of the maximum error ratio of the camera in the vertical direction and the maximum error ratio of the robotic arm; if the average residual is less than the seventh threshold and the maximum residual is less than the eighth threshold, or if the maximum error ratio of the camera in the horizontal and vertical directions is less than the ninth threshold and the maximum error ratio of the camera in the vertical direction is less than the tenth threshold, then troubleshooting guidance information is generated based on the maximum systematic error in the horizontal and vertical directions, the maximum systematic error in the vertical direction, the maximum allowable error in the horizontal and vertical directions, and the maximum allowable error in the vertical direction. The maximum allowable error in the horizontal and vertical directions and the maximum allowable error in the vertical direction are determined based on the accuracy requirement information; if the maximum residual is greater than the eleventh threshold, then troubleshooting guidance information is generated directly. The eleventh threshold is greater than the eighth threshold; if the maximum residual is greater than or equal to the eighth threshold and less than or equal to the eleventh threshold, then troubleshooting guidance information is generated based on the main error source. The main error source is the main error source in the vertical direction determined based on the second pose of the reference position and the second poses of other positions.

[0201] For example, the input information in this embodiment may include:

[0202] (1) Path to the external parameters folder (Extri Folder). The external parameters folder includes the nominal dimensions of the calibration board in the configuration file, N sets of external parameter calibration data collected in the external parameter calibration process, and calibration external parameters.

[0203] (2) The relationship between the grasping pose and the grasping pose when the grasping error occurs (Grasp Pose Relation). When this value has obvious movement in the xyz axis direction or obvious rotation and movement in the xyz axis direction, it will be checked through the input information and external parameter data in (3) to (6) below.

[0204] (3) The maximum allowable error in the horizontal direction under the camera coordinate system (e.g., expressed as maxErrorAlongXY).

[0205] (4) The distance moved in the horizontal direction in the camera coordinate system (e.g., represented by distAlongXY).

[0206] (5) The maximum allowable error in the vertical direction of the camera coordinate system (e.g., expressed as maxErrorAlongZ).

[0207] (6) The distance moved in the vertical direction in the robot arm coordinate system (e.g., represented by distAlongZ).

[0208] Based on the input information, the output information obtained is the troubleshooting guidance information. Specifically, in this embodiment, generating the troubleshooting guidance information may include the following steps:

[0209] (1) Based on multiple sets of extrinsic calibration data and calibration extrinsic parameters, obtain the maximum residual corresponding to the calibration extrinsic parameters (e.g., using error). max (represented by) and residual mean (e.g., using error) mean express).

[0210] (2) When acquiring camera error and robotic arm error, the description of the relevant steps in determining the inspection results corresponding to the preset inspection items in the above embodiments can be referred to. These mainly include: acquiring the scalar error ratio of the camera corresponding to the calibration plate at each position in the external parameter calibration data; acquiring the maximum error ratio of the camera in the horizontal and vertical directions (e.g., using ratio). cam_xy_max (Representation); Obtain the coordinates of the calibration plate at each position in the camera coordinate system from the external parameter calibration data; Obtain the x and y axis errors at other positions. error Determine whether the main source of error in the vertical direction is the camera; based on whether the main source of error in the vertical direction is the camera, obtain the camera's z-axis error z. error and the corresponding vertical axis error ratio z And obtain the maximum error in the z-direction and the corresponding moving distance dist. max ; Obtain the maximum error ratio of the camera in the vertical direction (e.g., using ratio) cam_z_max (represented as: ratio) cam_z_max =z error / distmax The error at each other position relative to the reference position is: error camera =[xy err or, z error ], get error camera The modulus, where the maximum value is denoted as ||error. camera || max The average value is denoted as ||error camera || mean ; Get other locations Fixed T Boardi The error value δt of the translation from the reference position system Thus, the error of the robotic arm at each position can be obtained as: error robot =max(||δt) system ||-||error camera ||,0), determine the maximum robot arm error. robot_max and average robotic arm error robot_mean ; Obtain the distance dist from other locations to the reference location i This allows us to determine the maximum distance dist. max ; Obtain the maximum error ratio of the robotic arm in the vertical direction (e.g., using ratio). robot_max (represented as: ratio) robot_max =error robot_max / dist max Determine the maximum systematic error in the horizontal and vertical directions (i.e., the horizontal direction) using a ratio. system_xy_max (represented as: ratio) system_xy_max =ratio cam_xy_max +ratio robot_max Determine the maximum systematic error in the vertical direction (i.e., the vertical direction) using a ratio. system_z_max (represented as: ratio) system_z_max =ratio cam_z_max +ratio robot_max Based on the input information maxErrorAlongXY and distAlongXY, obtain the maximum allowable error in the horizontal and vertical directions (e.g., using ratio). tolerance_xy_max (represented as: ratio) tolerance_xy_max =maxErrorAlongXY / distAlongXY, based on the input information maxErrorAlongZ and distAlongZ, obtain the maximum allowable error in the vertical direction (e.g., using ratio). tolerance_z_max (represented as: ratio) tolerance_z_max=maxErrorAlongZ / distAlongZ.

[0211] The seventh threshold is, for example, 0.0005 mm, the eighth threshold is, for example, 0.001 mm, the ninth threshold is, for example, 0.2%, the tenth threshold is, for example, 0.6%, and the eleventh threshold is, for example, 0.02 mm. When the system residual is small (error... mean <0.0005mm and error max When <0.001mm, or when the camera accuracy is high (ratio) cam_xy_max <0.2% and ratio cam_z_max When the ratio is <0.6%, it is determined that the system accuracy cannot be improved by correction methods; if the ratio is <0.6%, it is determined that the system accuracy cannot be improved by correction methods. system_xy_max >ratio tolerance_xy_max or ratio system_z_max >ratio tolerance_z_max If the error is high, the generated troubleshooting guidance information will be TechnicalStaffDueToHighRequirement in Table 1; otherwise, no further expansion will be made. When the system residual is large (error... max When the error is greater than 0.02 mm, the troubleshooting guidance information is directly generated as CheckExtrinsicFileReport in Table 1. For other system residuals, the inspection items are expanded according to the current main error source.

[0212] Further, optionally, generating troubleshooting guidance information based on the primary error source may include: if the primary error source is the camera, generating troubleshooting guidance information based on the maximum value of camera errors at other locations, the average value of camera errors, the maximum robotic arm error, the average robotic arm error, the maximum systematic error in the horizontal and vertical axes, the maximum systematic error in the vertical axis, the maximum allowable error in the horizontal and vertical axes, and the maximum allowable error in the vertical axis; wherein, the average robotic arm error is obtained based on the translation error of other locations relative to the reference position and the camera error; if the primary error source is the robotic arm, generating troubleshooting guidance information based on the maximum value of camera errors at other locations, the maximum robotic arm error, the maximum percentage of the robotic arm error, the maximum systematic error in the horizontal and vertical axes, the maximum systematic error in the vertical axis, the maximum allowable error in the horizontal and vertical axes, and the maximum allowable error in the vertical axis; if the primary error source is neither the camera nor the robotic arm, generating troubleshooting guidance information based on the maximum systematic error in the horizontal and vertical axes, the maximum systematic error in the vertical axis, the maximum allowable error in the horizontal and vertical axes, and the maximum allowable error in the vertical axis.

[0213] In this embodiment, specifically, (1)||error camera || max errorrobot_max And ||error camera || mean error robot_mean The main source of error is the camera; if the ratio system_xy_max >ratio tolerance_xy_max or ratio system_z_max >ratio tolerance_z_max If the ratio is not specified, the troubleshooting guide will be expanded to include Intrinsic_Parameters_Error from Table 1; robot_max >ratio cam_xy_max If so, the troubleshooting guide will be expanded to include Robot_Absolute_Accuracy from Table 1; otherwise, no further expansion will be made. (2) error robot_max >min(||error camera || max *2.0,||error camera || max +0.001m), the main source of error is the robotic arm. When the robotic arm error is large (error... robot_max / dist max >0.01), the troubleshooting guidance information will be expanded to include Robot_Absolute_Rotation_Accuracy in Table 1; when the robotic arm error is not significantly abnormal, and the ratio system_xy_max >ratio tolerance_xy_max or ratio system_z_max >ratio tolerance_z_max If the error is not found in the camera or the robotic arm, then the troubleshooting guide will be expanded to include Robot_Absolute_Accuracy in Table 1. Otherwise, no further expansion will be made. (3) If the main error source is neither the camera nor the robotic arm, i.e., there is no main error source, then if the ratio system_xy_max >ratio tolerance_xy_max or ratio system_z_max >ratio tolerance_z_max If the error occurs, the troubleshooting guide will be expanded to include Robot_Absolute_Accuracy and Intrinsic_Parameters_Error from Table 1; otherwise, no further expansion will be provided.

[0214] Based on the above embodiments, in another possible implementation, if there is a relative rotation between the grasping posture and the initial grasping posture, step S1006 may further include: obtaining the third pose of the calibration board in the fixed coordinate system in each set of external parameter verification data according to the calibration external parameters and multiple sets of external parameter verification data; determining the maximum difference based on the difference between every two third poses; and generating troubleshooting guidance information based on the maximum difference and accuracy requirement information.

[0215] For example, the input information in this embodiment may include:

[0216] (1) Path to the external parameter folder (i.e., Extri Folder), which includes the calibration external parameters.

[0217] (2) Path to the external parameter verification data folder (i.e., Validate Folder), which includes the nominal dimensions of the calibration board in the configuration file and three additional sets of external parameter calibration data.

[0218] (3) The relationship between the grasping pose and the grasping pose when the grasping error occurs (Grasp Pose Relation). When this value has obvious rotation or obvious rotation and movement in the xyz axis direction, it will be checked through the input information and external parameter data in (4) as follows.

[0219] (4) When the grasping posture is rotated, the maximum allowable error (e.g., expressed as maxErrorOfRotPoses).

[0220] Based on the input information, the output information obtained is the troubleshooting guidance information. Specifically, in this embodiment, generating the troubleshooting guidance information may include the following steps:

[0221] (1) Based on the calibration extrinsic parameters and multiple sets of extrinsic parameter verification data in the Validate Folder, obtain the third pose of the calibration plate in the fixed coordinate system in each set of extrinsic parameter verification data.

[0222] (2) Obtain the difference between every two third poses (e.g., represented by δT):

[0223]

[0224] (3) Based on δT, the maximum difference can be determined, and based on the maximum difference, the maximum modulus can be obtained (for example, using ||δt|| to represent ||δT||). max ), ||δt|| max The maximum translational error that can be introduced when the grasping posture rotates, similar to the teaching method of grasping.

[0225] (4) Compare the input maxErrorOfRotPoses with ||δt|| max When ||δt|| max When `maxErrorOfRotPoses` is exceeded, it indicates that the current external parameter error is greater than expected. The troubleshooting guidance information will be expanded to include `Robot_Absolute_Rotation_Accuracy` from Table 1. In other cases, no additional expansion will be made.

[0226] In addition to the two cases mentioned above where the grasping posture is relatively translated or relatively rotated compared to the initial grasping posture, the following items may also be optionally included for inspection:

[0227] (1) Photo Position Related Prompts: The input information is whether the photo position taken when creating the point cloud template is consistent with the photo position when the grasping error occurs in the EIH scenario (i.e., Photo Position Consistency), and the status. If it is not an EIH scenario, this item can be ignored; the output information obtained is the troubleshooting guidance information. Specifically, when the photo position taken when creating the point cloud template is inconsistent with the photo position when the grasping error occurs, set Photo Position Consistency to No, and the troubleshooting guidance information will be expanded to Robot_Absolute_Accuracy in Table 1; otherwise, no expansion will be added.

[0228] (2) Point Cloud Quality Check Prompt: The input information is whether the point cloud quality is normal (i.e., Is Point Cloud Quality Normal). The output information is the troubleshooting guidance information. Specifically, when the point cloud quality is abnormal, set Is Point Cloud Quality Normal to No, and the troubleshooting guidance information will be expanded to include Cloud_Quality in Table 2. Otherwise, no expansion will be added.

[0229] Based on the above embodiments, different crawling scenarios will be described in detail below.

[0230] Figure 14 A schematic diagram of a crawling scenario provided in the embodiments of this disclosure. Figure 1 ,like Figure 14As shown, in the deep-frame grasping scenario, a robotic arm is used for grasping. This embodiment does not limit the camera model or the robotic arm model. The stack size is approximately 1000mm*1000mm*1000mm, the grasping accuracy requirement is 2mm@500mm, and the template is located at the center of the stack. Initial deployment resulted in grasping errors that did not meet requirements, with a maximum error of 4.4mm@540mm. The output troubleshooting guidance was to quickly correct the camera. The result was that after quick camera correction, the grasping accuracy met requirements, with a maximum error of less than 2mm@550mm. Specific processing steps included:

[0231] (1) Perform external parameter calibration.

[0232] (2) Check the external parameter calibration report. The external parameter calibration report does not show any obvious abnormalities. Figure 15 A schematic diagram of the external parameter calibration report provided for embodiments of this disclosure. Figure 6 ,like Figure 15 As shown, you can click on virtual control 1501 to view the external parameter calibration report corresponding to 1502. The check results for the preset check items in the external parameter calibration report are all passed.

[0233] (3) Execute the deployment process, as shown in Table 3. Assuming the error threshold is 4mm, it is found during deployment that the capture error is greater than the error threshold of 4mm in the deep box capture scenario.

[0234] Table 3

[0235] direction Sources of error Distance traveled (mm) Error (mm) superior z direction 547.841260 3.406209 Down z direction 541.979458 4.439827 Left The direction of the plane containing xy 448.373618 2.060465 right The direction of the plane containing xy 477.343471 2.200254 forward The direction of the plane containing xy 389.007815 2.166286 back The direction of the plane containing xy 401.265427 2.269038

[0236] (4) Input information: Figure 16 This is an illustration of the generation of troubleshooting guidance information provided in the embodiments of this disclosure. Figure 1 ,like Figure 16 As shown, the input information required in step 1601 includes:

[0237] Extri Folder: Used to obtain the folder path of the external parameters input by the user;

[0238] Grasp Pose Relation: In the grasping scenario, the gripper needs to translate significantly, and the grasping pose relationship should have a significant movement in the xyz axis direction (Significant XYZMovement).

[0239] Photo Position Consistency: Select True for a single shooting pose;

[0240] Is Point Cloud Quality Normal: Check the point cloud quality. If the point cloud quality is found to be normal, select True.

[0241] Max Error Along XY: The maximum allowable error for this scenario is 2mm, or 0.002m.

[0242] Dist Along XY: Compared to the center position, the incoming material can have a translational change of up to 500mm along the xy plane, which is based on the user input of 0.5m;

[0243] Max Error Along Z: The maximum allowable error for this scenario is 2mm, or 0.002m.

[0244] Dist Along Z: Compared to the center position, the material can vary by up to 500mm along the z-axis, and the value of 0.5m is obtained from the user input.

[0245] It should be noted that, based on the preset full input information, the corresponding input information can be obtained according to the actual application scenario, such as obtaining partial input information to meet the actual application scenario.

[0246] (5) Generate investigation guidance information for reference. Figure 16 The troubleshooting guide in error 1602 mainly focuses on camera intrinsic parameter errors (Intrinsic_Parameters_Error), and the corresponding measure is to quickly correct the camera intrinsic parameters.

[0247] (6) Process according to the troubleshooting guidelines (quickly correct camera internal parameters).

[0248] (7) After redeploying the project and removing outliers, the maximum error is less than 2mm@500mm.

[0249] If the current accuracy still does not meet the requirements, you can check the main system errors after correction. Figure 17 This is an illustration of the generation of troubleshooting guidance information provided in the embodiments of this disclosure. Figure 2 ,like Figure 17 As shown, 1701 contains the input information, while 1702 provides troubleshooting guidance, primarily checking the robot's accuracy (Robot_Absolute_Accuracy). The limitation of the current system is attributed to insufficient robotic arm accuracy. Technical support personnel can also view the complete external parameter calibration report themselves through the output of the development steps.

[0250] Figure 18 A schematic diagram of a crawling scenario provided in the embodiments of this disclosure. Figure 2 ,like Figure 18As shown, in a near-plane disordered grasping scenario, a camera and robotic arm are used to grasp a stack of objects approximately 200*200*100mm in size, with a maximum rotation angle of ±100°. The grasping accuracy requirement is 2mm@200mm. The grasping template is located at the center of the bottom plane. Initial deployment revealed that the grasping error did not meet requirements, with a maximum of 3mm, occurring when the grasping pose rotated. The output troubleshooting guidance instructed checking the robotic arm rotation and extrinsic parameters. The information provided indicated that by verifying the position data, the teaching method could potentially introduce an error of up to 4mm. The result was that checking the robotic arm revealed a precision deviation (near a singularity). Lowering the arm by a certain height and recalibrating the extrinsic parameters resulted in a grasping outcome that met requirements. Specific processing steps included:

[0251] (1) Perform external parameter calibration.

[0252] (2) Upon reviewing the external parameter calibration report, a problem was discovered. There was a significant error in the rotational attitude. Figure 19 A schematic diagram of the external parameter calibration report provided for embodiments of this disclosure. Figure 7 ,like Figure 19 As shown, the robot absolute accuracy check result in the external parameter calibration report indicates that there is a risk, and risk positioning suggestions are provided.

[0253] (3) Check the robotic arm. No obvious problems were found at the zero point. Continue to grasp. It was found that the grasping accuracy was poor, especially when rotating. The maximum grasping error of all translation data was less than 1mm, and the maximum error of rotation data was 3.1mm (the maximum rotation was only ±90°. If it rotated 180°, the error would be even greater).

[0254] (4) Input information obtained: Figure 20 This is an illustration of the generation of troubleshooting guidance information provided in the embodiments of this disclosure. Figure 3 ,like Figure 20 As shown, the input information required in 2001 includes:

[0255] Extri Folder: Used to obtain the folder path of the external parameters input by the user;

[0256] Validate Folder: Collect rotation pose data;

[0257] Grasp Pose Relation: In the grasping scenario, the gripper needs to have obvious translation and rotation. Choose one with obvious rotation and movement in the xyz axis direction (Significant Rotation And XYZMovement).

[0258] Photo Position Consistency: Select True for a single shooting pose;

[0259] Is Point Cloud Quality Normal: Check the point cloud quality. If the point cloud quality is found to be normal, select True.

[0260] Max Error of Rotations: The maximum allowable error during gripping is 2mm, or 0.002m.

[0261] Max Error Along XY: The maximum allowable error for this scenario is 2mm, or 0.002m.

[0262] Dist Along XY: Compared to the center position, the incoming material can have a maximum translational change of 200mm, and the user input is 0.2m;

[0263] Max Error Along Z: The maximum allowable error for this scenario is 2mm, or 0.002m.

[0264] Dist Along Z: Compared to the center position, the incoming material can have a translational change of up to 100mm, and the user input is 0.1m.

[0265] (5) Generate investigation guidance information for reference. Figure 20 The 2002 troubleshooting guidelines mainly recommend checking the rotation accuracy of the robotic arm and evaluating external parameter errors, as well as using the teaching method to rotate 180°, with a maximum rotation error of 4mm.

[0266] (6) Follow the troubleshooting guidelines to process the problem. The results include: the robot arm has certain problems with its posture accuracy, especially near the calibration area. However, this is an unsolvable problem, possibly because the accuracy is poor near singular points. Try to reduce the robot arm calibration space and recalibrate the external parameters.

[0267] (7) Update the investigation guidelines information. Figure 21 This is a schematic diagram of the updated troubleshooting guidance information provided in one embodiment of this disclosure, such as... Figure 21 As shown, using the teaching method, the maximum introduced error is 2.3mm, which is larger than the input requirement of 2mm. Therefore, it is still recommended to evaluate the external parameters. However, it is known that the rotation angle used is not ±180, but ±90. Therefore, try to grab first.

[0268] (8) Grasping status after updating the troubleshooting guidance information: The maximum grasping error is 1.49mm, which occurs when the grasping posture rotates significantly, meeting the 2mm grasping requirement.

[0269] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.

[0270] Figure 22 This is a schematic diagram of the structure of a risk positioning device provided in an embodiment of the present disclosure, as shown below. Figure 22 As shown, the risk location device 2200 of this embodiment includes: an acquisition module 2201, a first processing module 2202, and a determination module 2203. Wherein:

[0271] The acquisition module 2201 is used to acquire the calibration extrinsic parameters and the first pose of the calibration plate in the fixed coordinate system in response to the completion of camera extrinsic parameter calibration.

[0272] The first processing module 2202 is used to check preset check items based on multiple sets of extrinsic calibration data, calibration extrinsic parameters and first pose collected during camera extrinsic calibration, and to determine the check results corresponding to the preset check items. The preset check items include at least one of Euler angle type check, camera accuracy check and robot absolute accuracy check in the extrinsic calibration data. The extrinsic calibration data includes the flange pose and the coordinate values ​​of the marker points on the calibration plate in the camera coordinate system.

[0273] The determination module 2203 is used to determine the corresponding risk positioning suggestions based on the inspection result if the inspection result is that the inspection fails.

[0274] In some embodiments, the preset check items include Euler angle type check, and the flange pose includes relative rotation data. The first processing module 2202 is specifically used to: determine whether to check the Euler angle type based on multiple sets of extrinsic calibration data, calibration extrinsic parameters and first pose collected during camera extrinsic calibration; if so, based on the flange pose, obtain multiple groups containing a preset number of relative rotation data, the multiple groups including a first group of reference pose data used to determine relative rotation; perform Euler angle type search according to the first group to determine the optimal Euler angle type; if the current Euler angle type is different from the optimal Euler angle type, determine that the check result corresponding to the Euler angle type check is a check failure.

[0275] Optionally, the first processing module 2202 is further configured to: for each of the multiple groups, perform an Euler angle value search based on the group and the optimal Euler angle type to determine the optimal Euler angle value corresponding to each group; update the relative rotation data based on the optimal Euler angle type and the optimal Euler angle value corresponding to each group to obtain updated extrinsic parameter calibration data; if the flange pose is different from the flange pose in the updated extrinsic parameter calibration data, determine that the check result corresponding to the Euler angle type check is a check failure.

[0276] Optionally, when the first processing module 2202 performs Euler angle type search based on the first group to determine the optimal Euler angle type, it specifically performs the following steps: for each relative rotation data in the first group, convert the relative rotation data into a first Euler angle value using the current Euler angle type; expand the first Euler angle value according to the Euler angle configuration properties to obtain multiple expanded second Euler angle values; combine each second Euler angle value with each of the multiple preset Euler angle types to obtain multiple combinations; obtain the first extrinsic parameter corresponding to each of the multiple combinations; for each of the multiple combinations, obtain the first residual corresponding to the combination based on the first extrinsic parameter; and determine the Euler angle type corresponding to the combination with the smallest first residual among the first residuals as the optimal Euler angle type.

[0277] Optionally, when the first processing module 2202 performs Euler angle value search based on the group and the optimal Euler angle type for each of the multiple groups to determine the optimal Euler angle value for each group, it specifically performs the following steps: for each of the multiple groups, converting each relative rotation data in the group into a third Euler angle value using the current Euler angle type; expanding the third Euler angle value according to the Euler angle configuration properties to obtain multiple expanded fourth Euler angle values; combining each fourth Euler angle value with the optimal Euler angle type to obtain multiple combinations; obtaining the second extrinsic parameter corresponding to each of the multiple combinations; for each of the multiple combinations, obtaining the second residual corresponding to the combination based on the second extrinsic parameter; and for each group, determining the fourth Euler angle value corresponding to the second extrinsic parameter with the smallest difference from the calibration extrinsic parameter as the optimal Euler angle value for that group.

[0278] In some embodiments, the determining module 2203 is specifically used to: if the inspection result is that the inspection fails, determine the corresponding risk positioning suggestion based on the inspection result, including replacing the current Euler angle type with the optimal Euler angle type and / or replacing the calibration extrinsic parameter with the updated extrinsic parameter, wherein the updated extrinsic parameter is obtained based on the updated extrinsic parameter calibration data.

[0279] Optionally, when the first processing module 2202 determines whether to check the Euler angle type based on multiple sets of extrinsic calibration data, calibration extrinsic parameters, and the first pose acquired during camera extrinsic calibration, it specifically performs the following: for each set of extrinsic calibration data, based on the extrinsic calibration data and calibration extrinsic parameters, obtains the second pose of the calibration board in a fixed coordinate system corresponding to the extrinsic calibration data; obtains the maximum error modulus based on the difference between the first pose and each second pose; if the maximum error modulus is greater than a first threshold, it determines to perform an Euler angle type check; if the maximum error modulus is less than or equal to the first threshold, it determines not to perform an Euler angle type check.

[0280] In some embodiments, the preset check items include camera accuracy checks. The first processing module 2202 is specifically used for: obtaining the scalar error of the camera corresponding to each position of the calibration plate and the coordinate value of the calibration plate in the camera coordinate system based on the extrinsic calibration data and the nominal size of the calibration plate; determining the reference position and other positions based on the coordinate value of the calibration plate in the camera coordinate system, wherein the other positions are the positions of the calibration plate other than the reference position; determining whether the main error source in the vertical direction is the camera based on the second pose of the reference position and the second pose of the other positions, wherein the second pose is the pose of the calibration plate in the fixed coordinate system corresponding to the extrinsic calibration data obtained based on the extrinsic calibration data and the calibration extrinsic parameters; if the main error source in the vertical direction is the camera, then the vertical coordinates in the second poses of the other positions and the second poses of the other positions relative to the second pose of the reference position are... The translation error is fitted, and based on the fitting result, the vertical axis error and the vertical axis error ratio corresponding to other positions are determined respectively; or, if the main error source in the vertical axis direction is not the camera, the vertical axis error and the vertical axis error ratio corresponding to other positions are determined based on the scalar error and the vertical axis difference between other positions and the reference position respectively; if the horizontal and vertical axis error ratios corresponding to any other position are greater than the second threshold, or the vertical axis error ratios corresponding to any other position are greater than the third threshold, the inspection result corresponding to the camera accuracy check is determined to be an inspection failure. The horizontal and vertical axis error ratios are obtained based on the distances between other positions and the reference position and the scalar error, or, if the main error source in the vertical axis direction is the camera, they are determined based on the distances between other positions and the reference position, the deviations between other positions and the reference position, and the scalar error.

[0281] Optionally, the first processing module 2202 is further configured to: if the ratio of horizontal to vertical axis error corresponding to any position in other positions is greater than the fourth threshold, or the ratio of vertical axis error corresponding to any position in other positions is greater than the fourth threshold by a preset multiple, then determine that the inspection result corresponding to the camera accuracy inspection is risky, the fourth threshold is less than the second threshold, and the fourth threshold by a preset multiple is less than the third threshold.

[0282] Optionally, when the first processing module 2202 determines whether the main error source in the vertical direction is the camera based on the second pose at the reference position and the second poses at other positions, it specifically performs the following steps: obtaining the correlation between the vertical coordinates and the translation error in the vertical direction based on the vertical coordinates and the translation error in the vertical direction in the second poses at other positions; obtaining the significance of the correlation based on the correlation; if the significance is less than a fifth threshold, then determining that the main error source in the vertical direction is the camera; if the significance is greater than or equal to the fifth threshold, then determining that the main error source in the vertical direction is not the camera.

[0283] In some embodiments, the determining module 2203 is specifically used to: if the camera accuracy check result is a failure, determine the corresponding risk positioning suggestion based on the check result, including checking whether there is a problem with the calibration board and / or performing manual processing; if the camera accuracy check result is a risk, determine the corresponding risk positioning suggestion based on the check result, including checking whether there is a problem with the calibration board, correcting the camera intrinsic parameters, and replacing the camera, at least one of these.

[0284] In some embodiments, the preset inspection items include robot absolute accuracy inspection. The first processing module 2202 is specifically used to: obtain the robot arm errors corresponding to other positions based on the translation errors of other positions relative to the reference position and the camera errors. The camera errors include the horizontal and vertical axis errors and the vertical axis errors corresponding to other positions. The horizontal and vertical axis errors are obtained based on the distances of other positions to the reference position and the scalar errors. The reference position and other positions are determined based on the coordinate values ​​of the calibration plate in the camera coordinate system. The other positions are the positions of the calibration plate other than the reference position. Determine the maximum robot arm error among multiple robot arm errors. Determine the maximum distance among the distances of other positions to the reference position. If the ratio of the maximum robot arm error to the maximum distance is greater than a sixth threshold, then the inspection result corresponding to the robot absolute accuracy inspection is determined to be risky.

[0285] In some embodiments, the determining module 2203 is specifically used to: if the inspection result of the robot absolute accuracy inspection is that there is a risk, determine the corresponding risk positioning suggestions based on the inspection result, including checking the robot accuracy and / or checking whether the robot zero point is accurate.

[0286] Optionally, the determination module 2203 is also used to: after determining the corresponding risk positioning recommendations based on the inspection results, generate an external parameter calibration report, which includes preset inspection items, the inspection results corresponding to the preset inspection items, and risk positioning recommendations for the inspection results if the inspection results are not passed; and output the external parameter calibration report.

[0287] In some embodiments, the risk location device 2200 further includes a second processing module 2204, which, after the determination module 2203 determines the corresponding risk location suggestion based on the inspection results, if there is an inaccuracy problem during the deployment process, responds to the user's input operation, obtains the input target information, the target information including the nominal size of the calibration board, multiple sets of external parameter calibration data, calibration external parameters, multiple sets of external parameter verification data, relevant information of the capture scene, and accuracy requirement information; based on the target information, generates investigation guidance information, which is used to guide risk location; and outputs the investigation guidance information.

[0288] Optionally, if there is a relative translation between the grasping posture and the initial grasping posture, the second processing module 2204, when generating investigation guidance information based on target information, specifically performs the following: Based on multiple sets of extrinsic parameter calibration data and calibration extrinsic parameters, obtain the maximum residual and average residual value corresponding to the calibration extrinsic parameters; obtain the maximum error ratio of the camera in the horizontal and vertical axes; based on whether the main error source in the vertical axis is the camera, the second pose of the reference position, or the second pose of other positions, obtain the maximum error ratio of the camera in the vertical axis. The reference position and other positions are determined based on the coordinate values ​​of the calibration board in the camera coordinate system. The coordinates of the calibration board in the camera coordinate system are obtained based on the external parameter calibration data and the nominal dimensions of the calibration board. The maximum error ratio of the robotic arm is obtained based on the maximum robotic arm error and the distances from other positions to the reference position. The maximum robotic arm error is obtained based on the translational errors of other positions relative to the reference position and the camera error. The camera error includes the horizontal and vertical axis errors and the vertical axis error corresponding to other positions. The horizontal and vertical axis errors are obtained based on the distances from other positions to the reference position and scalar errors. The vertical axis error is determined based on whether the main error source in the vertical axis direction is the camera or the second position of the reference position. The pose and other second poses are obtained; the maximum systematic error in the horizontal and vertical directions is determined as the sum of the maximum error ratio of the camera in the horizontal and vertical directions and the maximum error ratio of the robotic arm; the maximum systematic error in the vertical direction is determined as the sum of the maximum error ratio of the camera in the vertical direction and the maximum error ratio of the robotic arm; if the average residual is less than the seventh threshold and the maximum residual is less than the eighth threshold, or, the maximum error ratio of the camera in the horizontal and vertical directions is less than the ninth threshold and the maximum error ratio of the camera in the vertical direction is less than the tenth threshold, then based on the maximum systematic error in the horizontal and vertical directions and the maximum systematic error in the vertical direction... The system calculates the total error, the maximum allowable error in the horizontal and vertical directions, and the maximum allowable error in the vertical direction, and generates troubleshooting guidance information. The maximum allowable error in the horizontal and vertical directions and the maximum allowable error in the vertical direction are determined based on the accuracy requirements. If the maximum residual is greater than the eleventh threshold, troubleshooting guidance information is generated directly. The eleventh threshold is greater than the eighth threshold. If the maximum residual is greater than or equal to the eighth threshold and less than or equal to the eleventh threshold, troubleshooting guidance information is generated based on the main error source. The main error source is the main error source in the vertical direction determined based on the second pose of the reference position and the second pose of other positions.

[0289] Optionally, when the second processing module 2204 generates troubleshooting guidance information based on the main error source, it specifically performs the following: If the main error source is the camera, it generates troubleshooting guidance information based on the maximum value of camera error at other positions, the average value of camera error, the maximum robotic arm error, the average robotic arm error, the maximum systematic error in the horizontal and vertical axes, the maximum systematic error in the vertical axis, the maximum allowable error in the horizontal and vertical axes, and the maximum allowable error in the vertical axis; wherein, the average robotic arm error is obtained based on the translation error of other positions relative to the reference position and the camera error; if the main error source is the robotic arm, it generates troubleshooting guidance information based on the maximum value of camera error at other positions, the maximum robotic arm error, the maximum error ratio of the robotic arm, the maximum systematic error in the horizontal and vertical axes, the maximum systematic error in the vertical axis, the maximum allowable error in the horizontal and vertical axes, and the maximum allowable error in the vertical axis; if the main error source is neither the camera nor the robotic arm, it generates troubleshooting guidance information based on the maximum systematic error in the horizontal and vertical axes, the maximum systematic error in the vertical axis, the maximum allowable error in the horizontal and vertical axes, and the maximum allowable error in the vertical axis.

[0290] Optionally, if there is a relative rotation between the grasping posture and the initial grasping posture, the second processing module 2204, when generating troubleshooting guidance information based on target information, specifically performs the following: based on the calibration extrinsic parameters and multiple sets of extrinsic parameter verification data, obtains the third pose of the calibration board in the fixed coordinate system in each set of extrinsic parameter verification data; determines the maximum difference based on the difference between every two third poses; and generates troubleshooting guidance information based on the maximum difference and accuracy requirement information.

[0291] The apparatus disclosed herein can be used to execute the technical solutions of any of the method embodiments shown above. Its implementation principle and technical effects are similar, and will not be repeated here.

[0292] Figure 23 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present disclosure. Figure 23 As shown, the electronic device 2300 may include at least one processor 2301 and a memory 2302.

[0293] The memory 2302 is used to store programs. Specifically, the program may include program code, which includes computer-executable instructions.

[0294] The memory 2302 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0295] The processor 2301 executes computer execution instructions stored in the memory 2302 to implement the risk location method described in the foregoing method embodiments. The processor 2301 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this disclosure. Specifically, when implementing the risk location method described in the foregoing method embodiments, the electronic device may be, for example, a server or other electronic device with processing capabilities.

[0296] Optionally, the electronic device 2300 may also include a communication interface 2303. In specific implementations, if the communication interface 2303, memory 2302, and processor 2301 are implemented independently, they can be interconnected via a bus to complete communication. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc., but this does not imply that there is only one bus or one type of bus.

[0297] Optionally, in a specific implementation, if the communication interface 2303, memory 2302 and processor 2301 are integrated on a single chip, then the communication interface 2303, memory 2302 and processor 2301 can communicate through an internal interface.

[0298] This disclosure also provides a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the risk location method described above.

[0299] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the risk location method described above.

[0300] The aforementioned computer-readable storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0301] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in a risk location device.

[0302] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0303] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit them. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this disclosure.

Claims

1. A risk positioning method, characterized in that, include: In response to the completion of camera extrinsic calibration, the calibration extrinsic parameters and the first pose of the calibration plate in the fixed coordinate system are obtained; Based on multiple sets of extrinsic parameter calibration data collected during the camera extrinsic parameter calibration, the calibration extrinsic parameters, and the first pose, preset inspection items are checked to determine the inspection results corresponding to the preset inspection items; the preset inspection items include at least one of Euler angle type check, camera accuracy check, and robot absolute accuracy check in the extrinsic parameter calibration data, and the extrinsic parameter calibration data includes the flange pose and the coordinate values ​​of the marker points on the calibration plate in the camera coordinate system; If the inspection result is that the inspection fails, then the corresponding risk positioning recommendations will be determined based on the inspection result.

2. The risk location method according to claim 1, characterized in that, The preset check items include the Euler angle type check, the flange pose includes relative rotation data, and the preset check items are checked based on multiple sets of extrinsic calibration data collected during the camera extrinsic calibration, the calibration extrinsic parameters, and the first pose to determine the check results corresponding to the preset check items, including: Based on the multiple sets of extrinsic parameter calibration data collected during the camera extrinsic parameter calibration, the calibration extrinsic parameters, and the first pose, it is determined whether to check the Euler angle type; If so, based on the flange pose, obtain multiple groups containing a preset number of relative rotation data, wherein the multiple groups include a first group of reference pose data for determining relative rotation; Based on the first group, perform an Euler angle type search to determine the optimal Euler angle type; If the current Euler angle type is different from the optimal Euler angle type, then the check result corresponding to the Euler angle type check is determined to be a check failure.

3. The risk location method according to claim 2, characterized in that, Also includes: For each of the plurality of groups, an Euler angle value search is performed based on the group and the optimal Euler angle type to determine the optimal Euler angle value corresponding to each group; The relative rotation data is updated according to the optimal Euler angle type and the optimal Euler angle value corresponding to each group to obtain the updated extrinsic parameter calibration data; If the flange pose is different from the flange pose in the updated extrinsic calibration data, then the check result corresponding to the Euler angle type check is determined to be a check failure.

4. The risk location method according to claim 2, characterized in that, The step of performing Euler angle type search based on the first group to determine the optimal Euler angle type includes: For each relative rotation data in the first group, the relative rotation data is converted into a first Euler angle value using the current Euler angle type; Based on the properties of Euler angle configuration, the first Euler angle value is expanded to obtain multiple expanded second Euler angle values; Each of the second Euler angle values ​​is combined with each of the multiple preset Euler angle types to obtain multiple combinations; Obtain the first extrinsic parameter corresponding to each of the plurality of combinations; For each of the plurality of combinations, based on the first extrinsic parameter corresponding to the combination, obtain the first residual corresponding to the combination; The Euler angle type corresponding to the combination of the smallest first residuals among all the first residuals is determined as the optimal Euler angle type.

5. The risk location method according to claim 3, characterized in that, The step of performing an Euler angle value search for each of the plurality of groups, based on the group and the optimal Euler angle type, to determine the optimal Euler angle value corresponding to each group includes: For each of the plurality of groups, each relative rotation data in the group is converted into a third Euler angle value using the current Euler angle type; Based on the properties of Euler angle configuration, the third Euler angle value is extended to obtain multiple extended fourth Euler angle values; Each of the fourth Euler angle values ​​is combined with the optimal Euler angle type to obtain multiple combinations; Obtain the second extrinsic parameter corresponding to each of the plurality of combinations; For each of the plurality of combinations, the second residual corresponding to the combination is obtained based on the second extrinsic parameter corresponding to the combination; For each group, the fourth Euler angle value corresponding to the second extrinsic parameter with the smallest difference from the calibration extrinsic parameter is determined as the optimal Euler angle value for that group.

6. The risk location method according to claim 3, characterized in that, If the inspection result is that the inspection fails, then a corresponding risk positioning recommendation is determined based on the inspection result, including: If the inspection result is that the inspection fails, the corresponding risk positioning suggestions are determined based on the inspection result, including replacing the current Euler angle type with the optimal Euler angle type and / or replacing the calibration extrinsic parameters with updated extrinsic parameters, wherein the updated extrinsic parameters are obtained based on the updated extrinsic parameter calibration data.

7. The risk positioning method according to any one of claims 1 to 6, characterized in that, After determining the corresponding risk positioning recommendations based on the inspection results, the method further includes: If there are problems with inaccurate targeting during deployment, the system responds to user input and obtains the target information, which includes the nominal size of the calibration board, the multiple sets of external parameter calibration data, the calibration external parameters, the multiple sets of external parameter verification data, relevant information about the capture scenario, and accuracy requirement information. Based on the target information, investigation guidance information is generated, which is used to guide risk identification; Output the troubleshooting guidance information.

8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the risk location method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the risk location method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed, it implements the risk location method as described in any one of claims 1 to 7.

Images