Calibration method and device for pose relationship

Abstract

The embodiment of the present application relates to the technical field of robots, and discloses a kind of calibration method and equipment of pose relationship, which comprises: for sample vehicle, the observation sample information of target display surface of sample vehicle is observed by mechanical arm in each standard observation pose;Pose solution is carried out based on the observation sample information, and a plurality of candidate pose relationship solutions are obtained;Wherein, the pose relationship solution represents the pose transformation relationship of target display surface coordinate system relative to mechanical arm base coordinate system;The candidate pose relationship solution is screened and processed to obtain a reliable candidate solution;The reliable candidate solution is optimized and solved to obtain the target pose relationship of the sample vehicle.The technical scheme of the application can solve the problem of low accuracy and stability of pose transformation relationship in the prior art.

Description

Technical Field

[0001] This invention relates to the field of robotics, specifically to a method and apparatus for calibrating pose relationships. Background Technology

[0002] As the functions of intelligent vehicle cockpits continue to become more complex, in-vehicle screens (such as central control screens, instrument panel screens, and passenger-side screens) and interactive surfaces (such as touch panels and gesture recognition areas) have become the core entry points for human-machine interaction. In the processes of vehicle function regression testing, compatibility testing, human-machine interface (HMI) testing, vehicle interaction testing, and production line consistency testing, more and more scenarios require the use of robotic arms to simulate human hands performing click, swipe, drag, and multi-point combination touch operations on display interactive surfaces, in order to replace traditional manual testing methods.

[0003] To achieve high-precision and stable touch control of a robotic arm on a target display surface, it is first necessary to establish the spatial relationship between the robotic arm's base coordinate system and the target display surface's coordinate system. Current conventional calibration methods mostly rely directly on observation data to perform a single pose calculation, obtaining only a single solution as the final calibration result. This method is highly dependent on the quality of a single observation and a single solution, and is easily affected by factors such as image occlusion, uneven feature distribution, and imaging noise. The resulting pose relationship lacks accuracy and stability, making it difficult to meet the requirements of high-precision touch control testing. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention provide a method and apparatus for calibrating pose relationships, which solves the problem of low accuracy and stability of pose transformation relationships in the prior art.

[0005] According to a first aspect of the present invention, a method for calibrating pose relationships is provided, the method comprising:

[0006] For the sample vehicle, obtain the observation sample information of the target display surface of the sample vehicle observed by the robotic arm in various standard observation poses;

[0007] Based on the observed sample information, pose calculation is performed to obtain multiple sets of candidate pose relationship solutions; wherein, the pose relationship solution characterizes the pose transformation relationship between the target display surface coordinate system and the robot arm base coordinate system;

[0008] The candidate pose relation solutions are filtered to obtain reliable candidate solutions;

[0009] The reliable candidate solutions are optimized to obtain the target pose relationship of the sample vehicle.

[0010] In one alternative approach, the observation sample information includes observation features. Pose calculation is performed based on the observation sample information to obtain multiple sets of candidate pose relation solutions, including:

[0011] Based on the observation characteristics, the corresponding observation sample information is evaluated to obtain the quality score of the observation sample information;

[0012] Based on the quality score, the observation sample information is filtered to obtain valid observation sample information;

[0013] Pose calculation is performed based on valid observation sample information to obtain multiple sets of candidate pose relationship solutions.

[0014] In one alternative approach, the quality of the corresponding observed sample information is assessed based on the observed features to obtain a quality score for the observed sample information, including:

[0015] Based on the observation characteristics, determine the quality assessment indicators of the observation sample information corresponding to the observation characteristics; among them, the quality assessment indicators include at least one of the following: feature quantity, feature coverage uniformity, feature detection confidence, image clarity, image sharpness, boundary closure, corner stability, exposure rationality, and local occlusion ratio.

[0016] The quality score of the observed sample information is determined based on the quality assessment indicators of the observed sample information.

[0017] In one alternative approach, the reliable candidate solutions are optimized to obtain the target pose relationship of the sample vehicles, including:

[0018] Based on the preset objective function, the credible candidate solution is optimized to obtain the target pose relationship of the sample vehicle; wherein, the objective function is to minimize the sum of geometric errors after processing by the robust loss function, and the geometric error represents the error of the observation features in the observation sample information corresponding to the credible candidate solution. The geometric error is at least one of the following errors: reprojection error, contour alignment error, boundary distance error, and surface model constraint error.

[0019] In one alternative approach, the objective function is to minimize the sum of the geometric error and the bias penalty term after processing by the robust loss function; the bias penalty term characterizes the degree of spatial offset between the reliable candidate solution and the historical prior pose.

[0020] In one alternative approach, multiple standard observation poses are determined, including:

[0021] Obtain multiple candidate observation poses;

[0022] For each candidate observation pose, determine the visibility index, coverage contribution index, and risk index of the candidate observation pose;

[0023] A comprehensive score for candidate observation poses is determined based on visibility indicators, coverage contribution indicators, and risk indicators.

[0024] Candidate observation poses with a comprehensive score greater than a preset threshold are used as standard observation poses.

[0025] According to a second aspect of the present invention, a method for calibrating pose relationships is provided, the method comprising:

[0026] For the target test vehicle, obtain the target pose relationship with the sample vehicle corresponding to the target test vehicle;

[0027] Based on the target pose relationship, determine the reuse verification accuracy index;

[0028] If the reuse verification accuracy index meets the preset conditions, then the target pose relationship is determined to be the calibration pose relationship of the target test vehicle;

[0029] If the reuse verification accuracy index does not meet the preset conditions, the target pose relationship is corrected to obtain the calibration pose relationship of the target test vehicle.

[0030] Wherein, the target pose relationship is the target pose relationship of the sample vehicle as described in any one of the first aspects.

[0031] According to a third aspect of the present invention, a calibration apparatus for pose relationships is provided, comprising:

[0032] The first acquisition module is used to acquire observation sample information of the target display surface of the sample vehicle observed by the robotic arm in various standard observation poses.

[0033] The solution module is used to perform pose calculation based on the observed sample information to obtain multiple sets of candidate pose relationship solutions; wherein, the pose relationship solution represents the pose transformation relationship between the target display surface coordinate system and the robot arm base coordinate system;

[0034] The filtering module is used to filter the candidate pose relation solutions to obtain reliable candidate solutions;

[0035] An optimization module is used to optimize the reliable candidate solutions to obtain the target pose relationship of the sample vehicle.

[0036] According to a fourth aspect of the present invention, a calibration apparatus for pose relationships is provided, comprising:

[0037] The second acquisition module is used to acquire the target pose relationship between the target test vehicle and the sample vehicle corresponding to the target test vehicle.

[0038] The determination module is used to determine the reuse verification accuracy index based on the target pose relationship;

[0039] The verification module is used to determine the target pose relationship as the calibration pose relationship of the target test vehicle if the reuse verification accuracy index meets the preset conditions; and to correct the target pose relationship to obtain the calibration pose relationship of the target test vehicle if the reuse verification accuracy index does not meet the preset conditions.

[0040] Wherein, the target pose relationship is the target pose relationship of the sample vehicle as described in any one of the first aspects.

[0041] According to a fifth aspect of the present invention, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus;

[0042] The memory is used to store at least one executable instruction that causes the processor to perform the operation of the above-described pose relationship calibration method.

[0043] According to a sixth aspect of the present invention, a calibration test system is provided, comprising: a test vehicle, a robotic arm, and an electronic device, wherein the electronic device is used to perform the above-described calibration method for pose relationships.

[0044] According to a seventh aspect of the present invention, a computer-readable storage medium is provided, the storage medium storing at least one executable instruction that causes an electronic device / pose relationship calibration device to perform the following operations:

[0045] For the sample vehicle, obtain the observation sample information of the target display surface of the sample vehicle observed by the robotic arm in various standard observation poses;

[0046] Based on the observed sample information, pose calculation is performed to obtain multiple sets of candidate pose relationship solutions; wherein, the pose relationship solution characterizes the pose transformation relationship between the target display surface coordinate system and the robot arm base coordinate system;

[0047] The candidate pose relation solutions are filtered to obtain reliable candidate solutions;

[0048] The reliable candidate solutions are optimized to obtain the target pose relationship of the sample vehicle; or,

[0049] For the target test vehicle, obtain the target pose relationship with the sample vehicles corresponding to the target test vehicle;

[0050] Based on the target pose relationship, determine the reuse verification accuracy index;

[0051] If the reuse verification accuracy index meets the preset conditions, then the target pose relationship is determined to be the calibration pose relationship of the target test vehicle;

[0052] If the reuse verification accuracy index does not meet the preset conditions, the target pose relationship is corrected to obtain the calibration pose relationship of the target test vehicle.

[0053] This invention first collects observation sample information of the robotic arm under multiple standard observation poses on a sample vehicle. Based on this information, multiple sets of candidate pose relationship solutions are calculated. Then, outlier solutions are removed through a screening process to obtain reliable candidate solutions. Finally, optimization is performed based on these reliable candidate solutions to obtain the target pose relationship corresponding to the sample vehicle. This eliminates the need to rely on a single solution obtained from a single calculation as the calibration result. By generating multiple sets of candidate solutions, the range of options is enriched. The screening process removes outlier solutions with large deviations, reducing interference from poor data. Global optimization is then performed based on the refined reliable candidate solutions, effectively mitigating calibration bias problems caused by single observation noise and poor feature quality. This continuously improves the accuracy and stability of the pose relationship calibration results between the robotic arm's base coordinate system and the target display surface, better meeting the requirements of automated high-precision touch control for in-vehicle display devices. This approach addresses the actual needs of control testing. Simultaneously, it retrieves the target pose relationship from a sample vehicle of the same model as a reuse benchmark. This target pose relationship is then applied to the target test vehicle, and the reuse verification accuracy index is calculated. Judgments are made based on the accuracy index and preset conditions. If the index is qualified, the existing calibration results are directly reused. If the index does not meet the requirements, a correction process is initiated to obtain a calibration pose relationship suitable for the target test vehicle. This fully utilizes the similarity of installation benchmarks between vehicles of the same model, prioritizing the inheritance of completed calibration results. It avoids repeatedly performing the tedious complete calibration process for each target test vehicle, only performing lightweight corrections when the reuse accuracy is insufficient. This significantly reduces the time and labor costs of batch vehicle calibration, while also considering mass production efficiency and the calibration accuracy required for single-vehicle touch interaction, better adapting to the production line operation scenario of large-scale automated testing of automotive displays.

[0054] The above description is merely an overview of the technical solutions of the embodiments of the present invention. In order to better understand the technical means of the embodiments of the present invention and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0055] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0056] Figure 1A flowchart of a first embodiment of the pose relationship calibration method provided by the present invention is shown;

[0057] Figure 2 A flowchart of a second embodiment of the pose relationship calibration method provided by the present invention is shown;

[0058] Figure 3 A schematic diagram of a coordinate system transformation link provided by the present invention is shown;

[0059] Figure 4 A schematic diagram illustrating the geometric type provided by the present invention is shown;

[0060] Figure 5 A schematic diagram of the multi-level screening process provided by the present invention is shown;

[0061] Figure 6 A flowchart of a third embodiment of the pose relationship calibration method provided by the present invention is shown;

[0062] Figure 7 A flowchart of a fourth embodiment of the pose relationship calibration method provided by the present invention is shown;

[0063] Figure 8 A schematic diagram of an application scenario provided by the present invention is shown;

[0064] Figure 9 A schematic diagram of a scene for manual correction provided by the present invention is shown;

[0065] Figure 10 A schematic diagram of an application scenario provided by the present invention is shown;

[0066] Figure 11 A schematic diagram of a process provided by the present invention is shown;

[0067] Figure 12 A schematic diagram of an embodiment of the pose relationship calibration device provided by the present invention is shown;

[0068] Figure 13 A schematic diagram of an embodiment of the pose relationship calibration device provided by the present invention is shown;

[0069] Figure 14 A schematic diagram of an embodiment of the electronic device provided by the present invention is shown. Detailed Implementation

[0070] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. Although exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein.

[0071] In related technologies, in order to enable the robotic arm to perform interactive operations such as clicking and sliding on the vehicle display surface, the pose relationship between the display surface and the robotic arm's base coordinate system is usually determined first through manual teaching, single visual calculation, or fixed parameter migration. The basic principle is to collect images of the display surface or manual corresponding point information, and then calculate the spatial transformation relationship between the robotic arm and the display surface based on this.

[0072] However, in the actual in-vehicle environment, display surfaces are often affected by factors such as reflections, shadows, partial occlusion, and vehicle assembly deviations. Data obtained from a single observation is prone to problems such as unstable feature recognition, pose calculation drift, or amplified local errors. Manual teaching suffers from low operational efficiency, poor repeatability, and strong reliance on human experience, while fixed parameter transfer is difficult to adapt to individual differences between vehicles of the same model. All of these problems lead to low accuracy and stability of the calibrated pose relationships, resulting in deviations in the robotic arm's touch position, inconsistent test results, and frequent recalibrations. This not only reduces the efficiency of automated testing but also increases production line deployment and subsequent maintenance costs.

[0073] To address this, the inventors proposed first generating multiple sets of candidate pose relationship solutions, then eliminating outlier solutions with significant deviations through a screening process to obtain reliable candidate solutions, and finally implementing optimization solutions based on these reliable candidate solutions. This approach avoids the random deviations caused by isolated single calculations and effectively improves the accuracy and stability of the pose relationship calibration results between the robotic arm and the display surface.

[0074] The execution subject of this invention embodiment can be an electronic device with processing capabilities, such as a host computer, industrial control computer, etc., and this invention embodiment is not limited thereto.

[0075] Figure 1 A flowchart illustrating a first embodiment of the pose relationship calibration method provided by the present invention is shown. Figure 1 As shown, the method includes the following steps:

[0076] Step 110: For the sample vehicle, obtain the observation sample information of the target display surface of the sample vehicle observed by the robotic arm in various standard observation poses.

[0077] For example, the sample vehicle is a vehicle used for calibration. The target display surface is the display interface area on the in-vehicle device where the robotic arm performs touch interaction, such as the screen surface of a device that can provide screen interaction operation inside the vehicle, such as a central control screen, vehicle infotainment system, instrument panel, or passenger-side screen. The standard observation pose is a pre-set observation position of the robotic arm, used to ensure that the visual acquisition device installed at the end of the robotic arm or fixedly associated with the robotic arm covers the effective observation area of ​​the target display surface from multiple angles. The observation sample information is a set of data collected and output by the robotic arm in the corresponding standard observation pose. This set of data includes at least image information of the target display surface, pose information of the robotic arm end, and observation features extracted from the image. It is understood that each standard observation pose needs to form a difference in viewing angle to minimize local occlusion of the target display surface. It should be noted that the embodiments of the present invention do not limit the resolution or storage format of the visual images.

[0078] In one example, after the sample vehicle is parked in place, a fixed state is established between the vehicle and the workstation. The robotic arm base is fixed to the test platform, and the target display surface is lit or identifiable. A feature pattern for calibration is displayed on the display surface as the identification object. This feature pattern can be, for example, a checkerboard pattern; coded patterns such as ArUco, AprilTag, and ChArUco; preset UI controls, crosshair cursors, known layout points; or stable features such as screen borders, screen corners, edge contours, interface icons, infrared dots, laser dots, luminous dot matrices, or other auxiliary markers as the identification object. It should be noted that this embodiment of the invention does not limit the content used as the identification object, as long as stable observation features can be extracted and a positional relationship established with the display surface.

[0079] Then, the electronic device retrieves a pre-configured set of standard observation poses. Each standard observation pose corresponds to the spatial position and attitude parameters of the robotic arm's end effector. The parameters can be in the form of three-dimensional translation and attitude angles, or a homogeneous transformation matrix. The robotic arm is controlled to move sequentially according to the standard observation poses. After each standard observation pose is stabilized and stopped, the vision acquisition device is triggered to acquire at least one image frame of the target display surface. At the same time, the robotic arm's end effector pose information relative to the robotic arm's base coordinate system, the current joint state of the robot, the acquisition time, exposure parameters, gain parameters, triggering method, and other acquisition metadata are read.

[0080] Furthermore, the image frame undergoes preprocessing steps such as image enhancement, contrast normalization, distortion correction, denoising, edge enhancement, thresholding, morphological processing, local magnification, or subpixel optimization. Then, feature extraction is performed to extract observational features such as feature points, boundaries, corners, coded points, or region features from the image frame, along with their pixel coordinates. This establishes a positional correspondence between the pixel coordinates of the observational features in the image frame and the three-dimensional coordinates of the target display surface, resulting in a visual feature set. The visual feature set includes at least pixel coordinates, feature type, quality index, and optional reference model coordinates. By combining the image data with timestamp association, the image's visual feature set, and the acquired metadata, the observation sample information corresponding to the standard observation pose is obtained.

[0081] For example, the information of the i-th observation sample It can be represented as Furthermore, the set of observation sample information It can be represented as ;in, Indicates the first The original images acquired from a standard observation pose; Indicates the first The rigid body transformation matrix from the robot arm base coordinate system to the end-effector coordinate system at the frame acquisition moment; Indicates the first Frame camera exposure parameters; Indicates the first Frame camera gain parameters; Indicates the first Frame image acquisition timestamp; Indicates the first A set of visual features extracted from a frame image; This represents the observation dataset consisting of all observed samples; This indicates the total number of standard observation poses.

[0082] Optionally, multiple images with different exposures, gains, or display states can be acquired at each standard observation pose to improve robustness in reflective and low-contrast scenes.

[0083] In some possible implementations, a standard observation pose set (PoseSet) is first established for the target vehicle model, platform, or display module. This pose set is used to constrain the robotic arm to complete data acquisition under relatively stable, observable, and discriminative viewpoints. The standard observation pose set can be associated with and saved with the vehicle model, platform, module, workstation, or camera installation configuration as part of the prior knowledge for subsequent reuse. The source of the standard observation pose set can include any of the following: (a) simulation generation based on CAD models, assembly models, and the robotic arm's workspace; (b) generation based on the results of the first vehicle teaching or the first vehicle automatic calibration; (c) generation based on statistical analysis of historical vehicle calibration data; (d) generation based on the fusion of multiple sources mentioned above.

[0084] Optionally, when generating the standard observation pose set, one or more of the following constraints may be considered: the proportion of the target display surface in the camera's field of view, the angle between the observation optical axis and the surface normal, the reachability of the robotic arm, joint collision constraints, vehicle interior occlusion constraints, reflection risk, observation area coverage, and redundancy observation requirements.

[0085] For example, multiple candidate observation poses are acquired; for each candidate observation pose, visibility indicators, coverage contribution indicators, and risk indicators are determined; based on the visibility indicators, coverage contribution indicators, and risk indicators, a comprehensive score for the candidate observation pose is determined; and candidate observation poses with a comprehensive score greater than a preset threshold are used as standard observation poses.

[0086] For example, the candidate observation pose is a series of alternative poses within the movement space of the robotic arm that can drive the vision acquisition device toward the target display surface. It can be generated by a preset spatial sampling grid of the robotic arm around the target display surface, or it can be calculated by combining historical calibration poses, vehicle cabin layout parameters and camera mounting poses.

[0087] Visibility metrics are used to characterize the degree of occlusion of the target display surface and the ability to fully image the target area under the current posture. They can be determined based on the effective imaging area of ​​the target display surface within the camera's imaging range, edge integrity, the number of detectable feature points, and the proportion of the occluded area.

[0088] The coverage contribution index is used to characterize the diversity of observation perspectives that the pose can supplement. It measures the gain of the pose in enriching spatial observation constraints and improving the feature spatial distribution. It can be determined based on whether the display area covered by the candidate observation pose is complementary to other candidate observation poses and whether it can fill the observation blind spots of edge regions or local low-texture regions.

[0089] Risk indicators characterize the risk of collision, motion singularity, and observation feature failure for the robotic arm in the current posture. These indicators are determined based on the minimum safe distance between the camera's line of sight and interior trim, steering wheel, A-pillar, or the robotic arm itself, as well as the proportion of reflective high-brightness areas, posture stability margin, and the degree of proximity to joint limits. The comprehensive score is an evaluation result obtained by merging the quantitative values ​​of these three types of indicators. The standard observation pose is a robotic arm posture that has been screened and meets the overall performance requirements, and is officially used to collect observation sample information. It is understandable that the selected standard observation poses have different perspectives, allowing observation of the target display surface from multiple directions.

[0090] For example, the electronic device first generates multiple candidate observation poses, and performs visibility, coverage contribution, and risk assessments on each candidate observation pose to obtain visibility, coverage contribution, and risk indicators. Then, the visibility, coverage contribution, and risk indicators are weighted and summed according to preset weight coefficients to obtain a comprehensive score. ,in, Indicates the first The overall score of each candidate observation pose is a comprehensive score, and the higher the score, the better the acquisition effect of that pose. This represents the visibility index weighting coefficient, a preset constant, which adjusts the degree of influence of visibility on the total score. Indicates the first The visibility index corresponding to each candidate observation pose characterizes the degree to which the target display surface is fully visible within the camera's field of view; This represents the coverage contribution weighting coefficient, a preset constant, which adjusts the impact of the observation area coverage rate on the total score. Indicates the first The coverage contribution index of each candidate observation pose represents the supplementary gain of the viewpoint to the area not covered by the screen. This represents the risk penalty weighting coefficient, a preset constant, which adjusts the penalty level for various risks; : No. The risk indicators for each candidate observation pose include three categories of negative risk quantification values: robotic arm collision, vehicle interior obstruction, and screen glare.

[0091] Finally, candidate observation poses with a comprehensive score higher than a preset threshold are selected as standard observation poses. This method ensures that subsequent observation samples are concentrated on observation poses with high visibility, more comprehensive coverage, and lower safety risks, thus providing more stable input for pose calculation.

[0092] It should be noted that the number of standard observation poses can be adjusted based on the target screen size, camera angle, accuracy requirements, and vehicle interior space; this embodiment of the invention does not limit this.

[0093] By adopting the above implementation method, the selection process of standard observation poses takes into account the observation effectiveness, regional coverage integrity and environmental risk constraints, making the selected poses more suitable for the automated observation and calibration of vehicle display surfaces, thereby reducing the proportion of invalid and unstable observations and improving the consistency and reliability of subsequent pose relationship solutions.

[0094] Step 120: Perform pose calculation based on the observed sample information to obtain multiple sets of candidate pose relationship solutions.

[0095] For example, pose calculation refers to the computational process of solving the rigid body space transformation between different coordinate systems based on the positional correspondence of observed features within the observed sample information. Candidate pose relationship solutions are multiple sets of pose transformation results that satisfy basic geometric constraints, and each set of solutions can serve as an alternative to the final calibration result. The pose relationship solution characterizes the pose transformation relationship between the target display surface coordinate system and the robot arm base coordinate system. The pose transformation relationship between the target display surface coordinate system and the robot arm base coordinate system represents the transformation matrix that can transform spatial points in the target display surface coordinate system to those in the robot arm base coordinate system. Optionally, the candidate pose relationship solutions can be combined into a candidate relationship solution set denoted as CandidateSet, which includes at least one or more candidate pose relationships and corresponding error evaluation information.

[0096] In one example, the electronic device can perform spatial pose calculation based on a single set of observation samples or a combination of multiple sets of observation samples to obtain candidate pose relationship solutions between the target display surface coordinate system and the robot arm base coordinate system. For example, the positional correspondence between the pixel coordinates of the observed features in the observation sample information and the three-dimensional coordinates of the target display surface can be used, along with the geometric constraints constructed by pre-calibrating the hand-eye relationship within the preset camera, to solve for the initial transformation matrix and iteratively optimize and generate a set of candidate pose relationship solutions. By repeating the calculation process with multiple sets of observation samples, multiple sets of candidate pose relationship solutions are finally output.

[0097] In one possible implementation, to obtain multiple sets of candidate pose relation solutions, the electronic device does not solve for each standard observation pose individually, but can also perform batch calculations using a multi-sample combination approach. For example, two, three, or more sets of samples can be selected from all valid observation samples according to a predetermined combination size for joint calculation, and a set of candidate pose relation solutions can be calculated for each combination. The resulting multiple sets of candidate pose relation solutions correspond to different observation conditions or different sample combinations, which can explicitly disperse unstable factors between samples into the candidate solution set, rather than directly fixing them into a single result. This step generates multiple spatial transformation results to be compared from multi-view observation information, so that local errors caused by factors such as reflection, shadow, and assembly deviation will not directly dominate the final calibration output, thus providing a basis for subsequent screening.

[0098] Step 130: Filter the candidate pose relation solutions to obtain reliable candidate solutions.

[0099] For example, different candidate pose relation solutions differ in error level, physical consistency, and cross-sample stability. The screening process involves retaining or eliminating candidate pose relation solutions according to preset judgment criteria. A reliable candidate solution is the set of candidate pose relation solutions retained after screening, which can more stably represent the actual pose transformation relationship between the target display surface coordinate system and the robot arm base coordinate system.

[0100] In one example, the electronic device filters candidate pose relation solutions according to a preset screening criterion, retaining candidate pose relation solutions with higher confidence as reliable candidate solutions.

[0101] Compared to existing technologies that directly generate calibration relationships based on single-frame or single-time calculation results, which do not fully utilize redundant information from multi-view observations and result in the inability to effectively eliminate abnormal solutions, this step filters candidate solutions, making the data entering the final optimization stage more concentrated near the actual pose, thereby controlling the dispersion of initial values ​​and convergence drift in subsequent optimization.

[0102] Step 140: Optimize the credible candidate solutions to obtain the target pose relationship of the sample vehicles.

[0103] For example, the optimization solution is a process of further refining the pose parameters of the target display surface coordinate system relative to the robot arm base coordinate system, based on the reliable candidate solution. The target pose relationship is the optimal pose transformation result obtained after iterative convergence, which can be directly used for robot arm touch positioning and can be used as the spatial positioning basis when the robot arm performs click, swipe, long press and trajectory interaction operations.

[0104] In one example, the electronic device uses a single candidate solution or a combination of multiple reliable candidate solutions as optimization input to establish an optimization model for relevant observation samples. It iteratively adjusts the pose parameters of the target display surface coordinate system relative to the robot arm base coordinate system until a preset termination condition is met, thus obtaining the target pose relationship of the sample vehicle. For example, an optimization objective function can be constructed, using reliable candidate solutions as initial values ​​for iterative optimization, and a nonlinear optimization algorithm can be used to iteratively solve the problem to obtain the target pose relationship.

[0105] This step, under the premise that the credible candidate solution has been constrained to the neighborhood of the true pose, further optimizes to obtain the final pose relationship solution corresponding to the sample vehicle, so that when the robotic arm performs touch actions in the subsequent process, it can accurately convert the target point in the display interface into the operation point in the robotic arm's base coordinate system, thereby reducing repeated calibration and touch offset.

[0106] In this embodiment, observation sample information of the robotic arm under multiple standard observation poses is first collected for the sample vehicle. Multiple sets of candidate pose relationship solutions are calculated based on this information. Then, outlier solutions are eliminated through a screening process to obtain reliable candidate solutions. Finally, optimization is performed based on these reliable candidate solutions to obtain the target pose relationship corresponding to the sample vehicle. This method no longer relies on a single solution obtained from a single calculation as the calibration result. By generating multiple sets of candidate solutions, the range of options is enriched. A screening process is used to remove outlier solutions with large deviations, reducing interference from poor data. Global optimization is then performed based on the purified reliable candidate solutions. This effectively alleviates the calibration deviation problem caused by single observation noise and poor feature quality, continuously improving the accuracy and stability of the pose relationship calibration results between the robotic arm's base coordinate system and the target display surface, better meeting the actual needs of automated high-precision touch testing of automotive display devices.

[0107] Figure 2 A flowchart illustrating a second embodiment of the pose relationship calibration method provided by the present invention is shown. Figure 2 As shown, the method includes the following steps:

[0108] Step 210: For the sample vehicle, obtain the observation sample information of the target display surface of the sample vehicle observed by the robotic arm in various standard observation poses.

[0109] It should be noted that this step is similar to step 110 mentioned above, and will not be repeated here.

[0110] Step 220: Based on the observation characteristics, evaluate the quality of the corresponding observation sample information to obtain the quality score of the observation sample information.

[0111] For example, as mentioned above, the observation sample information includes observation features, which are key visual information extracted from the observation image of the target display surface that can be used for pose solving. Sample quality assessment is a process of comprehensively quantifying and scoring the imaging quality, feature effectiveness, and observation reliability of a single set of observation samples based on multiple quantitative indicators of the observation features. The quality score is a quantitative value characterizing the quality of information in a single set of observation samples; a higher score indicates better imaging effect and feature quality of the set of observation samples, and a stronger ability to adapt to pose solving.

[0112] In one example, an electronic device can score each observation sample based on the number of features, the range of feature distribution, the confidence level of feature detection, and the image sharpness, thus forming a quality score.

[0113] Specifically, based on the observed characteristics, quality assessment indicators for the observed sample information corresponding to the observed characteristics are determined; among them, the quality assessment indicators include at least one of the following: feature quantity, feature coverage uniformity, feature detection confidence, image clarity, image sharpness, boundary closure, corner stability, exposure rationality, and local occlusion ratio; based on the quality assessment indicators for the observed sample information, the quality score of the observed sample information is determined.

[0114] For example, the observation features are effective feature information such as visual keypoints, edge contours, and texture features extracted from a single frame of observation image, which can be used for pose matching and coordinate calculation. The quality assessment index is a quantitative evaluation dimension used to comprehensively quantify the imaging quality and feature effectiveness of a single set of observation samples; among them, the number of features represents the total number of effective features that can be extracted from the current image; feature coverage uniformity represents the spatial distribution uniformity of features in the target display surface area; feature detection confidence represents the reliable probability of the model extracting features; image clarity and image sharpness represent the image's detail resolution capability and edge sharpness; boundary closure represents the complete closure of the screen display area's contour edges; corner point stability represents the positional stability of feature corner points within neighboring frames or neighboring windows; exposure rationality represents whether the image exposure brightness is within the normal imaging range and whether there are overexposure or underexposure problems; and local occlusion ratio represents the proportion of the target display surface that is occluded by external objects and cannot extract effective features. The quality score is a comprehensive quantitative score that can comprehensively represent the quality of a single set of observation samples after fusing and quantifying one or more of the above quality assessment indices.

[0115] For example, for each set of observed sample information, the electronic device extracts the corresponding multi-dimensional indicators such as the number of features, uniformity of feature distribution, detection confidence, image clarity, sharpness, boundary integrity, corner stability, exposure parameters, and occlusion ratio. It then normalizes each dimension of the indicators to eliminate dimensional differences and uses a preset weighted fusion algorithm to perform weighted summation calculation on all effective evaluation indicators. Finally, it outputs a unique comprehensive quality score corresponding to the set of observed sample information, thus completing the quantitative evaluation of single sample quality.

[0116] In practical implementation, regarding the number of features, the number of feature points in the current sample that meet the confidence threshold can be counted and mapped to the corresponding score, i.e. ,in, This represents the total number of valid features detected in the i-th frame image; This represents the element counting operation of a set, which calculates the total number of feature points within the set.

[0117] For feature coverage uniformity, it can be determined according to the formula. Calculate the feature coverage uniformity index, where, Indicates the first The frame image feature coverage uniformity index, with a value of 0 to 1. The closer to 1, the more balanced the feature distribution. Indicates the first All feature points of the frame The convex hull region formed; Indicates the first The first frame Pixel coordinates of image feature points; This represents the operation of calculating the area of ​​a region. This represents the total area of ​​pixels in the entire image.

[0118] For feature detection confidence, the mean or median confidence value output by the feature point detector can be used as the scoring basis.

[0119] For image sharpness, you can use the formula... Calculate the metrics for image sharpness; among which, Indicates the first A quantification of frame image sharpness; the higher the value, the higher the image sharpness. Indicates the image Perform Laplacian edge extraction operation; This represents the variance calculation operation, which statistically measures the dispersion of all pixel values ​​in the entire image after the Laplacian operation. It can be understood that the clearer the image, the more edges and fine textures there are, the greater the difference in pixel values ​​output by the Laplacian operator, the larger the variance, and the higher the clarity. The more blurred the image (out of focus, motion blur), the smoother the grayscale changes, the more the Laplacian output approaches 0, the smaller the difference in pixel values, the smaller the variance, and the lower the clarity.

[0120] For image sharpness, the score can be calculated based on Laplacian variance, gradient magnitude, or high-frequency energy in the frequency domain; for boundary closure, the score can be determined based on the ratio of the break length of the target boundary to the total boundary length; for corner stability, the positional variance of the same corner point in repeated observations across multiple frames can be converted into a score; for exposure rationality, the score can be calculated based on the high-brightness saturation ratio and low-brightness underexposure ratio of the grayscale histogram; for local occlusion ratio, the deduction can be obtained by the ratio of the occluded area to the effective observation area. These indicators can be used individually for scoring, or they can be weighted and aggregated according to preset weights to obtain a quality score for the observed sample information.

[0121] For example, mass fraction can be expressed as: ;in, Indicates the first The quality score of each observation sample is used to filter low-quality images. Indicates the feature quantity weight coefficient; Represents the normalization function, which will Mapped to a unified 0~1 range; Indicates the feature coverage uniformity weighting coefficient; This represents the image sharpness weighting coefficient; This represents the feature detection confidence weight coefficient; Indicates the first Average detection confidence of all features in the frame.

[0122] Understandably, the quality score is used to characterize whether the observed sample information is suitable for subsequent pose calculation. Samples with higher quality scores typically have more identifiable and spatially consistent features, while samples with lower quality scores indicate blurring, occlusion, abnormal exposure, or uneven feature distribution in the image. By mapping observed features to quality assessment indicators and further forming quality scores, subsequent screening processes can directly eliminate low-quality samples based on the quantification results, or reduce the weight of low-quality samples. This implementation method allows the observed sample information entering pose calculation to have more stable feature representations, thereby improving the reliability of candidate pose relation solutions and reducing calculation deviations caused by single-frame observation anomalies, thus improving the robustness and accuracy of the overall calibration results from the source.

[0123] Step 230: Based on the quality score, the observation sample information is filtered to obtain valid observation sample information.

[0124] For example, the screening process is a data filtering process that identifies and discards all observation sample information based on a preset quality threshold, removing inferior samples and retaining high-quality samples. Valid observation sample information refers to qualified observation sample data that meets the preset threshold requirements in terms of quality score, has reliable feature quality, and can be used for subsequent pose calculation.

[0125] In one example, the electronic device compares the quality score of each group of observation sample information with a preset quality threshold one by one, eliminates inferior observation samples with quality scores below the threshold, poor imaging quality, or feature failure, and filters and retains observation sample information that meets the quality standard to form valid observation sample information.

[0126] By adopting this method, samples with low resolution, strong occlusion, or unbalanced feature distribution will not enter the solution process, the dispersion of candidate solutions will be reduced, and the stability and consistency of pose relationship calibration results will be improved.

[0127] Step 240: Perform pose calculation based on the effective observation sample information to obtain multiple sets of candidate pose relationship solutions.

[0128] In one example, the electronic device solves the coordinate transformation by combining the positional correspondence between the two-dimensional image features represented by each set of valid observation sample information and the three-dimensional display surface with the pre-calibrated hand-eye relationship within the preset camera, and obtains a single set of candidate pose relationship solutions through local iterative optimization. After traversing all valid observation samples, multiple sets of candidate pose relationship solutions are obtained.

[0129] Specifically, for each valid observation sample, based on the pixel coordinates of the observed features, the positional correspondence, and the preset camera intrinsic parameters, the first pose transformation relationship between the camera coordinate system and the target display surface coordinate system is solved; based on the pre-calibrated hand-eye relationship, the first pose transformation relationship, and the end-effector pose during observation, the second pose transformation relationship between the target display surface and the robot arm base coordinate system is determined; based on the positional correspondence, a target error function is constructed, and the second pose transformation relationship is used as the initial value for iteration. The optimization objective is to minimize the target error function, resulting in a set of candidate pose relationship solutions.

[0130] As mentioned above, the effective observation sample information includes the end-effector pose during observation; the observation features include the positional correspondence between the pixel coordinates of the observation features and the three-dimensional coordinates of the target display surface. Specifically, the pixel coordinates of the observation features are the two-dimensional coordinates of the features extracted from the visual image in the image plane; the positional correspondence is a one-to-one matching mapping between the three-dimensional feature points on the target display surface and the two-dimensional observation features of the image.

[0131] Camera Insider The pre-calibrated inherent imaging parameters of the camera, including focal length, principal point coordinates, and distortion coefficients, can be expressed as: ;in, , Indicates the focal length in pixels; , Indicates the coordinates of the principal point.

[0132] The first pose transformation relationship characterizes the rigid body transformation matrix of the camera coordinate system relative to the target display surface coordinate system, enabling the transformation of 3D points on the display surface to the camera coordinate system. The hand-eye relationship is a fixed transformation relationship between the pre-calibrated camera coordinate system and the robotic arm end-effector coordinate system. The second pose transformation relationship is the initial transformation result of the target display surface coordinate system relative to the robotic arm base coordinate system, which is directly derived. The target error function is established based on feature matching constraints and is used to measure the geometric deviation caused by the projection of 3D points onto the image plane under the current pose. The iterative optimization process is a nonlinear optimization process that uses the second pose transformation relationship as the initial value and continuously adjusts the transformation parameters to make the error function value continuously decrease. A set of candidate pose relationship solutions are the candidate calibration pose results that satisfy the geometric constraints and are obtained after iterative convergence.

[0133] The electronic device reads the two-dimensional pixel coordinates of features within a single set of valid observation samples, the three-dimensional coordinates of the matched display surface, and the camera intrinsic parameters. It then uses algorithms such as the Perspective-n-Point (PnP) algorithm, planar pose estimation, homography matrix decomposition, and nonlinear least squares pose solving algorithms to obtain the first pose transformation relationship of the camera relative to the target display surface. By combining the stored hand-eye transformation matrix and the robotic arm end-effector pose recorded at the time of sample acquisition, the second pose transformation relationship between the target display surface and the base is derived through coordinate transformation chain operations. Based on the feature three-dimensional coordinates and two-dimensional pixel coordinates, a target error function in the form of reprojection error is constructed. The second pose transformation relationship is set as the initial value of the iteration. The pose parameters are continuously adjusted until convergence using a gradient-type nonlinear optimization algorithm, and finally a set of candidate pose relationship solutions corresponding to the effective observation samples of this group are generated.

[0134] This step relies on the matching constraints between 2D image features and 3D features of the display surface, combined with the camera imaging model to solve the pose between the camera and the screen, and then uses the hand-eye relationship and the pose of the robotic arm end effector to complete the coordinate link transformation to obtain the initial pose. Further, the initial pose is refined through iterative optimization to obtain candidate solutions. The complete geometric constraint link ensures that the initial solution has reasonable physical meaning. At the same time, the optimization process reduces the deviation caused by imaging distortion and feature detection perturbation, providing reliable alternative pose results for subsequent screening of multiple candidate solutions and global optimization.

[0135] Specifically, based on the type of observed features and the geometric type of the target display surface, the type of the target error function and the target optimization strategy are determined; the target error function is constructed based on the type of the target error function and the positional correspondence; according to the target optimization strategy, the second pose transformation relationship is used as the initial value for iteration, and the optimization objective is to minimize the target error function to perform iterative optimization processing, thereby obtaining a set of candidate pose relationship solutions.

[0136] The observed features include various forms of visual features such as discrete feature points, contour edges, and region masks. The geometric types of the target display surface include planar surfaces, regular surfaces, and freeform surfaces. The target error function is an error metric based on the adaptation of features and geometric shapes, and can be at least one of reprojection error, contour alignment error, or point-to-surface distance error. The target optimization strategy includes a complete set of optimization execution rules, such as optimization iteration termination conditions, iteration step size control, abnormal error suppression methods, and solver selection. For example, when the observed features are boundaries, contours, line segments, or curves, geometric fitting, contour alignment, minimum distance optimization, template matching, learning-based localization networks, or combinations thereof can be used to solve for candidate pose relationships. When the observed features are boundaries or contours, the aforementioned target error function can be contour alignment error, boundary distance error, region matching error, or a combination thereof.

[0137] For example, after receiving valid observation sample information, the electronic device first matches the error modeling method according to the observation feature type and the display surface geometry. When the observation feature is a discrete point and the target display surface is planar, the target error function is constructed as the sum of squares of the reprojection residuals of each corresponding point after pose transformation. When the target display surface is a curved surface or a boundary structure, the target error function is constructed as the residual of the normal distance from the corresponding point to the surface or the residual of the shortest distance from the point to the boundary curve. The target optimization strategy is determined simultaneously with the error function type, and abnormal residuals are given lower weights to reduce the impact of local occlusion or reflective points on the results.

[0138] For example, if an electronic device identifies the observed features extracted from the current valid observation sample as discrete feature points and determines that the target display surface is of planar geometry, then the sum of squared reprojection residuals is selected as the target error function type. A Gauss-Newton optimization strategy with robust loss is used in conjunction with this. The target error function in the form of the sum of squared reprojection residuals is constructed based on the positional correspondence between the three-dimensional feature points and the two-dimensional pixel coordinates. The second pose transformation relationship is set as the initial value for iteration. Iterative solutions are carried out according to the preset number of iterations, error convergence threshold, and large error suppression rules. After the optimization process converges, a set of candidate pose relationship solutions is output.

[0139] For example, Figure 3 A schematic diagram of a coordinate system transformation link provided by the present invention is shown, as follows: Figure 3 As shown, four coordinate systems are defined: Base (robotic arm base), Tool (robotic arm end effector), Camera (camera), and Surface (display surface). The camera is fixedly mounted on the end effector, and the hand-eye relationship between the end effector and the camera is defined. These are pre-calibrated fixed parameters. As mentioned earlier, the observation sample information includes the rigid body transformation from the base to the end effector under the standard observation pose. When the target interactive surface is a plane, the coordinates on the target display surface are... The A surface reference point, the pose of the base to be solved from the display surface. The coordinates of the reference point can be transformed from the display surface coordinate system to the base coordinate system, and then... The inverse transformation converts the coordinates to the coordinate system of the robotic arm's end effector. Multiplying by the hand-eye transformation matrix yields the predicted 3D coordinates of the reference point in the camera coordinate system. If hand-eye transformation is defined as the form of the camera pointing to the end, then simply change the formula... The coordinate link logic can be maintained by replacing it with its inverse transformation.

[0140] Furthermore, after obtaining the 3D predicted coordinates of the reference point in the camera coordinate system, these 3D coordinates are first compared with the camera intrinsic parameter matrix. Multiply, then pass through the perspective projection function Complete the projection from 3D space to a 2D image plane, and output the predicted pixel coordinates corresponding to the reference point. The actual pixel observation of the j-th feature point under the i-th standard observation pose is The predicted pixel coordinates and the actual pixel observations detected in the image are then compared. By taking the difference and calculating the L2 norm, we can obtain the projection geometric error of a single reference point. Therefore, the objective error function can be expressed as: ;in, This represents the reprojection geometric error of the j-th reference feature point in the i-th frame when the pose transformation is T; Indicates based on the first Candidate rigid body transformations obtained from frame sample solutions The inverse is obtained .

[0141] Optionally, to improve robustness, a candidate pose relation solution can be generated based on a single observation sample, or multiple candidate pose relation solutions can be generated by combining information from multiple valid observation samples. In this case, the target error function can be expressed as: ; This represents the effective combination of observation samples used to construct the k-th candidate pose relation solution.

[0142] It should be noted that Figure 4 A schematic diagram illustrating the geometric type provided by the present invention is shown, such as... Figure 4 As shown, when the geometry of the target surface is planar, the reference point can be... Defined in planar coordinates; when the geometry of the target display surface is a piecewise plane or a surface, It can be defined on the corresponding 3D surface model. For example, it can use a piecewise planar model, parametric surface model, combined surface model, discrete mesh model, spline model, or other predefined surface model to express its spatial reference points, and explicitly introduce surface model constraints during the solution process. It is understandable that when the target display surface is a piecewise plane, dual screen, curved screen, or irregularly shaped screen, the feature points no longer share a single plane equation. It is necessary to pre-construct a complete 3D surface model and add the geometric rules of the 3D surface to the optimization conditions of the pose solution, restricting the solved pose from violating the geometry of the screen itself.

[0143] This step no longer uses a uniform and fixed error function and optimization method. Instead, it can adaptively match the corresponding error metric and optimization execution strategy by combining the observed feature form and the display surface geometry. This makes the error constraint more in line with the geometric characteristics of the current sample, improves the rationality of the error metric, and then conducts iterative optimization to obtain candidate pose solutions. This can effectively reduce the solution deviation caused by the mismatch between feature type and screen geometry, further improve the credibility of a single set of candidate pose relationship solutions, and provide higher quality alternative results for subsequent candidate solution set selection and global optimization.

[0144] Step 250: Based on the preset filtering dimensions, filter the candidate pose relation solutions to obtain reliable candidate solutions.

[0145] The selection dimensions include at least one of the following:

[0146] Quality assessment index of the observation features corresponding to candidate pose relation solutions;

[0147] The average reprojection error of the observation features corresponding to the candidate pose relation solution;

[0148] Consensus score of candidate pose relation solutions;

[0149] Physical constraints;

[0150] Global statistical model.

[0151] For example, the selection dimension is an evaluation criterion for judging whether a single group of candidate pose relation solutions deviates from the true value from multiple perspectives. The quality assessment index of the observation features is used to trace the basic quality of the original samples that generated the candidate solution, and to eliminate observation samples with insufficient feature quantity, overly concentrated coverage, insufficient sharpness, or low confidence.

[0152] The average reprojection error characterizes the overall deviation of the 3D features corresponding to the candidate pose relationship projected onto the image plane, and is used to eliminate candidate solutions whose reprojection error exceeds a threshold. For example, the average reprojection error of the i-th candidate solution can be defined as follows: .in, Indicates the first The global average reprojection error corresponding to each candidate solution; Indicates the first The total number of valid features involved in the calculation of each candidate solution; Indicates the first One candidate rigid body transformation; Indicates the th transformation Projection error of each feature point.

[0153] The consensus score measures the consistency level between the current candidate solution and the majority of solutions by comparing the similarity between each group of candidate solutions. It is used to eliminate abnormal candidate solutions that are significantly inconsistent with the majority of candidate solutions. The steps for determining the consensus score include: for each candidate pose relation solution, determining the translational and rotational angle deviations between the candidate pose relation solution and other candidate pose relation solutions; determining the consensus score of the candidate pose relation solution based on the number of other candidate pose relation solutions that satisfy the consistency condition; the consistency condition is that the translational deviation is less than or equal to a translational deviation consistency threshold and the rotational angle deviation is less than or equal to a rotational deviation consistency threshold.

[0154] The candidate pose relation solution is a set of rigid body transformation matrices representing the target display surface coordinate system relative to the robot arm base coordinate system. Any set of rigid body transformations can be decomposed into translation and rotation components. Translational deviation is the spatial distance between the translation vectors of two sets of candidate pose solutions, used to measure the difference in their spatial positions. Rotational angle deviation is the equivalent rotation angle between the rotational transformations corresponding to the two sets of candidate pose solutions, used to measure the difference in attitude deflection. The consistency condition is used to determine whether two sets of candidate pose solutions are close to each other and belong to the same true value range. The translational deviation consistency threshold and rotational deviation consistency threshold are pre-set or adaptively updated tolerance ranges. Other candidate pose relation solutions that satisfy the consistency condition represent alternative solutions with smaller deviations and consistent trends with the current candidate solution. The consensus score is an indicator that quantifies the level of consensus of the current candidate solution within the entire candidate solution set. It is positively correlated with the number of matching candidate solutions; the higher the consensus score, the more similar the candidate solution is to the majority of solutions, and the lower the probability of it being an anomalous solution. It is understandable that anomalous solutions often deviate significantly from the vast majority of candidate solutions, have fewer candidate solutions that match the consistency condition, and thus have lower consensus scores.

[0155] For example, an electronic device can iterate through each set of candidate pose relation solutions, comparing each solution pairwise with all other candidate pose relation solutions in the set. It calculates the Euclidean distance between the translation vectors of each pairwise solution as the translation deviation, and solves for the equivalent rotation angle between the rotation matrices as the rotation angle deviation. For instance, the translation deviation can be expressed as... The rotation angle deviation can be expressed as ,in, Indicates candidate solutions With candidate solutions The L2 distance of the translation component; Indicates candidate solutions The translation vector; Indicates candidate solutions The translation vector; This represents the L2 norm (Euclidean distance) operation on vectors; Indicates candidate solutions The included angle deviation of the rotating part, in radians; Indicates candidate solutions rotation matrix; Indicates candidate solutions The rotation matrix; Representing the rotation matrix Transpose of; This represents the function for solving the angle between the rotation axes corresponding to the matrix.

[0156] Pairs with both deviations not exceeding their corresponding thresholds are considered to meet the consensus condition. The total number of other candidate solutions that can reach consensus with the current candidate solution is counted, and this number is normalized to obtain the consensus score corresponding to the current candidate pose relation solution. For example, the consensus score can be represented as... ;in, Indicates the first The consensus score of each candidate solution; the higher the value, the stronger the consistency with other candidate solutions. : Count of the number of candidate solutions that satisfy the conditions in parentheses; Indicates the consistency threshold for translation deviation; This represents the rotational deviation consistency threshold.

[0157] This step compares the translation and rotation deviations of all candidate solutions pairwise, and determines the consistency between solutions based on the dual constraints of translation and rotation. It counts the number of matching candidate solutions to generate a consensus score, and uses the group consistency feature to identify isolated outliers that deviate from the cluster. It can complete the reliability assessment of candidate solutions without relying on external truth values, and can effectively detect outlier pose solutions caused by noise interference. This provides a reliable quantitative basis for subsequent candidate solution screening and improves the overall quality of the set of reliable candidate solutions.

[0158] Physical constraints define the reasonable range of spatial transformations formed by the robotic arm's motion space and the screen's mounting boundary. These constraints are used to eliminate candidate solutions that violate spatial geometric constraints, posture constraints, workspace constraints, or surface model constraints. The global statistical model identifies candidate solutions with abnormal deviations based on error distribution patterns, such as through clustering or Random Sample Consensus (RANSAC) algorithms. Figure 1 Consistency analysis, outlier elimination, or other consensus mechanisms are used to select a set of reliable solutions.

[0159] In one example, Figure 5 The flowchart illustrating the multi-level screening process provided by this invention is shown below. Figure 5 As shown, the electronic device sequentially calculates the original sample feature index (feature quality screening), average reprojection error (reprojection consistency screening), and inter-group consensus score (multi-view consistency screening) corresponding to each group of candidate pose relationship solutions. At the same time, it verifies whether the spatial physical constraints are met (physical space constraint screening), identifies outlier solutions with the help of a global statistical model (statistical robustness screening), removes candidate solutions that do not meet the requirements of any one of the indicators, and retains qualified results to form a set of credible candidate solutions.

[0160] In another example, after receiving multiple sets of candidate pose relation solutions, the electronic device first establishes a mapping relationship between each set of candidate solutions and the corresponding observation features, then calculates the quality assessment index and the average reprojection error, and removes candidate solutions with excessively large errors or low quality scores. For the retained candidate solutions, the consistency between them and other candidate solutions in terms of rotation angle, translation amount, and observation consistency is further statistically analyzed to form a consensus score. This score is then combined with the reachable space of the robotic arm end effector, the camera's field of view, and the actual installation posture of the display surface to construct physical constraints. When a candidate solution conflicts with the physical constraints, or its pose parameters significantly deviate from the historical distribution described by the global statistical model, the candidate solution is determined to be an unreliable solution. After comprehensive screening, a set of reliable candidate solutions is output. This processing method can separate local observation anomalies, occasional mismatches, and abnormal pose results from the candidate set, making the solution set entering the optimization stage more stable.

[0161] Step 260: Optimize the reliable candidate solutions according to the preset objective function to obtain the target pose relationship of the sample vehicles.

[0162] The objective function minimizes the sum of geometric errors processed by the robust loss function. Geometric errors characterize the errors in the observed features of the observed samples corresponding to the reliable candidate solutions. Geometric errors are at least one of the following: reprojection error, contour alignment error, boundary distance error, and surface model constraint error. The robust loss function is used to suppress the interference of excessively large errors caused by individual outliers on the overall optimization results, reduce the weight of outliers, and prevent a few poor observation data from dominating the optimization direction. The geometric errors processed by the robust loss function are the loss values ​​calculated by inputting the original geometric errors into the robust loss function. The robust loss function can employ Huber loss, Cauchy loss, or Geman-McClure loss to adapt to the error distribution generated by the automotive display surface under complex lighting conditions.

[0163] In one example, if the observed feature is an image feature point, then based on the in-camera relationship solution involving the current pose, the spatial points on the target display surface are projected onto the image plane, and the reprojection error between the projected position and the actual observed pixel position is calculated. If the observed feature is a surface contour, then the contour alignment error between the projected contour and the segmented contour in the image is calculated. If the observed feature is an edge point, then the shortest distance from the edge point to the target boundary is calculated as the boundary distance error. Therefore, the form of the geometric error can be determined according to the type of observed feature, and the geometric error is first robustly processed through Huber loss, Cauchy loss, or Geman-McClure loss, and then weighted and summed to form a unified optimization objective. For example, the objective function can be expressed as... ;in, This indicates the global optimization output optimal calibration transformation. ; Indicates the first Frame number The observation weights for each feature point are higher for high-quality samples. This represents a robust loss function that suppresses outlier interference. This represents the squared term of the reprojection error of the feature points.

[0164] The optimization process can be implemented using Gauss-Newton's method, Levenberg-Marquardt (LM) method, Trust Region Reflective (TRF) algorithm, Doggleg Algorithm (Dogleg), Robust Least Squares, gradient descent, graph optimization, or other nonlinear optimization methods. During the solution process, the robot arm's end-effector pose, pre-calibrated hand-eye relationship, and camera intrinsic parameters are used as constraint inputs to ensure that the pose update always satisfies the geometric relationship between the robot arm and the display surface. The output is the sample vehicle target pose relationship. The final solution is obtained by joint optimization of multiple sets of reliable candidate solutions and is used for coordinate mapping and motion control in subsequent touch tests.

[0165] With this implementation method, the solution of the target pose relationship no longer depends on a single observation result, but is refined twice on the credible candidate solution. This can reduce the touch position offset, improve the calibration accuracy in complex in-vehicle environments, and enhance the reusability of pose results between different sample vehicles.

[0166] Optionally, weight terms, prior terms, or covariance terms can be introduced during the optimization process to make high-quality observations contribute more and allow historical reuse results to participate in the optimization as initial priors.

[0167] Specifically, when introducing reused priors, the objective function is to minimize the sum of geometric error and bias penalty term after processing by the robust loss function; the bias penalty term represents the degree of spatial offset between the reliable candidate solution and the historical prior pose.

[0168] Among them, the historical prior pose relationship refers to the pose between the robotic arm and the target display surface in the past calibration. The historical prior pose relationship can be stored in the calibration database of the electronic device. The database records include vehicle model, camera assembly status, robotic arm installation reference, and the most recent stable calibration result.

[0169] The bias penalty term quantifies the rigid body deviation of the pose to be optimized relative to the historical prior pose relationship. The objective function aims to minimize the sum of the two types of losses, simultaneously considering the accuracy of visual geometric fitting and the stability of the pose optimization result relative to the benchmark.

[0170] For example, first, a mapping relationship between candidate poses and observed features is established based on the current observed sample information. Then, the geometric residuals corresponding to the candidate poses are calculated, and the residuals are substituted into the robust loss function to form a geometric term. Simultaneously, historical prior pose relationships are extracted, and the incremental rigid body transformation between the two poses is obtained by matrix multiplication with the current pose to be optimized through inverse transformation. Then, the matrix transformations consisting of rotation and translation are transformed into a six-dimensional Lie algebraic perturbation vector using the Lie group logarithmic mapping. The L2 norm squared of the perturbation vector is calculated to quantify the spatial offset between the two, obtaining the bias penalty term. The weighted sum of the two types of terms is used as the objective function, i.e., the objective function can be expressed as: ,in, Indicates historical prior pose relationships; This represents the inverse transform of the prior transformation; This represents the Lie group to Lie algebra logarithmic mapping, which converts rigid body transformations into perturbation vectors; Denotes the squared L2 norm of the Lie algebra perturbation vector; This represents the prior constraint weight, controlling the strength of the constraint on the calibration results by historical priors. By iteratively updating the translation and attitude parameters of candidate solutions, the objective function converges to a local minimum, ultimately outputting the target pose relationship that balances observation consistency and historical continuity.

[0171] By adopting the above objective function, the optimization process is not only constrained by the geometric consistency of the current observation, but also by the continuity of the historical prior pose relationship. This makes it less likely for reliable candidate solutions to deviate from the stable solution domain when there are noisy observations, thereby improving the stability of the calibration results and the ability to reuse across batches, and reducing pose drift caused by a single abnormal observation.

[0172] Optionally, after global optimization is completed, the system can output target pose relationships, error statistics, pass rate statistics, rejection reason records, and verification reports.

[0173] Optionally, after the calibration test of the sample vehicles is completed, a reusable calibration asset set can be generated and stored in the calibration asset library. This calibration asset set includes one or more of the following: a standard observation pose set; target pose relationships; optimization initial values ​​or historical priors; quality screening rules and thresholds; a validation observation pose set; definitions of validation points, validation controls, or validation regions; expected interactive response rules; an error statistical model; and local correction parameters.

[0174] In this embodiment, observation sample information is first collected under multiple standard observation poses. Based on the observation features, the quality of each group of observation samples is assessed and a quality score is generated. Valid observation sample information is obtained based on the quality scores. Multiple candidate pose relationship solutions are calculated using the valid observation samples. Then, abnormal candidate solutions are eliminated through multi-dimensional screening to obtain reliable candidate solutions. Finally, the reliable candidate solutions are optimized using a preset objective function to obtain the target pose relationship of the sample vehicle. This method performs pre-processing quality filtering at the observation sample level, reducing the input of bad data from the source. Simultaneously, by generating multiple candidate solutions and implementing multi-level verification and screening, abnormal pose solutions caused by various factors are effectively eliminated. It no longer relies on a single result obtained from a single solution. Based on this, global optimization is implemented to continuously reduce calibration deviations caused by observation noise and feature defects, significantly improving the accuracy and stability of the pose calibration results between the robotic arm and the target display surface, fully meeting the usage requirements of automated high-precision touch testing for automotive displays.

[0175] The inventors also discovered that existing methods for calibrating pose relationships often require point-by-point teaching or manual repositioning for each vehicle, resulting in long calibration cycles and a significant amount of repetitive work. When the number of test vehicles is large, the cumulative working hours increase significantly.

[0176] Especially for mass-produced vehicles of the same model, platform, in-vehicle screen module, and assembly tolerance standards, the installation reference of the in-vehicle display interaction surface is highly consistent. Only minor local deviations exist between vehicles due to assembly processes, camera installation, and environmental disturbances. However, existing calibration schemes cannot reuse completed reference calibration results. If the reuse accuracy is insufficient, the entire calibration process must be repeated, preventing minor adjustments for small deviations and resulting in low calibration efficiency and high costs. Therefore, this invention also provides a pose relationship calibration method. This method retrieves the optimal pose relationship of a sample vehicle of the same specification as a reference. It first verifies whether the accuracy of directly reusing this reference pose on the target test vehicle meets the requirements. If the accuracy meets the requirements, the reference pose is directly reused. If the accuracy is insufficient, only a small incremental correction to the reference pose is needed to obtain the calibration pose adapted to the current vehicle, eliminating the need to completely rerun the entire calibration process. This significantly improves vehicle calibration efficiency in mass production scenarios and reduces the manpower and time costs of batch calibration.

[0177] Figure 6 A flowchart of a third embodiment of the pose relationship calibration method provided by the present invention is shown. Figure 6 As shown, the method includes the following steps:

[0178] Step 310: For the target test vehicle, obtain the target pose relationship with the sample vehicle corresponding to the target test vehicle.

[0179] For example, the target test vehicle is the vehicle for which the pose reuse calibration of the robotic arm and display surface is to be completed. The vehicle carries at least one target display surface among the central control screen, instrument screen, and passenger screen, and works with the robotic arm, vision acquisition device, and electronic equipment to perform subsequent verification and correction processes.

[0180] The sample vehicle is a reference vehicle that has already undergone prior calibration, and its purpose is to provide reusable pose relationship results. The sample vehicle and the target test vehicle are vehicles of the same model, platform, and assembly process.

[0181] The target pose relationship represents the spatial transformation relationship between the robot arm's base coordinate system and the target display surface coordinate system. Specifically, it can be represented by a 4x4 homogeneous transformation matrix, where the rotation component corresponds to the orientation of the target display surface relative to the robot arm's base coordinate system, and the translation component corresponds to the position of the origin of the target display surface in the robot arm's base coordinate system. Alternatively, the rotation matrix and translation vector can be stored separately, or a combination of quaternions and translation vectors can be used, as long as the spatial pose relationship can be fully represented. Optionally, the electronic device can obtain the target pose relationship from the reusable calibration assets of the corresponding sample vehicle in the calibration asset library.

[0182] In one example, the electronic device first receives the vehicle identifier, model identifier, cockpit platform identifier, display module number, and robotic arm installation station number of the target test vehicle, and uses this information to determine the corresponding sample vehicle. After retrieving the sample vehicle, it reads the target pose relationship generated by previous calibration from the sample vehicle's historical calibration records. This previous calibration generation process can involve collecting observation sample information of the target display surface from the sample vehicle under multiple standard observation poses, generating candidate pose relationship solutions, filtering reliable candidate solutions, and optimizing the solution to obtain the final solution. Since this result has already undergone effective calculation and filtering on the sample vehicle, it can be directly used as the basic input for the reuse verification of the target test vehicle.

[0183] Step 320: Determine the reuse verification accuracy index based on the target pose relationship.

[0184] For example, the reuse verification accuracy index is used to characterize the degree to which the target pose relationship of the sample vehicle is suitable for the target test vehicle, so as to confirm whether the target pose relationship meets the accuracy requirements and whether it meets the requirements for use in real vehicle-mounted automated testing.

[0185] In one example, an electronic device controls a robotic arm to move to a preset verification observation pose, observes and collects verification observation sample information on the target display surface, obtains verification data for calculating the reuse effect based on the target pose relationship and the verification observation sample information, and determines the reuse verification accuracy index based on the acquired verification data. By directly applying the target pose relationship to the target test vehicle and calculating the verification results, this step quantitatively determines the reuse scheme without re-performing a complete calibration, ensuring that subsequent reuse decisions are based on the measured data of the target test vehicle.

[0186] In one possible implementation, if the target test vehicle contains multiple target display surfaces, then multiplexing verification is performed on each target display surface separately, and the multiplexing verification accuracy index corresponding to each target display surface is determined separately.

[0187] It should be noted that the reuse verification process is not limited to a unified judgment of the entire vehicle at once, but can also be completed independently at the target display surface level.

[0188] Step 330: Determine whether the reuse verification accuracy index meets the preset conditions.

[0189] For example, the preset condition is a threshold for determining whether direct reuse is allowed. This threshold can be a single condition or a combination of conditions where multiple verification items are satisfied simultaneously. For error-type indicators, the preset condition can be set to the indicator being less than or equal to a preset threshold; for success rate-type indicators, the preset condition can be set to the indicator being greater than or equal to a preset threshold; for comprehensive indicators, the preset condition can be set to the comprehensive score meeting a preset range.

[0190] In one example, after obtaining the reuse verification accuracy index, the electronic device compares the index with preset conditions. If the preset conditions are met, step 340 is executed; otherwise, step 350 is executed.

[0191] Step 340: Determine the target pose relationship as the calibration pose relationship of the target test vehicle.

[0192] For example, the calibration pose relationship is the final pose calibration output used for the target test vehicle. Subsequent click, swipe, long press and regression test actions of the robotic arm on the target display surface are all based on this result to generate trajectories and postures.

[0193] In one example, when preset conditions are met, the electronic device directly writes the target pose relationship obtained from the sample vehicle into the calibration result file corresponding to the target test vehicle. Simultaneously, it records the time of this reuse verification, the set of verification points, the index calculation results, and the execution device number, enabling the target test vehicle to directly call the calibrated pose relationship solution in subsequent test tasks. The result file can be stored on a local industrial control computer, production line server, or centralized calibration database and associated with the vehicle's unique identification code.

[0194] In one possible implementation, before writing the calibration pose relationship solution, the electronic device performs format standardization processing on the target pose relationship, including unifying coordinate system naming, completing matrix dimensions, adding version numbers, and writing target display surface identifiers, to ensure that subsequent control software can directly parse and use it across different vehicles and test stations. After writing is completed, a calibration completion signal is issued to the robotic arm control module, allowing the test task scheduling module to switch the vehicle to the formal automated touch test process.

[0195] When the reuse verification accuracy index meets the preset conditions, it indicates that the sample vehicle and the target test vehicle have sufficient consistency in display module position, assembly status and workstation relationship. The target pose relationship can directly support the robotic arm to perform interactive actions on the target display surface. Therefore, there is no need to recalibrate completely. The target test vehicle can form a calibration pose relationship solution that can be executed by the target test vehicle simply by registering the results.

[0196] Step 350: Correct the target pose relationship to obtain the calibration pose relationship of the target test vehicle.

[0197] For example, the correction process is used to compensate for and optimize the target pose relationship when the target pose relationship of the sample vehicle cannot directly meet the accuracy requirements of the target test vehicle. The correction process is a lightweight fine-tuning process based on the original target pose relationship to compensate for the small individual assembly deviations of the target test vehicle relative to the sample vehicle, which is different from the complete full-process calibration starting from scratch; the correction method can rely on a small number of visual observation samples and a small number of interactive samples to carry out small perturbation incremental optimization and output an updated pose that adapts to the deviation of a single vehicle.

[0198] In one example, an electronic device controls a robotic arm to move to a preset verification observation pose to collect visual data and obtain verification observation sample information. Then, optimization constraints can be constructed based on the verification observation features. The target pose relationship of the sample vehicle is used as a priori benchmark. Incremental correction is carried out through a Lie group small perturbation model to obtain the pose result that adapts to the individual deviation of the target test vehicle, which serves as the calibration pose relationship of the target test vehicle.

[0199] In this embodiment, the target pose relationship of a sample vehicle of the same model is first retrieved as a reuse reference. This target pose relationship is then applied to the target test vehicle, and the reuse verification accuracy index is calculated. A judgment is made based on the accuracy index and preset conditions. If the index is qualified, the existing calibration results are directly reused; if the index does not meet the requirements, a correction process is initiated to obtain a calibration pose relationship adapted to the target test vehicle. This method fully utilizes the similarity of installation references among vehicles of the same model, prioritizing the inheritance of completed calibration results. It avoids repeatedly performing the tedious complete calibration process for each target test vehicle, only performing lightweight corrections when the reuse accuracy is insufficient. This significantly reduces the time and labor costs of batch vehicle calibration, while simultaneously considering mass production efficiency and the calibration accuracy required for single-vehicle touch interaction, better adapting to the production line operation scenario of large-scale automated testing of automotive displays.

[0200] Figure 7 A flowchart of a fourth embodiment of the pose relationship calibration method provided by the present invention is shown. Figure 8 A schematic diagram of an application scenario provided by the present invention is shown, such as... Figure 7 and Figure 8 As shown, the method includes the following steps:

[0201] Step 410: For the target test vehicle, obtain the target pose relationship with the sample vehicle corresponding to the target test vehicle.

[0202] It should be noted that this step is similar to step 310 mentioned above, and will not be repeated here.

[0203] Step 420: Determine the reuse verification accuracy index based on the target pose relationship.

[0204] For example, based on the target pose relationship, verification observations (geometric verification) and / or verification interactions (interaction verification) can be performed to obtain reuse verification accuracy metrics. Optionally, the reuse verification accuracy metrics include at least one of the following: average geometric error, interaction success rate, and average interaction landing point deviation.

[0205] Geometric verification can be performed by repeatedly detecting the target display surface under one or more verification observation poses to obtain the average geometric error. This average geometric error can be reprojection error, transformation consistency error, boundary alignment error, or attitude deviation. For example, the average geometric error can be expressed as: ;in, It represents the average geometric error and is used to determine the calibration geometric accuracy; This represents the total number of all verification observation samples that participated in the review; Indicates the target pose relationship; This represents the reprojection error of each feature point under the target pose relationship.

[0206] Specifically, based on the target pose relationship, the average geometric error is determined, including: acquiring verification observation sample information of the robotic arm observing the display surface of the target test vehicle under preset verification observation poses; for each verification observation sample information, the geometric error of the verification observation feature is determined according to the verification observation feature and the target pose relationship; wherein, the geometric error is at least one of reprojection error, contour alignment error, boundary distance error, and pose transformation deviation; and the average geometric error is determined according to the geometric error of each verification observation feature.

[0207] The verification observation pose consists of a small number of preset poses used to collect image data during the verification phase. The verification observation sample information includes images acquired in the verification observation pose, extracted verification observation features, and the corresponding robotic arm pose at the given time. Verification observation features are visual elements such as feature points and contours extracted from the images. Geometric error characterizes the deviation generated when mapping 3D features to the image plane or matching the target contour under the target pose relationship. The average geometric error is the quantized result obtained by averaging the geometric errors corresponding to all verification observation features.

[0208] For example, the robotic arm first collects verification observation sample information of the target display surface of the test vehicle under various verification observation poses. Then, based on the target pose relationship, it calculates the geometric error of the verification observation features in the verification observation sample information and further summarizes them to obtain the average geometric error. This method is similar to the principle of geometric error calculation based on observation sample information in the aforementioned embodiments, and will not be described in detail here.

[0209] Among these methods, interactive verification, as an enhanced verification method in addition to geometric verification, involves controlling a robotic arm to perform clicks, long presses, swipes, drags, or other verification interactions on one or more verification points, controls, or areas on the display surface based on the current target pose. The success rate and landing point deviation are then calculated based on the interaction response results. The success of the k-th verification interaction is denoted as... {0,1} Interaction hit successful. If the interaction fails, the success rate can be expressed as: ;in, Indicates the success rate of the interaction, with a value ranging from 0 to 1; This represents the total number of interactive verifications. The deviation of the landing point of the kth interaction can be expressed as: ;in, Indicates the first The Euclidean deviation between the actual landing point and the target landing point of the second interaction; This represents the actual touch coordinate vector of the screen feedback; Represents the theoretical touch coordinate vector; Let L2 represent the Euclidean distance between two-dimensional coordinate vectors. The average cross-landing deviation can be expressed as... .

[0210] Specifically, based on the target pose relationship, the interaction success rate and / or average interaction landing point deviation are determined, including: based on the target pose relationship, determining the execution coordinates of the preset verification interaction target in the robot arm base coordinate system; controlling the robot arm end effector to move to the execution coordinates to execute the preset interaction action and obtaining the interaction response result; and determining the interaction success rate and / or average interaction landing point deviation based on the interaction response result.

[0211] The verification interaction targets are pre-defined screen touch points and control areas. Execution coordinates are the spatial coordinates that the robotic arm's end effector needs to reach, calculated using the target pose relationship. Pre-defined interaction actions include touch behaviors such as clicking, long-pressing, and swiping. Interaction response results include changes in screen display status, visual feedback after the target control is hit, whether the coordinate landing point falls within a preset tolerance range, system log feedback, ADB return results, vehicle bus feedback, touch event feedback information, or other response signals that characterize the success of the interaction. The interaction success rate is the percentage of times the test result matches expectations out of multiple interaction actions. The average interaction landing point deviation is the average spatial difference between the theoretical touch coordinates of the robotic arm and the actual touch coordinates recognized by the vehicle.

[0212] For example, the interaction points of a preset verification interaction target are read, and the spatial execution coordinates of the robotic arm are calculated based on the target's pose relationship. After the robotic arm's end effector reaches the execution coordinates, it performs preset interactive actions such as clicking, sliding, or dragging, and collects onboard feedback signals to obtain the interaction response results. The interaction success rate is obtained by statistically analyzing the success rate of multiple interaction response results, or the average interaction landing point deviation is determined by averaging the offset between the actual landing point and the preset verification interaction target.

[0213] By adopting this method, the reuse verification accuracy index can reflect quantitative information on geometric consistency and / or actual interactive usability, directly reflecting the stability of the target pose relationship under different verification observation poses and its hit rate in real touch, thus providing a basis for subsequently determining whether to identify it as the calibration pose relationship solution. For example, if , and If the reuse verification accuracy meets the preset conditions, then the reuse verification accuracy index is determined to not meet the preset conditions. If any item fails to meet the preset conditions, then the reuse verification accuracy index is determined to not meet the preset conditions. This indicates the maximum allowable error threshold for geometric verification; This represents the minimum success rate threshold for interactive reuse. This indicates the threshold for average hit deviation allowed by the interaction.

[0214] Step 430: If the reuse verification accuracy index does not meet the preset conditions, the preset initial perturbation vector is substituted into the preset optimization objective function for iterative optimization to obtain the optimal perturbation vector.

[0215] Here, the perturbation vector represents the translational and rotational angle deviations; the optimization objective function represents minimizing the sum of the geometric error, interaction error, and perturbation vector magnitude penalty term corresponding to the pose relationship corrected based on the perturbation vector. The purpose of this step is that if the reuse verification fails, incremental correction can be performed using only a small number of new observations and / or a small number of new interaction samples, without having to complete the entire calibration process again. Incremental correction can be performed based on one frame, a small number of frames, a small number of observation poses, or a small number of interaction samples, and the correction target can be global rigid body transformation, local attitude deviation, local surface mapping error, interaction hit deviation, or a combination thereof.

[0216] For example, the initial perturbation vector is the starting value for iterative optimization. The perturbation vector is a six-dimensional Lie algebra vector, used to characterize the small translational and rotational deviations of the target test vehicle relative to the sample reference. The optimal perturbation vector is the perturbation vector that minimizes the total loss after iterative convergence. Geometric error is the visual matching deviation corresponding to the observed sample after perturbation updates the pose relationship; interaction error is the deviation of the robotic arm's touch landing point after perturbation updates the pose relationship; the perturbation vector magnitude penalty term is used to constrain the perturbation scale, limiting the correction amplitude to a small range to avoid over-correction and destruction of the original reference pose.

[0217] In one example, the initial perturbation vector can be represented as a combination of three-dimensional translation correction and three-axis rotation correction. Substituting this perturbation vector into the optimization objective function, the electronic device calculates the reprojection error, contour alignment error, or boundary distance error corresponding to the current correction result based on the verification observation sample information. It then determines the interaction landing point deviation or interaction success rate loss by combining the interaction response results, and finally adds a perturbation vector magnitude penalty term to form the total loss. The optimizer updates the perturbation vector in each iteration, gradually approximating the actual spatial relationship of the target test vehicle after correction. Finally, the optimal perturbation vector is applied to the target pose relationship to obtain the calibration pose relationship of the target test vehicle that meets the calibration requirements. In this embodiment, the optimization parameters, convergence threshold, and penalty coefficient can all be configured according to the vehicle model, display panel size, and robotic arm accuracy. In practical applications, other models of this component can also be selected; this embodiment of the invention does not limit this.

[0218] For example, incremental correction can use the current target pose relationship as a priori and update it through small perturbations. And thus, we obtain ,in, This represents the transformation of a rigid body under minute perturbations. Represents a small perturbation vector The corresponding antisymmetric matrix; This represents the small perturbation vector to be solved in six degrees of freedom; This represents the calibration pose relationship obtained after incremental correction. Indicates the target pose relationship; This represents the exponential mapping from Lie algebras to Lie group groups. The optimal perturbation vector can be obtained by combining geometric and interaction errors, and the objective function can be expressed as: ;in, This indicates that the optimal small perturbation vector is obtained by solving the problem; Indicates the application of a disturbance The geometric reprojection error term follows; Indicates the application of a disturbance The subsequent interaction landing point deviation error term; Interaction error weighting coefficient; This indicates the perturbation regularization weight, limiting excessively large correction magnitudes; This represents the L2 norm squared regularization term of the perturbation vector.

[0219] By incorporating both geometric and interaction errors into the optimization objective and constraining the perturbation amplitude, the corrected result can simultaneously meet the requirements of spatial consistency and interaction availability.

[0220] Step 440: Based on the optimal perturbation vector and the target pose relationship, obtain the calibration pose relationship of the target test vehicle.

[0221] For example, the calibration pose relationship is a pose transformation result that has been corrected for minor perturbations and adapted to the individual assembly deviations of the target test vehicle, allowing the robotic arm to directly perform touch positioning. The optimal perturbation vector is converted into an incremental transformation matrix through Lie group exponential mapping, and a minor offset is superimposed on the original target pose relationship to achieve lightweight correction.

[0222] In one example, the electronic device obtains the incremental perturbation matrix by exponentially mapping the optimal perturbation vector, and then multiplies the incremental perturbation matrix by the target pose relationship of the sample vehicle to complete the pose update operation, thereby obtaining the calibration pose relationship adapted to the current target test vehicle.

[0223] By combining the optimal perturbation vector obtained through iterative optimization, the original target pose relationship can be directionally compensated, thereby outputting a more suitable target test vehicle pose relationship.

[0224] In some embodiments, Figure 9 This illustration shows a scenario diagram of manual correction provided by the present invention, such as... Figure 9As shown, after completing the automated pose calibration and solving the target pose relationship of the sample vehicle, a post-calibration manual correction operation can be carried out based on the benchmark calibration result to further improve the landing accuracy of the robotic arm when performing touch interaction, and at the same time compensate for various special deviations caused by assembly, imaging, environment and other factors. The post-calibration manual correction can be selected individually or in combination using the following five correction methods.

[0225] The first type is a two-dimensional affine correction based on three or more discrete reference points. The transformation expression for this correction method is: Where A represents the two-dimensional affine transformation matrix, and b represents the two-dimensional translation vector. The original pixel coordinate vector of the screen. This is the accurate coordinate vector after affine correction; pixel-level overall deviation correction is achieved through matrix linear transformation and superposition of translation. This type of correction is applicable to local translation, scaling, and rotation-related landing point deviations.

[0226] The second type is a two-dimensional perspective correction based on four or more discrete reference points. This correction relies on the homography matrix to complete pixel coordinate mapping, and the transformation expression is denoted as... In the formula, H is the homography transformation matrix, which corrects screen perspective distortion; (symbol) It represents the perspective equivalence mapping relationship under homogeneous coordinates, and represents the arbitrary scale factor difference, which can correct the landing point deviation caused by perspective distortion in screen imaging; , Input and output pixel coordinates in homogeneous form. This type of correction is suitable for compensating for changes in viewpoint or planar perspective deviation.

[0227] The third type involves the reconstruction and correction of a three-dimensional user coordinate system based on three or more discrete reference points. This method relies on three-dimensional rigid body transformation to complete the fine-tuning of the reference pose and construct a correction transformation matrix. The reference calibration pose output by automation is updated, and the final corrected complete calibration pose satisfies the following relation. ,in, To automatically calibrate the output of the optimal target pose relationship between the base and the display surface. This is the final calibration pose relationship solution after 3D manual correction.

[0228] The fourth type is a correction method based on multi-point sampling to construct a local error mapping table. By collecting the actual deviation data of multi-point touch landing points in different areas of the screen, a mapping table is established that corresponds one-to-one between screen coordinates and landing point compensation offset. The local deviation compensation is completed by looking up the table during subsequent touch positioning of the robotic arm.

[0229] The fifth type is a correction method that relies on machine learning training to obtain the error compensation model. It uses the original calibration landing point error data obtained from multiple sets of sampling to train the model. By inputting the touch point on the screen, the corresponding compensation offset can be output, realizing adaptive compensation of nonlinear error across the entire domain.

[0230] The final calibration pose relationship solution obtained after any one or more of the above-mentioned post-approval manual corrections can be synchronously stored in the calibration asset library along with the original target pose relationship obtained by the automated process to complete the archiving and storage. This makes it easy for subsequent target test vehicles of the same model to directly reuse this set of corrected calibration assets, while also supporting continuous iterative updates of correction parameters and calibration assets during subsequent batch vehicle testing.

[0231] In this embodiment, the target pose relationship of a sample vehicle of the same model is first retrieved as a reuse benchmark. By collecting and verifying observation samples and executing verification interaction actions, multiple reuse verification accuracy indicators are calculated from the perspectives of visual geometry and real touch conditions. When the reuse accuracy does not meet the preset conditions, a perturbation vector is introduced to represent the small deviations of individual vehicles. An optimization objective function that integrates geometric error, interaction error, and perturbation amplitude constraints is constructed to solve for the optimal perturbation vector. Based on the optimal perturbation vector, a lightweight correction is performed on the benchmark pose to obtain the calibration pose relationship solution corresponding to the target test vehicle. In this way, the benchmark reuse is achieved by making full use of the calibration results of the same model, without having to repeat the complete calibration process for each target test vehicle. At the same time, a small perturbation model is used to carry out lightweight incremental correction, only compensating for the small assembly deviations caused by individual vehicles. This balances the calibration efficiency in batch testing scenarios with the pose accuracy required for robotic arm touch interaction, improving the adaptability and practicality of mass production automated testing solutions.

[0232] Figure 10 A schematic diagram of an application scenario provided by the present invention is shown, such as... Figure 10 As shown, the calibration and testing system architecture is divided into three layers: the equipment layer, the algorithm layer, and the asset layer. Specifically, it includes a robotic arm 510, a camera 520, a display surface 530, a controller 540, a calibration processing module 550, and a calibration asset library 560. Automatic calibration and subsequent maintenance are completed collaboratively through observation data streams, solution data streams, and calibration asset reuse links. Specifically, the controller 540 controls the robotic arm 510 and camera 520 to perform observation, acquisition, and interaction actions; the calibration processing module 550 performs automatic calibration, result verification, and correction processing; and the calibration asset library 560 stores and loads reusable calibration assets.

[0233] The device layer includes four types of hardware units: controller, robotic arm, camera, and target display surface. The controller is responsible for overall motion and acquisition control. On the one hand, it schedules the robotic arm to complete displacement motion and outputs the pose data of its own base and end effector. On the other hand, it schedules the camera to complete shooting observation and output image data. The camera acquires images of the target display surface. The two types of data, robotic arm pose and camera image, are simultaneously transmitted to the calibration processing module of the algorithm layer (deployed in the electronic device).

[0234] The algorithm layer is the core computing unit. The calibration processing module executes multiple processing logics sequentially: the standard observation pose set generation or acquisition unit 551 first acquires the standard observation pose; the robotic arm motion and image acquisition unit 552 performs motion and image acquisition; the visual feature extraction and quality assessment unit 553 performs feature extraction and sample quality assessment; the candidate relationship solution calculation unit 554 calculates multiple sets of candidate pose relationship solutions; and the multi-level screening and robust optimization unit 555 performs screening and robust optimization on the candidate solutions to obtain the benchmark calibration pose (target pose relationship). This benchmark calibration result is stored in the calibration asset library of the asset layer to form a reusable calibration asset. When calibrating the vehicle to be tested in subsequent mass production, the result verification, reuse, and incremental correction unit 556 retrieves historical calibration assets from the calibration asset library to perform reuse verification and incremental correction when necessary. The posterior manual correction unit 557 can perform subsequent manual posterior correction, and finally outputs the optimized pose transformation result from the base to the display surface. The incremental correction stage will also connect to the calibration asset library again to complete data loading and verification.

[0235] The asset layer stores all completed calibration results in the calibration asset library. It can receive new calibration assets output by the algorithm layer for archiving, and also provide historical calibration assets to the algorithm layer to support the calibration reuse, verification and incremental fine-tuning of subsequent vehicles. It does not need to repeat the complete calibration process, realizes the reuse of calibration assets across vehicles, and improves the calibration efficiency of batch vehicle testing.

[0236] Figure 11 A schematic diagram of a process provided by the present invention is shown, such as... Figure 11 As shown, the embodiments of the present invention can be applied to the consistency testing scenario of intelligent vehicle production lines, and may specifically include the following steps:

[0237] S0. Before implementing the calibration process, first confirm the vehicle model, vehicle platform, vehicle display module specifications, and camera installation layout of the sample vehicle, and complete the camera intrinsic parameter calibration and hand-eye calibration in advance. Among them, the camera intrinsic parameter calibration is used to calculate the camera focal length, principal point coordinates, and lens distortion parameters to establish an accurate imaging model; the hand-eye calibration is used to solve the fixed pose transformation relationship between the camera coordinate system and the robot arm base coordinate system, realize the accurate conversion of the two-dimensional image observation information to the three-dimensional executable space coordinates of the robot arm, and provide basic calibration conditions for subsequent visual observation, pose positioning, and robot arm motion control.

[0238] S1. Based on the model or platform of the sample vehicle, retrieve or generate a standard observation pose set PoseSet. This standard observation pose set preferably covers the central region, edge region, and tilted view region of the target display interactive surface. Sufficient geometric constraints are constructed through multi-region and multi-view observations to ensure that the candidate pose relationship solutions obtained in subsequent solutions are effective and reliable.

[0239] S2. Drive the robotic arm to move sequentially to each preset standard observation pose in PoseSet. At least one scene image is acquired under each standard observation pose. For complex working conditions such as screen reflection and large fluctuations in ambient brightness, multiple sets of image samples with different exposure parameters and different screen display states can be acquired. The corresponding robotic arm end pose, sampling time, and image parameters are recorded simultaneously. The observation sample information set ObsSet is then constructed in a unified manner to complete the collection of calibration raw data.

[0240] The control target displays the output of preset calibration reference information. This reference information preferably adopts a single type or a combination of multiple types of ChArUco code, AprilTag code, checkerboard pattern, and screen boundary corner points. It takes into account both the high recognition and positioning capabilities of the encoded features and the sub-pixel precision positioning capabilities of the geometric corner points, providing a stable benchmark for subsequent feature extraction and pose calculation.

[0241] S3. Perform image preprocessing and sample quality assessment on all image samples in the observation sample information set ObsSet in sequence, and output the effective feature set FeatureSet. The preprocessing process includes image distortion correction, local contrast enhancement, edge optimization processing, and then accurately extracts corner points, coded feature points, screen boundaries and contour features in the image. At the same time, the sample quality is evaluated from multiple dimensions such as feature quantity, spatial coverage uniformity, image clarity and feature confidence.

[0242] Optionally, the number of standard observation poses can be set to 6 to 12, and 1 to 3 image samples can be acquired for each observation pose. For planar target display interactive surfaces, the three-dimensional reference point can be defined on the Z=0 reference plane, and the geometric error can be calculated based on the aforementioned reprojection model.

[0243] Optionally, the unified calibration threshold can be configured as follows: the effective feature quantity threshold is not less than 20, the reprojection error threshold is 1 to 5 pixels, the translation deviation consistency threshold for candidate pose relationship solution consistency screening is 1 to 5 mm, and the rotation angle deviation consistency threshold is 1° to 5°; the interaction success rate threshold in the reuse verification stage is not less than 90%, or the average touch interaction landing point deviation threshold is 1 to 5 mm. All threshold parameters can be adaptively adjusted according to the display interaction surface size, camera resolution, camera installation method, and actual calibration accuracy requirements.

[0244] S4. Based on the effective feature set (FeatureSet), camera intrinsic parameters, hand-eye transformation relationship, and robotic arm pose data corresponding to each observation pose, multiple sets of candidate pose relationship solutions are obtained from multiple perspectives, constructing a candidate solution (CandidateSet). When the calibration reference information is discrete feature points, PnP pose solving, planar pose estimation, and nonlinear least squares algorithms can be used to solve the pose; when the calibration reference information is screen boundaries or continuous contours, contour fitting and contour alignment algorithms can be used to solve the pose. Each set of candidate pose relationship solutions is solved iteratively by minimizing the geometric error of a single observation sample or the joint geometric error of multiple observation samples, and the average reprojection error and solution consistency index corresponding to each set of solutions are recorded simultaneously to provide a quantitative basis for subsequent multi-level screening.

[0245] S5. Perform multi-level hierarchical filtering on the candidate solution set CandidateSet, eliminating outlier solutions at each level. Preferably, the filtering is performed in the order of feature quality filtering, reprojection error filtering, and multi-view consistency filtering. At the same time, physical constraint filtering and statistical robustness filtering can be added according to the calibration scenario to completely eliminate invalid candidate pose relationship solutions caused by noise and abnormal observations, and retain the set of highly reliable candidate solutions.

[0246] Robust global optimization is performed on the high-confidence candidate pose relationship solutions retained after multi-level screening. The optimization objective is to minimize the overall geometric error of all valid observation samples, and the optimal target pose relationship from the base to the display surface is obtained through iterative solution. During the optimization process, a robust loss function, observation sample weights, and optional historical prior pose constraints are introduced to effectively suppress the interference of individual abnormal observation samples and outliers on the overall optimization results, thereby improving the robustness and accuracy of the optimal pose solution.

[0247] S6. The optimal target pose relationship, along with the standard observation pose set PoseSet, verification observation pose, verification reference point, verification control definition, candidate solution selection rules, optimized prior parameters, and corrected parameters, are uniformly encapsulated into a calibration asset set AssetSet and persistently stored. For subsequent target test vehicles of the same model, the same vehicle platform, the same display module, and the same assembly tolerance level, the calibration asset set AssetSet can be directly loaded into the reuse verification process, without having to repeat the entire calibration process. The AssetSet synchronously archives geometric accuracy judgment thresholds and interaction accuracy judgment thresholds, providing a standardized basis for the rapid reuse and judgment of calibration assets for subsequent vehicles.

[0248] S7. For subsequent batch target test vehicles, only a small amount of verification observation sampling and / or verification touch interaction tests will be performed. The robotic arm, relying on the reused calibration pose, will perform standard verification interaction actions such as clicking and swiping on the preset verification reference points and preset UI controls. The interaction will be determined by combining the screen feedback signal, vehicle system log, and touch event feedback results to determine whether the interaction effectively hits the target area. If the reused verification accuracy index meets the preset accuracy conditions, the existing calibration assets will be directly reused as the vehicle's calibration pose relationship solution. If the accuracy index does not meet the preset conditions, incremental correction processing will be carried out based on a small number of newly added observation samples and a small number of newly added interaction samples. The rigid body transformation relationship from the base to the display surface, local posture deviation, local mapping error, and touch interaction landing point deviation will be updated, and the incrementally corrected updated pose relationship solution will be output.

[0249] S8. After the automatic calibration and incremental correction process is completed, if it is still necessary to further improve the accuracy of the robotic arm's touch landing point and compensate for residual minor deviations, post-calibration manual correction can be carried out based on three or more reference points on the target display interactive surface. Post-calibration manual correction can be used alone or in combination with two-dimensional affine correction, two-dimensional perspective correction, three-dimensional user coordinate system reconstruction, and multi-point sampling local error mapping compensation methods to achieve secondary optimization and reinforcement of calibration accuracy.

[0250] S9. This entire process forms a complete closed-loop calibration mechanism, including fully automated and accurate calibration of the first vehicle, archiving and storing calibration assets, rapid reuse of batch vehicles, incremental correction of abnormal deviations, and manual post-test reinforcement. While significantly reducing calibration time and repetitive labor in mass production scenarios, it also takes into account calibration posture accuracy, result stability, reusability of calibration assets, and long-term operation and maintenance costs, making it highly adaptable to batch calibration scenarios for automated touch testing of mass production vehicles.

[0251] In summary, the embodiments of the present invention have the following technical advantages:

[0252] 1. High degree of automation. Due to the adoption of a multi-view acquisition and automatic solution mechanism driven by a standard observation pose set, the traditional point-by-point manual teaching process is reduced or eliminated, thus significantly shortening the deployment time of the first vehicle.

[0253] 2. Enhanced robustness. Because it does not rely on a single frame or a single solution, but rather determines the final result through multi-view redundancy, quality assessment, outlier removal, and robust optimization, it can better adapt to in-vehicle reflections, shadows, occlusions, and changes in viewing angle.

[0254] 3. More stable accuracy. Because the target relation solution is determined based on the consistency among multiple candidate relation solutions and the minimization of global error, rather than directly using the result of a single solution, the overall accuracy and repeatability are more stable.

[0255] 4. High reusability. Since the pose set, calibration results, prior and error models accumulated in the first vehicle stage can form reusable calibration assets, they can be directly reused on vehicles of the same model, platform or module, greatly reducing the cost of repeated calibration.

[0256] 5. Low maintenance costs. Since subsequent vehicles can be quickly verified first, and the availability of the actual touch link can be directly confirmed through a small number of verification interactions, and then a small number of incremental corrections can be performed as needed, without having to return to the complete calibration process, the long-term maintenance costs are lower.

[0257] 6. Higher practical value. Since the verification phase can not only check whether the geometric relationship is correct, but also directly check whether the robotic arm's actual interaction with the vehicle's screen is successful, it is closer to the actual use scenario in vehicle automation testing.

[0258] 7. Wide applicability. Because the solution does not rely on a single calibration pattern or a single surface morphology, but rather revolves around the generalized modeling of the interactive display surface, it can be applied to flat screens, as well as segmented planes, curved surfaces, or other predefined interactive display surfaces.

[0259] 8. High scalability. Since the feature types, pose solving methods, nonlinear optimizers, posterior correction methods, calibration asset reuse mechanisms, and interactive verification mechanisms in this solution can be freely replaced, adapted, or flexibly combined, this calibration method can adapt to in-vehicle display interaction surfaces of different sizes and geometries, is compatible with different camera installation layouts and hardware configurations, and can flexibly adapt to mass-produced vehicle calibration scenarios with multiple vehicle models, platforms, and assembly tolerance levels. Furthermore, it can iteratively update algorithm modules and parameter configurations according to actual test accuracy requirements and changes in working conditions, possessing extremely strong scenario adaptability and functional scalability.

[0260] Figure 12 A schematic diagram of an embodiment of the pose relationship calibration device provided by the present invention is shown. Figure 12 As shown, the calibration device 600 for the pose relationship includes: a first acquisition module 610, a calculation module 620, a filtering module 630, and an optimization module 640.

[0261] The first acquisition module 610 is used to acquire observation sample information of the target display surface of the sample vehicle observed by the robotic arm in various standard observation poses.

[0262] The solution module 620 is used to perform pose calculation based on the observed sample information to obtain multiple sets of candidate pose relationship solutions; among them, the pose relationship solution represents the pose transformation relationship between the target display surface coordinate system and the robot arm base coordinate system;

[0263] The filtering module 630 is used to filter candidate pose relation solutions to obtain reliable candidate solutions;

[0264] The optimization module 640 is used to optimize the credible candidate solutions to obtain the target pose relationship of the sample vehicle.

[0265] In an alternative approach, the observation sample information includes observation features, and the solution module 620 is used for:

[0266] Based on the observation characteristics, the corresponding observation sample information is evaluated to obtain the quality score of the observation sample information;

[0267] Based on the quality score, the observation sample information is filtered to obtain valid observation sample information;

[0268] Pose calculation is performed based on valid observation sample information to obtain multiple sets of candidate pose relationship solutions.

[0269] In one alternative approach, the solver module 620 is used for:

[0270] Based on the observation characteristics, determine the quality assessment indicators of the observation sample information corresponding to the observation characteristics; among them, the quality assessment indicators include at least one of the following: feature quantity, feature coverage uniformity, feature detection confidence, image clarity, image sharpness, boundary closure, corner stability, exposure rationality, and local occlusion ratio.

[0271] The quality score of the observed sample information is determined based on the quality assessment indicators of the observed sample information.

[0272] In one alternative approach, the effective observation sample information includes the end-effector pose at the time of observation; the observation features include the positional correspondence between the pixel coordinates of the observation features and the three-dimensional coordinates of the target display surface; the solution module 620 is used for:

[0273] For each valid observation sample, based on the pixel coordinates and positional correspondence of the observed features and the preset camera intrinsic parameters, the first pose transformation relationship between the camera coordinate system and the target display surface coordinate system is solved.

[0274] Based on the pre-calibrated hand-eye relationship, the first pose transformation relationship, and the end-effector pose during observation, determine the second pose transformation relationship of the target display surface relative to the robot arm base coordinate system.

[0275] The target error function is constructed based on the position correspondence, and the second pose transformation relationship is used as the initial value for iteration. The optimization is carried out iteratively with the goal of minimizing the target error function, and a set of candidate pose relationship solutions are obtained.

[0276] In one alternative approach, the solver module 620 is used for:

[0277] Based on the type of observed features and the geometric type of the target display surface, determine the type of target error function and the target optimization strategy;

[0278] Construct the target error function based on the correspondence between the type and location of the target error function;

[0279] According to the objective optimization strategy, the second pose transformation relationship is used as the initial value for iteration, and the optimization objective is to minimize the objective error function to perform iterative optimization, thereby obtaining a set of candidate pose relationship solutions.

[0280] In an alternative approach, the observation sample information includes observation features, and the filtering module 630 is used for:

[0281] Based on the preset screening dimensions, the candidate pose relation solutions are screened to obtain reliable candidate solutions;

[0282] The selection dimensions include at least one of the following:

[0283] Quality assessment index of the observation features corresponding to candidate pose relation solutions;

[0284] The average reprojection error of the observation features corresponding to the candidate pose relation solution;

[0285] Consensus score of candidate pose relation solutions;

[0286] Physical constraints;

[0287] Global statistical model.

[0288] In one alternative approach, the filtering module 630 is used for:

[0289] For each candidate pose relation solution, determine the translational deviation and rotational angle deviation between the candidate pose relation solution and other candidate pose relation solutions;

[0290] The consensus score of a candidate pose relation is determined based on the number of other candidate pose relation solutions that satisfy the consistency condition. The consistency condition is that the translation deviation is less than or equal to the translation deviation consistency threshold and the rotation angle deviation is less than or equal to the rotation deviation consistency threshold.

[0291] In one alternative approach, optimization module 640 is used for:

[0292] Based on the preset objective function, the credible candidate solution is optimized to obtain the target pose relationship of the sample vehicle; wherein, the objective function is to minimize the sum of geometric errors after processing by the robust loss function, and the geometric error represents the error of the observation features in the observation sample information corresponding to the credible candidate solution. The geometric error is at least one of the following errors: reprojection error, contour alignment error, boundary distance error, and surface model constraint error.

[0293] In one alternative approach, the objective function is to minimize the sum of the geometric error and the bias penalty term after processing by the robust loss function; the bias penalty term characterizes the degree of spatial offset between the reliable candidate solution and the historical prior pose.

[0294] In one alternative approach, the first acquisition module 610 is used for:

[0295] Obtain multiple candidate observation poses;

[0296] For each candidate observation pose, determine the visibility index, coverage contribution index, and risk index of the candidate observation pose;

[0297] A comprehensive score for candidate observation poses is determined based on visibility indicators, coverage contribution indicators, and risk indicators.

[0298] Candidate observation poses with a comprehensive score greater than a preset threshold are used as standard observation poses.

[0299] As can be seen from the above, the pose relationship calibration device provided in this embodiment of the invention can no longer rely on a single solution obtained from a single operation as the calibration result. By generating multiple sets of candidate solutions to enrich the selection range, and by using a screening process to remove abnormal solutions with large deviations to reduce the interference caused by bad data, global optimization is carried out based on the purified reliable candidate solutions. This effectively alleviates the calibration deviation problem caused by single observation noise and poor feature quality, and continuously improves the accuracy and stability of the pose relationship calibration results between the robot arm base coordinate system and the target display surface, thus better meeting the actual needs of automated high-precision touch testing of vehicle display devices.

[0300] Figure 13 A schematic diagram of an embodiment of the pose relationship calibration device provided by the present invention is shown. Figure 13 As shown, the calibration device 700 for the pose relationship includes: a second acquisition module 710, a determination module 720, and a verification module 730.

[0301] The second acquisition module 710 is used to acquire the target pose relationship of the sample vehicle corresponding to the target test vehicle for the target test vehicle.

[0302] The determination module 720 is used to determine the reuse verification accuracy index based on the target pose relationship;

[0303] The verification module 730 is used to determine the target pose relationship as the calibration pose relationship of the target test vehicle if the reuse verification accuracy index meets the preset conditions; if the reuse verification accuracy index does not meet the preset conditions, the target pose relationship is corrected to obtain the calibration pose relationship of the target test vehicle.

[0304] In one alternative approach, the reused verification accuracy metrics include at least one of the following: average geometric error, interaction success rate, and average interaction landing point deviation.

[0305] Determine module 720, used for:

[0306] Acquire verification observation sample information of the display surface of the target test vehicle observed by the robotic arm in various preset verification observation poses;

[0307] For each verification observation sample, the geometric error of the verification observation feature is determined based on the relationship between the verification observation feature and the target pose. The geometric error is at least one of the following: reprojection error, contour alignment error, boundary distance error, and pose transformation deviation.

[0308] The average geometric error is determined based on the geometric error of each verification observation feature;

[0309] Determine module 720, used for:

[0310] Based on the target pose relationship, determine the execution coordinates of the preset verification interaction target in the robot arm base coordinate system;

[0311] Control the robotic arm's end effector to move to the execution coordinates to perform a preset interactive action and obtain the interactive response result;

[0312] Based on the interaction response results, determine the interaction success rate and / or average interaction landing point deviation.

[0313] In one alternative approach, the verification module 730 is used for:

[0314] The preset initial perturbation vector is substituted into the preset optimization objective function for iterative optimization to obtain the optimal perturbation vector; where the perturbation vector represents the translation deviation and rotation angle deviation; the optimization objective function represents minimizing the sum of geometric error, interaction error, and perturbation vector magnitude penalty term corresponding to the pose relationship after correction based on the perturbation vector;

[0315] Based on the optimal perturbation vector and the target pose relationship, the calibration pose relationship of the target test vehicle is obtained.

[0316] As can be seen from the above, the pose relationship calibration device provided by the embodiments of the present invention can make full use of the similarity of the installation reference between vehicles of the same model, prioritize the inheritance of the completed calibration results, avoid repeating the tedious complete calibration process for each target test vehicle, and only carry out lightweight correction when the reuse accuracy is insufficient, which greatly reduces the time and manpower cost of batch vehicle calibration, while taking into account the mass production efficiency and the calibration accuracy required for single vehicle touch interaction, and better adapts to the production line operation scenario of large-scale vehicle display automated testing.

[0317] Figure 14 The diagram shows a structural schematic of an embodiment of the electronic device provided by the present invention. The specific embodiments of the present invention do not limit the specific implementation of the electronic device.

[0318] like Figure 14 As shown, the electronic device may include: a processor 802, a communications interface 804, a memory 806, and a communications bus 808.

[0319] The processor 802, communication interface 804, and memory 806 communicate with each other via communication bus 808. Communication interface 804 is used to communicate with other network elements such as clients or other servers. Processor 802 executes program 810, specifically performing the relevant steps described in the embodiment of the pose relationship calibration method.

[0320] Specifically, program 810 may include program code, which includes computer-executable instructions.

[0321] Processor 802 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The electronic device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0322] Memory 806 is used to store program 810. Memory 806 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0323] Specifically, program 810 can be called by processor 802 to cause the electronic device to perform the following operations:

[0324] For the sample vehicle, acquire the observation sample information of the target display surface of the sample vehicle observed by the robotic arm in various standard observation poses;

[0325] Pose calculation is performed based on observed sample information to obtain multiple sets of candidate pose relationship solutions; among them, the pose relationship solution characterizes the pose transformation relationship between the target display surface coordinate system and the robot arm base coordinate system;

[0326] The candidate pose relation solutions are filtered to obtain reliable candidate solutions;

[0327] The target pose relationship of the sample vehicle is obtained by optimizing the credible candidate solution.

[0328] In one alternative approach, the observation sample information includes observation features. Pose calculation is performed based on the observation sample information to obtain multiple sets of candidate pose relation solutions, including:

[0329] Based on the observation characteristics, the corresponding observation sample information is evaluated to obtain the quality score of the observation sample information;

[0330] Based on the quality score, the observation sample information is filtered to obtain valid observation sample information;

[0331] Pose calculation is performed based on valid observation sample information to obtain multiple sets of candidate pose relationship solutions.

[0332] In one alternative approach, the quality of the corresponding observed sample information is assessed based on the observed features to obtain a quality score for the observed sample information, including:

[0333] Based on the observation characteristics, determine the quality assessment indicators of the observation sample information corresponding to the observation characteristics; among them, the quality assessment indicators include at least one of the following: feature quantity, feature coverage uniformity, feature detection confidence, image clarity, image sharpness, boundary closure, corner stability, exposure rationality, and local occlusion ratio.

[0334] The quality score of the observed sample information is determined based on the quality assessment indicators of the observed sample information.

[0335] In one alternative approach, the effective observation sample information includes the pose of the robotic arm's end effector at the time of observation; the observation features include the positional correspondence between the pixel coordinates of the observation features and the three-dimensional coordinates of the target display surface; and pose calculation is performed based on the effective observation sample information to obtain multiple sets of candidate pose relationship solutions, including:

[0336] For each valid observation sample, based on the pixel coordinates and positional correspondence of the observed features and the preset camera intrinsic parameters, the first pose transformation relationship between the camera coordinate system and the target display surface coordinate system is solved.

[0337] Based on the pre-calibrated hand-eye relationship, the first pose transformation relationship, and the end-effector pose during observation, determine the second pose transformation relationship of the target display surface relative to the robot arm base coordinate system.

[0338] The target error function is constructed based on the position correspondence, and the second pose transformation relationship is used as the initial value for iteration. The optimization is carried out iteratively with the goal of minimizing the target error function, and a set of candidate pose relationship solutions are obtained.

[0339] In one alternative approach, a target error function is constructed based on the positional correspondence, and the second pose transformation relationship is used as the initial value for iteration. Iterative optimization is then performed with the goal of minimizing the target error function, resulting in a set of candidate pose relationship solutions, including:

[0340] Based on the type of observed features and the geometric type of the target display surface, determine the type of target error function and the target optimization strategy;

[0341] Construct the target error function based on the correspondence between the type and location of the target error function;

[0342] According to the objective optimization strategy, the second pose transformation relationship is used as the initial value for iteration, and the optimization objective is to minimize the objective error function to perform iterative optimization, thereby obtaining a set of candidate pose relationship solutions.

[0343] In one alternative approach, the observation sample information includes observation features, and candidate pose relation solutions are filtered to obtain reliable candidate solutions, including:

[0344] Based on the preset screening dimensions, the candidate pose relation solutions are screened to obtain reliable candidate solutions;

[0345] The selection dimensions include at least one of the following:

[0346] Quality assessment index of the observation features corresponding to candidate pose relation solutions;

[0347] The average reprojection error of the observation features corresponding to the candidate pose relation solution;

[0348] Consensus score of candidate pose relation solutions;

[0349] Physical constraints;

[0350] Global statistical model.

[0351] In one alternative approach, the steps for determining the consensus score include:

[0352] For each candidate pose relation solution, determine the translational deviation and rotational angle deviation between the candidate pose relation solution and other candidate pose relation solutions;

[0353] The consensus score of a candidate pose relation is determined based on the number of other candidate pose relation solutions that satisfy the consistency condition. The consistency condition is that the translation deviation is less than or equal to the translation deviation consistency threshold and the rotation angle deviation is less than or equal to the rotation deviation consistency threshold.

[0354] In one alternative approach, the reliable candidate solutions are optimized to obtain the target pose relationship of the sample vehicles, including:

[0355] Based on the preset objective function, the credible candidate solution is optimized to obtain the target pose relationship of the sample vehicle; wherein, the objective function is to minimize the sum of geometric errors after processing by the robust loss function, and the geometric error represents the error of the observation features in the observation sample information corresponding to the credible candidate solution. The geometric error is at least one of the following errors: reprojection error, contour alignment error, boundary distance error, and surface model constraint error.

[0356] In one alternative approach, the objective function is to minimize the sum of the geometric error and the bias penalty term after processing by the robust loss function; the bias penalty term characterizes the degree of spatial offset between the reliable candidate solution and the historical prior pose.

[0357] In one alternative approach, multiple standard observation poses are determined, including:

[0358] Obtain multiple candidate observation poses;

[0359] For each candidate observation pose, determine the visibility index, coverage contribution index, and risk index of the candidate observation pose;

[0360] A comprehensive score for candidate observation poses is determined based on visibility indicators, coverage contribution indicators, and risk indicators.

[0361] Candidate observation poses with a comprehensive score greater than a preset threshold are used as standard observation poses.

[0362] Alternatively, perform the following operations:

[0363] For the target test vehicle, obtain the target pose relationship with the sample vehicles corresponding to the target test vehicle;

[0364] Based on the target pose relationship, determine the reuse verification accuracy index;

[0365] If the reuse verification accuracy index meets the preset conditions, the target pose relationship is determined as the calibration pose relationship of the target test vehicle; if the reuse verification accuracy index does not meet the preset conditions, the target pose relationship is corrected to obtain the calibration pose relationship of the target test vehicle.

[0366] In one alternative approach, the reused verification accuracy metrics include at least one of the following: average geometric error, interaction success rate, and average interaction landing point deviation.

[0367] Based on the target pose relationship, the average geometric error is determined, including:

[0368] Acquire verification observation sample information of the display surface of the target test vehicle observed by the robotic arm in various preset verification observation poses;

[0369] For each verification observation sample, the geometric error of the verification observation feature is determined based on the relationship between the verification observation feature and the target pose. The geometric error is at least one of the following: reprojection error, contour alignment error, boundary distance error, and pose transformation deviation.

[0370] The average geometric error is determined based on the geometric error of each verification observation feature;

[0371] Based on the target pose relationship, determine the interaction success rate and / or average interaction landing point deviation, including:

[0372] Based on the target pose relationship, determine the execution coordinates of the preset verification interaction target in the robot arm base coordinate system;

[0373] Control the robotic arm's end effector to move to the execution coordinates to perform a preset interactive action and obtain the interactive response result;

[0374] Based on the interaction response results, determine the interaction success rate and / or average interaction landing point deviation.

[0375] In one alternative approach, the target pose relationship is corrected to obtain the calibration pose relationship of the target test vehicle, including:

[0376] The preset initial perturbation vector is substituted into the preset optimization objective function for iterative optimization to obtain the optimal perturbation vector; where the perturbation vector represents the translation deviation and rotation angle deviation; the optimization objective function represents minimizing the sum of geometric error, interaction error, and perturbation vector magnitude penalty term corresponding to the pose relationship after correction based on the perturbation vector;

[0377] Based on the optimal perturbation vector and the target pose relationship, the calibration pose relationship of the target test vehicle is obtained.

[0378] As can be seen from the above, the electronic device provided by the embodiments of the present invention can no longer rely on a single solution obtained from a single calculation as the calibration result. By generating multiple sets of candidate solutions to enrich the selection range, and by using a screening process to remove abnormal solutions with large deviations to reduce the interference caused by bad data, global optimization is carried out based on the purified reliable candidate solutions. This effectively alleviates the calibration deviation problem caused by single observation noise and poor feature quality, and continuously improves the accuracy and stability of the calibration results of the pose relationship between the robotic arm base coordinate system and the target display surface, thus better meeting the actual needs of automated high-precision touch testing of automotive display devices. At the same time, it can make full use of the similarity of installation benchmarks among vehicles of the same model, and prioritize the inheritance of completed calibration results, avoiding the need to repeatedly perform the tedious complete calibration process for each target test vehicle. Lightweight correction is only performed when the reuse accuracy is insufficient, which greatly reduces the time and labor costs of batch vehicle calibration, while taking into account both mass production efficiency and the calibration accuracy required for single-vehicle touch interaction, thus better adapting to the production line operation scenario of large-scale automated testing of automotive displays.

[0379] This invention provides a calibration test system, including a test vehicle, a robotic arm, and an electronic device, which is used to perform the pose relationship calibration method in any of the above method embodiments.

[0380] This invention provides a computer-readable storage medium storing at least one executable instruction. When the executable instruction is executed on an electronic device / pose relationship calibration device, the electronic device / pose relationship calibration device performs the pose relationship calibration method in any of the above method embodiments.

[0381] Specifically, the executable instructions can be used to cause the calibration device for electronic devices / pose relationships to perform the following operations:

[0382] For the sample vehicle, acquire the observation sample information of the target display surface of the sample vehicle observed by the robotic arm in various standard observation poses;

[0383] Pose calculation is performed based on observed sample information to obtain multiple sets of candidate pose relationship solutions; among them, the pose relationship solution characterizes the pose transformation relationship between the target display surface coordinate system and the robot arm base coordinate system;

[0384] The candidate pose relation solutions are filtered to obtain reliable candidate solutions;

[0385] The target pose relationship of the sample vehicle is obtained by optimizing the credible candidate solution.

[0386] In one alternative approach, the observation sample information includes observation features. Pose calculation is performed based on the observation sample information to obtain multiple sets of candidate pose relation solutions, including:

[0387] Based on the observation characteristics, the corresponding observation sample information is evaluated to obtain the quality score of the observation sample information;

[0388] Based on the quality score, the observation sample information is filtered to obtain valid observation sample information;

[0389] Pose calculation is performed based on valid observation sample information to obtain multiple sets of candidate pose relationship solutions.

[0390] In one alternative approach, the quality of the corresponding observed sample information is assessed based on the observed features to obtain a quality score for the observed sample information, including:

[0391] Based on the observation characteristics, determine the quality assessment indicators of the observation sample information corresponding to the observation characteristics; among them, the quality assessment indicators include at least one of the following: feature quantity, feature coverage uniformity, feature detection confidence, image clarity, image sharpness, boundary closure, corner stability, exposure rationality, and local occlusion ratio.

[0392] The quality score of the observed sample information is determined based on the quality assessment indicators of the observed sample information.

[0393] In one alternative approach, the effective observation sample information includes the pose of the robotic arm's end effector at the time of observation; the observation features include the positional correspondence between the pixel coordinates of the observation features and the three-dimensional coordinates of the target display surface; and pose calculation is performed based on the effective observation sample information to obtain multiple sets of candidate pose relationship solutions, including:

[0394] For each valid observation sample, based on the pixel coordinates and positional correspondence of the observed features and the preset camera intrinsic parameters, the first pose transformation relationship between the camera coordinate system and the target display surface coordinate system is solved.

[0395] Based on the pre-calibrated hand-eye relationship, the first pose transformation relationship, and the end-effector pose during observation, determine the second pose transformation relationship of the target display surface relative to the robot arm base coordinate system.

[0396] The target error function is constructed based on the position correspondence, and the second pose transformation relationship is used as the initial value for iteration. The optimization is carried out iteratively with the goal of minimizing the target error function, and a set of candidate pose relationship solutions are obtained.

[0397] In one alternative approach, a target error function is constructed based on the positional correspondence, and the second pose transformation relationship is used as the initial value for iteration. Iterative optimization is then performed with the goal of minimizing the target error function, resulting in a set of candidate pose relationship solutions, including:

[0398] Based on the type of observed features and the geometric type of the target display surface, determine the type of target error function and the target optimization strategy;

[0399] Construct the target error function based on the correspondence between the type and location of the target error function;

[0400] According to the objective optimization strategy, the second pose transformation relationship is used as the initial value for iteration, and the optimization objective is to minimize the objective error function to perform iterative optimization, thereby obtaining a set of candidate pose relationship solutions.

[0401] In one alternative approach, the observation sample information includes observation features, and candidate pose relation solutions are filtered to obtain reliable candidate solutions, including:

[0402] Based on the preset screening dimensions, the candidate pose relation solutions are screened to obtain reliable candidate solutions;

[0403] The selection dimensions include at least one of the following:

[0404] Quality assessment index of the observation features corresponding to candidate pose relation solutions;

[0405] The average reprojection error of the observation features corresponding to the candidate pose relation solution;

[0406] Consensus score of candidate pose relation solutions;

[0407] Physical constraints;

[0408] Global statistical model.

[0409] In one alternative approach, the steps for determining the consensus score include:

[0410] For each candidate pose relation solution, determine the translational deviation and rotational angle deviation between the candidate pose relation solution and other candidate pose relation solutions;

[0411] The consensus score of a candidate pose relation is determined based on the number of other candidate pose relation solutions that satisfy the consistency condition. The consistency condition is that the translation deviation is less than or equal to the translation deviation consistency threshold and the rotation angle deviation is less than or equal to the rotation deviation consistency threshold.

[0412] In one alternative approach, the reliable candidate solutions are optimized to obtain the target pose relationship of the sample vehicles, including:

[0413] Based on the preset objective function, the credible candidate solution is optimized to obtain the target pose relationship of the sample vehicle; wherein, the objective function is to minimize the sum of geometric errors after processing by the robust loss function, and the geometric error represents the error of the observation features in the observation sample information corresponding to the credible candidate solution. The geometric error is at least one of the following errors: reprojection error, contour alignment error, boundary distance error, and surface model constraint error.

[0414] In one alternative approach, the objective function is to minimize the sum of the geometric error and the bias penalty term after processing by the robust loss function; the bias penalty term characterizes the degree of spatial offset between the reliable candidate solution and the historical prior pose.

[0415] In one alternative approach, multiple standard observation poses are determined, including:

[0416] Obtain multiple candidate observation poses;

[0417] For each candidate observation pose, determine the visibility index, coverage contribution index, and risk index of the candidate observation pose;

[0418] A comprehensive score for candidate observation poses is determined based on visibility indicators, coverage contribution indicators, and risk indicators.

[0419] Candidate observation poses with a comprehensive score greater than a preset threshold are used as standard observation poses.

[0420] Alternatively, perform the following operations:

[0421] For the target test vehicle, obtain the target pose relationship with the sample vehicles corresponding to the target test vehicle;

[0422] Based on the target pose relationship, determine the reuse verification accuracy index;

[0423] If the reuse verification accuracy index meets the preset conditions, the target pose relationship is determined as the calibration pose relationship of the target test vehicle; if the reuse verification accuracy index does not meet the preset conditions, the target pose relationship is corrected to obtain the calibration pose relationship of the target test vehicle.

[0424] In one alternative approach, the reused verification accuracy metrics include at least one of the following: average geometric error, interaction success rate, and average interaction landing point deviation.

[0425] Based on the target pose relationship, the average geometric error is determined, including:

[0426] Acquire verification observation sample information of the display surface of the target test vehicle observed by the robotic arm in various preset verification observation poses;

[0427] For each verification observation sample, the geometric error of the verification observation feature is determined based on the relationship between the verification observation feature and the target pose. The geometric error is at least one of the following: reprojection error, contour alignment error, boundary distance error, and pose transformation deviation.

[0428] The average geometric error is determined based on the geometric error of each verification observation feature;

[0429] Based on the target pose relationship, determine the interaction success rate and / or average interaction landing point deviation, including:

[0430] Based on the target pose relationship, determine the execution coordinates of the preset verification interaction target in the robot arm base coordinate system;

[0431] Control the robotic arm's end effector to move to the execution coordinates to perform a preset interactive action and obtain the interactive response result;

[0432] Based on the interaction response results, determine the interaction success rate and / or average interaction landing point deviation.

[0433] In one alternative approach, the target pose relationship is corrected to obtain the calibration pose relationship of the target test vehicle, including:

[0434] The preset initial perturbation vector is substituted into the preset optimization objective function for iterative optimization to obtain the optimal perturbation vector; where the perturbation vector represents the translation deviation and rotation angle deviation; the optimization objective function represents minimizing the sum of geometric error, interaction error, and perturbation vector magnitude penalty term corresponding to the pose relationship after correction based on the perturbation vector;

[0435] Based on the optimal perturbation vector and the target pose relationship, the calibration pose relationship of the target test vehicle is obtained.

[0436] As can be seen from the above, the computer-readable storage medium provided in this embodiment of the invention stores at least one executable instruction. When this executable instruction runs on the calibration device for electronic devices / pose relationships, it can no longer rely on a single solution obtained from a single calculation as the calibration result. By generating multiple sets of candidate solutions to enrich the selection range, and by using a screening process to remove abnormal solutions with large deviations to reduce interference from bad data, global optimization is carried out based on the purified reliable candidate solutions. This effectively alleviates the calibration deviation problem caused by single observation noise and poor feature quality, and continuously improves the accuracy and stability of the pose relationship calibration results between the robot arm base coordinate system and the target display surface, better meeting the actual needs of automated high-precision touch testing of automotive display devices. At the same time, it can make full use of the similarity of installation benchmarks between vehicles of the same model, prioritize the inheritance of completed calibration results, avoid repeating the tedious complete calibration process for each target test vehicle, and only perform lightweight correction when the reuse accuracy is insufficient. This significantly reduces the time and labor costs of batch vehicle calibration, while taking into account both mass production efficiency and the calibration accuracy required for single-vehicle touch interaction, and better adapts to the production line operation scenario of large-scale automated testing of automotive displays.

[0437] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Furthermore, the embodiments of this invention are not directed to any particular programming language.

[0438] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. Similarly, in order to streamline the invention and aid in understanding one or more aspects of the invention, in the description of exemplary embodiments of the invention above, various features of the embodiments of the invention are sometimes grouped together in a single embodiment, figure, or description thereof.

[0439] Those skilled in the art will understand that the modules in the device of the embodiment can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiment can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components, except that at least some of such features and / or processes or units are mutually exclusive.

[0440] It should be noted that the above embodiments are illustrative of the invention and not restrictive of it. The invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. It is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. Unless otherwise specified, the steps in the above embodiments should not be construed as limiting the order of execution.

Claims

1. A method for calibrating pose relationships, characterized in that, include: For the sample vehicle, obtain the observation sample information of the target display surface of the sample vehicle observed by the robotic arm in various standard observation poses; Based on the observed sample information, pose calculation is performed to obtain multiple sets of candidate pose relationship solutions; wherein, the pose relationship characterizes the pose transformation relationship between the target display surface coordinate system and the robot arm base coordinate system; The candidate pose relation solutions are filtered to obtain reliable candidate solutions; The reliable candidate solutions are optimized to obtain the target pose relationship of the sample vehicle.

2. The method according to claim 1, characterized in that, The observation sample information includes observation features. The pose calculation based on the observation sample information yields multiple sets of candidate pose relationship solutions, including: Based on the observed characteristics, the corresponding observed sample information is evaluated for sample quality to obtain a quality score for the observed sample information. Based on the quality score, the observation sample information is filtered to obtain valid observation sample information; Based on the effective observation sample information, pose calculation is performed to obtain multiple sets of candidate pose relationship solutions.

3. The method according to claim 2, characterized in that, The effective observation sample information includes the end-effector pose of the robotic arm during observation; the observation features include the positional correspondence between the pixel coordinates of the observation features and the three-dimensional coordinates of the target display surface; The pose calculation process based on the effective observation sample information yields multiple sets of candidate pose relationship solutions, including: For each valid observation sample, based on the pixel coordinates of the observed features, the position correspondence, and the preset camera intrinsic parameters, the first pose transformation relationship between the camera coordinate system and the target display surface coordinate system is solved. Based on the pre-calibrated hand-eye relationship, the first pose transformation relationship, and the end-effector pose during observation, the second pose transformation relationship of the target display surface relative to the robot arm base coordinate system is determined. Based on the position correspondence, a target error function is constructed, and the second pose transformation relationship is used as the initial value for iteration. The optimization objective is to minimize the target error function, and a set of candidate pose relationship solutions are obtained.

4. The method according to claim 3, characterized in that, The step involves constructing a target error function based on the position correspondence, using the second pose transformation relationship as the initial value for iteration, and performing iterative optimization with the goal of minimizing the target error function, to obtain a set of candidate pose relationship solutions, including: Based on the type of the observed features and the geometric type of the target display surface, determine the type of the target error function and the target optimization strategy; Construct the target error function based on the type of the target error function and the positional correspondence; According to the target optimization strategy, the second pose transformation relationship is used as the initial value for iteration, and the optimization objective is to minimize the target error function to perform iterative optimization processing to obtain a set of candidate pose relationship solutions.

5. The method according to claim 1, characterized in that, The observation sample information includes observation features. The process of filtering the candidate pose relation solutions to obtain reliable candidate solutions includes: Based on preset filtering dimensions, the candidate pose relationship solutions are filtered to obtain reliable candidate solutions; The filtering dimensions include at least one of the following: Quality assessment index of the observation features corresponding to candidate pose relation solutions; The average reprojection error of the observation features corresponding to the candidate pose relation solution; Consensus score of candidate pose relation solutions; Physical constraints; Global statistical model; The steps for determining the consensus score include: For each candidate pose relation solution, determine the translational deviation and rotational angle deviation between the candidate pose relation solution and other candidate pose relation solutions; The consensus score of a candidate pose relation solution is determined based on the number of other candidate pose relation solutions that satisfy the consistency condition; the consistency condition is that the translation deviation is less than or equal to the translation deviation consistency threshold and the rotation angle deviation is less than or equal to the rotation deviation consistency threshold.

6. The method according to any one of claims 1-5, characterized in that, The optimization of the reliable candidate solutions to obtain the target pose relationship of the sample vehicles includes: According to the preset objective function, the reliable candidate solution is optimized to obtain the target pose relationship of the sample vehicle; wherein, the objective function is to minimize the sum of geometric errors after processing by the robust loss function, the geometric error represents the error of the observation features in the observation sample information corresponding to the reliable candidate solution, and the geometric error is at least one of the following errors: reprojection error, contour alignment error, boundary distance error, and surface model constraint error.

7. A method for calibrating pose relationships, characterized in that, include: For the target test vehicle, obtain the target pose relationship with the sample vehicles corresponding to the target test vehicle; Based on the target pose relationship, determine the reuse verification accuracy index; If the reuse verification accuracy index meets the preset conditions, then the target pose relationship is determined to be the calibration pose relationship of the target test vehicle; If the reuse verification accuracy index does not meet the preset conditions, the target pose relationship is corrected to obtain the calibration pose relationship of the target test vehicle. Wherein, the target pose relationship is the target pose relationship of the sample vehicle as described in any one of claims 1-6.

8. The method according to claim 7, characterized in that, The reuse verification accuracy index includes at least one of the following: average geometric error, interaction success rate, and average interaction landing point deviation. Based on the target pose relationship, the average geometric error is determined, including: Acquire verification observation sample information of the display surface of the target test vehicle observed by the robotic arm in various preset verification observation poses; For each verification observation sample, the geometric error of the verification observation feature is determined based on the relationship between the verification observation feature and the target pose; wherein the geometric error is at least one of reprojection error, contour alignment error, boundary distance error, and pose transformation deviation. The average geometric error is determined based on the geometric error of each verification observation feature; Based on the target pose relationship, determine the interaction success rate and / or average interaction landing point deviation, including: Based on the target pose relationship, determine the execution coordinates of the preset verification interaction target in the robot arm base coordinate system; The robotic arm's end effector is controlled to move to the execution coordinates to perform a preset interactive action, and the interactive response result is obtained. Based on the interaction response results, determine the interaction success rate and / or the average interaction landing point deviation.

9. The method according to claim 7, characterized in that, The step of correcting the target pose relationship to obtain the calibration pose relationship of the target test vehicle includes: The preset initial perturbation vector is substituted into the preset optimization objective function for iterative optimization to obtain the optimal perturbation vector; wherein, the perturbation vector represents the translation deviation and rotation angle deviation; the optimization objective function represents minimizing the sum of the geometric error, interaction error, and perturbation vector magnitude penalty term corresponding to the pose relationship after correction based on the perturbation vector; The calibration pose relationship of the target test vehicle is obtained based on the optimal perturbation vector and the target pose relationship.

10. An electronic device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation of the pose relationship calibration method as described in any one of claims 1-9.

Images