Visual sign board pose estimation interlocking method, system and equipment for bucket arm vehicle and medium

Abstract

The invention relates to a method, a system, equipment and a medium for estimating and interlocking the pose of a visual sign board of a bucket arm truck, and the method comprises the steps: obtaining preprocessed camera image data, carrying out the sign detection of the preprocessed camera image data, obtaining the angular point coordinates of a multi-scale redundant coding visual sign board, and generating a corresponding observation set; performing pose solution based on the observation set to obtain a candidate pose solution, performing rollback updating on the candidate pose solution when joint solution fails or observation is missing, and reconstructing the observation set; calculating a quality index according to the candidate pose solution and the observation set, generating a corresponding confidence coefficient, and carrying out time domain stabilization fusion on the candidate pose solution based on the confidence coefficient to obtain pose output; and executing a security policy according to the pose output and the confidence to generate a corresponding control instruction. The invention has the effect of improving the interlocking reliability.

Description

Technical Field

[0001] This invention belongs to the technical field of safety control for high-altitude operations of bucket trucks, and in particular relates to a method, system, equipment and medium for estimating the pose of visual signboards on bucket trucks. Background Technology

[0002] Currently, boom lifts are widely used in high-altitude operations such as live-line maintenance, municipal maintenance, and emergency repairs. During operation, it is necessary to obtain the spatial pose information of the bucket end in real time under constraints such as complex lighting, shading interference, and vibration disturbance, and use the pose information for operation motion control and safety risk management to meet the requirements of operation accuracy and safety.

[0003] Existing methods for acquiring the pose of boom trucks typically rely on the detection and pose calculation of landmarks in camera images to output pose results, which are then directly used for safety judgment and control command generation. During operation, these methods are easily affected by missing observations, calculation failures, and pose jitter, resulting in insufficient stability of pose output and causing instability in the triggering of safety strategies. Summary of the Invention

[0004] The purpose of this invention is to provide a method, system, device and medium for estimating the pose of a visual signboard on a boom truck, so as to solve the technical problem of unstable safety interlock control caused by pose output fluctuations in existing methods.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a method for interlocking the pose estimation of a visual signboard on a boom lift vehicle, the method comprising: Acquire preprocessed camera image data, perform sign detection on the preprocessed camera image data, obtain the corner coordinates of the multi-scale redundant coded visual sign board, and generate the corresponding observation set; Based on the observation set, perform pose calculation to obtain candidate pose solutions. When the joint calculation fails or the observation is missing, perform a rollback update on the candidate pose solutions to reconstruct the observation set. The quality index is calculated based on the candidate pose solution and the observation set, the corresponding confidence score is generated, and the candidate pose solution is subjected to temporal stabilization fusion based on the confidence score to obtain the pose output. Based on the pose output and the confidence level, a safety policy is executed to generate corresponding control commands.

[0006] By adopting the above technical solutions, preprocessed camera image data is acquired and marker detection is performed to obtain the corner coordinates of the visual marker board and generate an observation set. This forms a stable observation basis for pose estimation, thereby improving the availability and consistency of pose calculation input data. By performing pose calculation based on the observation set to obtain candidate pose solutions, and by backtracking and updating the candidate pose solutions and reconstructing the observation set when joint calculation fails or observations are missing, continuous output of pose estimation can be maintained under abnormal observation conditions, thereby reducing the risk of pose interruption during operation. By calculating quality indicators based on candidate pose solutions and the observation set to generate confidence scores and performing temporal stabilization fusion on candidate pose solutions to obtain pose output, pose fluctuations caused by noise and jitter can be suppressed, thereby improving the stability and reliability of pose output. By executing safety strategies and generating control commands based on pose output and confidence scores, control actions can be matched with pose reliability, thereby improving the safety interlock reliability of the boom truck operation process.

[0007] In one example, the present invention can be further configured as follows: performing sign detection on the preprocessed camera image data to obtain the corner coordinates of a multi-scale redundant coded visual sign board and generate a corresponding observation set includes: In the preprocessed camera image data, identify the image region containing the multi-scale redundant coded visual signboard; Identify the multi-scale redundant coded visual signboard within the image area and extract the initial corner coordinates of the visual signboard; The initial corner coordinates are refined to obtain the final corner coordinates; Based on the corner coordinates, observation data containing the association information between the corner coordinates and the markers is constructed, and the observation data is summarized to generate the observation set.

[0008] By adopting the above technical solution, by determining the image region containing the visual signboard in the camera image data and identifying the visual signboard within the image region to extract the initial corner coordinates, the interference from non-target areas can be reduced and the accuracy of corner extraction can be improved, thereby enhancing the effectiveness of observation set construction. By refining the initial corner coordinates to obtain the corner coordinates and constructing observation data containing the association information between the corner coordinates and the signboard to generate the observation set, the corner positioning accuracy can be enhanced and the observation data structure can be improved, thereby improving the accuracy and robustness of subsequent pose calculation.

[0009] In one example, the present invention can be further configured as follows: The step of performing pose calculation based on the observation set to obtain candidate pose solutions, and then performing a rollback update on the candidate pose solutions to reconstruct the observation set when the joint calculation fails or observations are missing, includes: The joint solution conditions for pose calculation are determined based on the effective observation data in the observation set. If the joint solution conditions meet the preset conditions, joint solution is performed based on the observation set to obtain a first candidate pose solution, and the first candidate pose solution is used as the candidate pose solution. If the joint solution conditions are not met or the joint solution fails, a non-joint solution is performed based on the observation set to obtain a second candidate pose solution, and the candidate pose solution is updated by backtracking based on the second candidate pose solution. In the case of missing observations, the preprocessed camera image data is re-examined based on the candidate pose solution after rollback to update the corner coordinates, generate updated observation data, and reconstruct the observation set based on the updated observation data.

[0010] By adopting the above technical solutions, the joint solution conditions for pose estimation are determined based on the effective observation data in the observation set. When the conditions are met, joint solution is performed to obtain the first candidate pose solution. This fully utilizes multi-marker observation information to improve the accuracy of pose estimation, thereby improving the stability and reliability of the candidate pose solution. When the joint solution conditions are not met or the joint solution fails, a non-joint solution is performed to obtain the second candidate pose solution and the candidate pose solution is updated by rollback. This maintains the pose calculation closed loop in the case of degradation, thereby reducing the impact of pose estimation failure on job control. When observations are missing, re-examination is performed based on the updated candidate pose solution to update the corner coordinates and generate updated observation data to reconstruct the observation set. This quickly supplements missing observations and restores the solvable state, thereby improving the continuity of pose estimation in the case of occlusion or missing marks.

[0011] In one example, the present invention can be further configured as follows: reconstructing the observation set based on the candidate pose solution after backtracking in the case of missing observations includes: Obtain historical candidate pose solutions, and determine the predicted pose based on the historical candidate pose solutions; The re-detection region is determined based on the predicted pose, and the preprocessed camera image data is marked for re-detection within the re-detection region. The corner coordinates of the visual signboard are updated based on the sign re-detection results, and updated observation data is generated. The observation set is reconstructed based on the updated observation data, and the candidate pose solution is updated based on the reconstructed observation set.

[0012] By adopting the above technical solutions, prior constraints on the current pose state can be formed by obtaining historical candidate pose solutions and determining the predicted pose, thereby improving the re-detection direction in scenarios with missing observations. By determining the re-detection region based on the predicted pose and performing flag re-detection within the re-detection region, false detections and computational overhead caused by full-image search can be reduced, thereby improving re-detection efficiency and success rate. By updating corner coordinates based on the re-detection results to generate updated observation data and reconstruct the observation set, the integrity of the observation set can be restored and the temporal consistency of the observation data can be improved, thereby supporting the stable updating of candidate pose solutions in subsequent pose calculations.

[0013] In one example, the present invention can be further configured as follows: calculating a quality index based on the candidate pose solution and the observation set, generating a corresponding confidence score, and performing temporal stabilization fusion on the candidate pose solution based on the confidence score to obtain a pose output, including: The reprojection error is calculated based on the candidate pose solution and the observation set to obtain the first quality index; The inlier rate is determined based on the matching consistency of the observation set, thus obtaining the second quality index; Based on the visual geometric relationships of the observation set, a visual geometric score is determined to obtain a third quality index; The first quality indicator, the second quality indicator, and the third quality indicator are normalized and fused to generate the confidence score. The candidate pose solutions are fused based on the confidence level, and the fusion result is then temporally stabilized to obtain the pose output.

[0014] By adopting the above technical solutions, the first quality index is obtained by calculating the reprojection error based on the candidate pose solution and the observation set, which can quantify the degree of fit between the pose solution and the observation data, thereby improving the objectivity of the confidence assessment. The second quality index is obtained by determining the inlier rate based on the matching consistency of the observation set, which can characterize the effective proportion and consistency level of the observation data, thereby improving the ability to identify abnormal observations. The third quality index is obtained by determining the visual geometry score based on the visual geometric relationship of the observation set, which can reflect the impact of observation conditions on the observability of the pose solution, thereby enhancing the adaptability to complex viewpoints and occluded scenes. By normalizing and fusing multiple quality indices to generate a confidence score, and then performing fusion processing and temporal stabilization processing on the candidate pose solution based on the confidence score to obtain the pose output, the pose output can be adaptively adjusted according to the observation quality, thereby improving the stability and reliability of the pose output.

[0015] In one example, the present invention can be further configured such that: the temporal stabilization processing of the fusion result to obtain the pose output includes: The smoothing intensity of the pose output is determined based on the confidence level, and smoothing parameters are generated based on the smoothing intensity. Spherical interpolation smoothing is performed on the rotated portion of the fusion result according to the smoothing parameters to obtain a rotationally stable result; A moving average process is performed on the translation portion of the fusion result based on the smoothing parameters to obtain a translationally stable result; The pose output is generated based on the rotational stabilization result and the translational stabilization result.

[0016] By adopting the above technical solutions, and by determining the smoothing intensity of the pose output based on confidence level and generating smoothing parameters, the degree of stabilization and observation reliability can be dynamically matched, thus balancing response speed and output stability. By performing spherical interpolation smoothing on the rotational part of the fusion result according to the smoothing parameters to obtain rotational stabilization results, rotational jitter can be reduced and attitude changes can be maintained continuously, thereby improving the controllability of attitude output. By performing moving average processing on the translational part of the fusion result according to the smoothing parameters to obtain translational stabilization results, translational noise can be suppressed and jumps can be reduced, thereby improving the smoothness of position output. By generating pose output based on rotational stabilization results and translational stabilization results, a unified and stable pose result can be formed for subsequent interlocking control, thereby improving the reliability of control command generation.

[0017] In one example, the present invention can be further configured as follows: the step of executing a safety policy based on the pose output and the confidence level to generate corresponding control instructions includes: The confidence level is compared with a preset confidence threshold to determine the safety status; When the safety state is normal, control instructions that allow operation are generated; When the safety state is a speed-limited and amplitude-limited state, the speed limit and amplitude limit are generated based on the pose output and written into the control command; When the safety state is the interlocked state, an audible and visual alarm command and an interlock command are generated and written into the control command; A hold time window is set for the safety state, and a control command corresponding to the safety state is output within the hold time window.

[0018] By adopting the above technical solutions, the confidence level can be compared with a preset confidence threshold to determine the safety state, thereby transforming the pose reliability into an executable safety judgment basis, thus improving the interpretability of safety policy triggering; by generating control instructions that allow operation when the safety state is normal, the continuity of operation can be maintained when the pose is reliable, thereby improving work efficiency; by generating speed limit and amplitude limit quantities based on the pose output and writing them into control instructions when the safety state is speed and amplitude limit state, the amplitude and speed of the action can be constrained when reliability decreases, thereby reducing the probability of risky actions; by generating audible and visual alarm instructions and interlocking instructions and writing them into control instructions when the safety state is interlocked, alarms and interlocking protection can be triggered in a timely manner under high-risk conditions, thereby improving the safety of personnel and equipment; by setting a hold time window for the safety state and outputting control instructions corresponding to the safety state within the hold time window, the control output jitter caused by frequent switching of the safety state can be avoided, thereby improving the stability and consistency of interlocking control.

[0019] In a second aspect, the present invention provides a visual signboard pose estimation interlocking system for a boom lift truck, the system comprising: The image acquisition module is used to acquire preprocessed camera image data, perform sign detection on the preprocessed camera image data, obtain the corner coordinates of the multi-scale redundant coded visual sign board, and generate the corresponding observation set. The pose calculation module is used to perform pose calculation based on the observation set to obtain candidate pose solutions, and to back up and update the candidate pose solutions when the joint calculation fails or the observation is missing, and to reconstruct the observation set. The confidence fusion module is used to calculate a quality index based on the candidate pose solution and the observation set, generate a corresponding confidence score, and perform temporal stabilization fusion on the candidate pose solution based on the confidence score to obtain the pose output. The interlocking control module is used to execute a safety strategy based on the pose output and the confidence level to generate corresponding control commands.

[0020] By adopting the above technical solutions, preprocessed camera image data is acquired and marker detection is performed to obtain the corner coordinates of the visual marker board and generate an observation set. This forms a stable observation basis for pose estimation, thereby improving the availability and consistency of pose calculation input data. By performing pose calculation based on the observation set to obtain candidate pose solutions, and by backtracking and updating the candidate pose solutions and reconstructing the observation set when joint calculation fails or observations are missing, continuous output of pose estimation can be maintained under abnormal observation conditions, thereby reducing the risk of pose interruption during operation. By calculating quality indicators based on candidate pose solutions and the observation set to generate confidence scores and performing temporal stabilization fusion on candidate pose solutions to obtain pose output, pose fluctuations caused by noise and jitter can be suppressed, thereby improving the stability and reliability of pose output. By executing safety strategies and generating control commands based on pose output and confidence scores, control actions can be matched with pose reliability, thereby improving the safety interlock reliability of the boom truck operation process.

[0021] In a third aspect, the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the aforementioned method for estimating and interlocking the pose of a visual signboard on a boom lift.

[0022] In a fourth aspect, the present invention provides a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the aforementioned method for interlocking the pose estimation of a visual signboard on a boom lift vehicle. Attached Figure Description

[0023] The accompanying drawings, which form part of this specification, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart of a method for estimating and interlocking the pose of a visual signboard on a boom truck, as described in an embodiment of the present invention. Figure 2 This is a structural block diagram of the visual signboard pose estimation interlocking system for a boom truck according to an embodiment of the present invention; Figure 3 This is a structural block diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0024] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0025] The following detailed description is exemplary and intended to provide further detailed explanation of the invention. Unless otherwise specified, all technical terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this invention is for describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention.

[0026] Example 1 like Figure 1 As shown, this invention discloses a method for estimating the pose of a visual signboard on a boom truck, specifically including the following steps: S10: Acquire preprocessed camera image data, perform sign detection on the preprocessed camera image data, obtain the corner coordinates of the multi-scale redundant coded visual sign board, and generate the corresponding observation set.

[0027] Specifically, during operation, the bottom camera acquires image frames at a preset sampling frequency and obtains preprocessed camera image data. In the camera image data, the visible signs of the multi-scale redundant coded visual signboard are detected and decoded to output the corner coordinates and sign identification information corresponding to each visible sign. The corner coordinates and sign identification information output in the same frame are then timestamped and summarized to form an observation set.

[0028] S20: Perform pose calculation based on the observation set to obtain candidate pose solutions. When the joint calculation fails or the observation is missing, backtrack and update the candidate pose solutions to reconstruct the observation set.

[0029] Specifically, the input data structure for pose calculation is established based on the corner coordinates and marker information in the observation set, and an observation constraint set for pose calculation is generated. When the joint calculation conditions are met, joint pose calculation is performed on multiple visible markers to obtain candidate pose solutions. When the joint calculation fails or the observation is missing, the available observations are used to perform a rollback update to obtain the updated candidate pose solutions. The updated candidate pose solutions are then used as prior guidance to complete the corner coordinates in order to reconstruct the observation set.

[0030] S30: Calculate the quality index based on the candidate pose solution and the observation set, generate the corresponding confidence score, and perform temporal stabilization fusion on the candidate pose solution based on the confidence score to obtain the pose output.

[0031] Specifically, based on the candidate pose solutions and the observation set, quality indicators that can characterize the fitting residuals, matching consistency and visual geometric relationships are calculated, and the quality indicators are mapped to generate confidence scores Q as a quantitative representation of the pose credibility. Furthermore, the candidate pose solutions are fused based on the confidence scores Q and time-domain stabilization is performed to output the pose output. The pose output and confidence scores Q are reused when generating subsequent control commands.

[0032] S40: Execute safety policies based on pose output and confidence level to generate corresponding control commands.

[0033] Specifically, the pose output and confidence level Q are used as inputs to the safety strategy and the safety status is determined. Under different safety statuses, corresponding control command types and their carried control parameters are generated, and the control commands are output to the boom truck controller to execute control actions such as allowing operation, limiting speed and amplitude, or interlocking alarms.

[0034] In one embodiment, step S10, namely, performing sign detection on the preprocessed camera image data to obtain the corner coordinates of the multi-scale redundant coded visual sign board and generate the corresponding observation set, includes: S11: Identify the image region containing the multi-scale redundant coded visual sign in the preprocessed camera image data.

[0035] Specifically, coarse localization is performed on the candidate response regions of the signboard in the preprocessed camera image data to obtain several candidate regions. The target candidate regions that meet the multi-scale redundancy coding features are selected by combining the consistency of the sign texture and the scale matching relationship of each candidate region in the current frame. Thus, the selected target candidate regions are determined as the image regions containing the multi-scale redundancy coding visual signboard for subsequent sign recognition and corner extraction.

[0036] S12: Identify multi-scale redundant coded visual signs within the image region and extract the initial corner coordinates of the visual signs.

[0037] Specifically, matching and decoding of the marker patterns at each scale are performed within the image area to obtain the marker identifier and pixel boundary information of each visible marker. Based on the pixel boundary information, the four corner points of each visible marker are located to extract the corresponding initial corner point coordinates. The initial corner point coordinates are used to characterize the distribution of corner point pixel coordinates of each visible marker in the image plane.

[0038] S13: Refine the initial corner coordinates to obtain the final corner coordinates.

[0039] Specifically, a sub-pixel-level corner optimization target is established for the neighborhood of the initial corner coordinates and iterative updates are performed to obtain the refined corner position. During the iterative update process, the refined result is quality filtered based on the gray-level gradient consistency and convergence residual of the corner neighborhood, and the refined result that passes the quality filter is output as the corner coordinate to be used to construct the observation set.

[0040] S14: Construct observation data containing the association information between corner coordinates and markers based on corner coordinates, and summarize the observation data to generate an observation set.

[0041] Specifically, the corner coordinates corresponding to each visible marker are associated with the marker identifier, scale category, corner order information and image timestamp of the visible marker to construct an observation data entry. All observation data entries in the same frame are summarized to form an observation data set containing corner coordinates and marker association information, thereby determining the observation data set as the observation set for pose calculation.

[0042] In one embodiment, step S20, which involves performing pose calculation based on the observation set to obtain candidate pose solutions, and then backtracking and updating the candidate pose solutions to reconstruct the observation set when the joint calculation fails or observations are missing, includes: S21: Determine the joint solution conditions for pose calculation based on the effective observation data in the observation set.

[0043] Specifically, valid observation data are extracted from the observation set, and the number of visible markers, the distribution span of corner points, and the proportion of valid observations are statistically analyzed to form joint solution condition parameters. The joint solution condition parameters are then compared with preset conditions to determine whether the observation set meets the observability and consistency requirements of multi-marker joint solution, thereby obtaining the joint solution condition judgment result used to select joint solution or non-joint solution.

[0044] S22: If the joint solution conditions meet the preset conditions, perform joint solution based on the observation set to obtain the first candidate pose solution, and use the first candidate pose solution as the candidate pose solution.

[0045] Specifically, under the premise that the joint solution conditions meet the preset conditions, the corner coordinates of the observation set are denoted as ui, and the corresponding three-dimensional markers are denoted as Xi. Given the camera intrinsic parameters K, the rotation R and translation t are solved to minimize the reprojection error. The minimum value is obtained, where π represents the projection function of the 3D point onto the image plane. In the solution process, positive depth constraints and ground plane normal constraints are applied to the solution space to suppress mirror solutions and abnormal flips, thereby obtaining the first candidate pose solution and using the first candidate pose solution as the candidate pose solution.

[0046] S23: If the joint solution conditions are not met or the joint solution fails, perform a non-joint solution based on the observation set to obtain a second candidate pose solution, and then backtrack and update the candidate pose solution based on the second candidate pose solution.

[0047] Specifically, when the joint solution conditions are not met or the joint solution fails, the corner coordinates and the set of three-dimensional points of the single marker with the largest visible area are selected from the observation set to construct the non-joint solution input. Based on the non-joint solution input, the single marker pose solution is calculated to obtain the second candidate pose solution. Then, the difference between the second candidate pose solution and the candidate pose solution of the previous frame is constrained, and the current candidate pose solution is updated accordingly to obtain the updated candidate pose solution.

[0048] S24: In the case of missing observations, perform a re-detection on the preprocessed camera image data based on the candidate pose solution after backtracking to update the corner coordinates, generate updated observation data, and reconstruct the observation set based on the updated observation data.

[0049] Specifically, in the case of missing observations, the candidate pose solution after rollback and update is used as the pose prior and the expected imaging range of the current frame flag is determined accordingly. Within the expected imaging range, guided flag re-detection is performed on the preprocessed camera image data to supplement the missing corner coordinates. The supplemented corner coordinates and corresponding flag identification information are used to generate updated observation data. The updated observation data and the still valid observation data are then summarized to reconstruct the observation set.

[0050] In one embodiment, step S24, i.e., reconstructing the observation set based on the candidate pose solution after rollback in the case of missing observations, includes: S241: Obtain historical candidate pose solutions and determine the predicted pose based on the historical candidate pose solutions.

[0051] Specifically, the candidate pose solution of the previous frame and the pose change of its adjacent frames are extracted from the historical candidate pose solutions. The pose change is extrapolated based on the time interval between adjacent frames to obtain the predicted pose. The predicted pose is used to characterize the prediction result of the camera pose in the current frame and serves as the input for subsequent re-detection region determination.

[0052] S242: Determine the re-detection region based on the predicted pose, and perform marker re-detection on the preprocessed camera image data within the re-detection region.

[0053] Specifically, the predicted pose is substituted into the projection relationship to calculate the predicted projection position of the three-dimensional corner point of the marker in the image plane, and a local window is constructed with the predicted projection position as the center as the re-detection region. Fast matching and decoding are performed in the re-detection region to obtain the marker re-detection result, thereby improving the re-detection success rate in the case of missing observations while keeping the computational load under control.

[0054] S243: Update the corner coordinates of the visual signboard based on the sign re-inspection results, and generate updated observation data.

[0055] Specifically, the corner coordinates of the visual signboard are updated based on the corner response positions output by the sign re-detection results. The updated corner coordinates are then associated with their corresponding sign identifiers, scale categories, and timestamps to generate updated observation data, which can then be used for subsequent observation set reconstruction and pose updates.

[0056] S244: Reconstruct the observation set based on the updated observation data, and update the candidate pose solutions based on the reconstructed observation set.

[0057] Specifically, the updated observation data and the still valid observation data are merged and deduplicated to form a reconstructed observation set. The reconstructed observation set is then re-input into the pose calculation process to obtain updated candidate pose solutions, thereby completing the closed loop of observation set reconstruction and candidate pose solution update in the scenario of missing observations.

[0058] In one embodiment, step S30, namely calculating a quality index based on the candidate pose solution and the observation set, generating a corresponding confidence score, and performing temporal stabilization fusion on the candidate pose solution based on the confidence score to obtain the pose output, includes: S31: Calculate the reprojection error based on the candidate pose solution and the observation set to obtain the first quality index.

[0059] Specifically, given a rotation R and a translation t for the candidate pose solution, the marker 3D point X in the observation set... i Calculate the predicted pixel position. i =π(K(RX i +t)) and label the corner coordinates of the observation set as u, then calculate the pixel distance e. i =||u i û i || represents the single-point reprojection error and applies to each e i The results are summarized to obtain the first quality index, which is used to characterize the fitting residual level of the candidate pose solution to the observation set.

[0060] S32: Determine the inlier rate based on the matching consistency of the observation set to obtain the second quality index.

[0061] Specifically, a consistency check is performed on the observation set, and the observation data that meets the preset pixel error condition are identified as interior points. The number of interior points is denoted as N. in The total number of observation data is denoted as N. all Calculate the interior point ratio r=N in / N all It serves as a second quality metric to characterize the consistency and robustness of the observation set in the pose solving process.

[0062] S33: Determine the visual geometry score based on the visual geometric relationships of the observation set to obtain the third quality index.

[0063] Specifically, visual geometric relationship features are extracted based on the observation set and a visual geometric score is generated. The visual geometric relationship features include at least the perimeter of the marker pixel, the cosine of the incident angle, the sub-pixel convergence quality of the corner point, and the field of view edge penalty term. The visual geometric relationship features are then converted into a visual geometric score according to a preset mapping rule as a third quality indicator.

[0064] S34: Normalize and merge the first, second, and third quality indicators to generate confidence scores.

[0065] Specifically, Q is obtained by monotonically normalizing the reprojection error E corresponding to the first quality index, the interior point rate r corresponding to the second quality index, and the geometric score g corresponding to the third quality index. e Q i With Q g and according to weight ω e ω i With ω g Performing a linear combination yields the confidence level Q=ω e ×Q e +ω i ×Q i +ω g ×Q g , where ω e +ω i +ω g =1 and Q∈[0,1], Q e Press Perform a mapping, where E0 and σ E This is the error scale parameter used to characterize the attenuation scale of the reprojection error.

[0066] S35: Based on the confidence level, the candidate pose solutions are fused and the fusion results are stabilized in the time domain to obtain the pose output.

[0067] Specifically, when multiple candidate pose solutions are valid at the same time, the confidence level corresponding to each candidate pose solution is used as the fusion weight to perform weighted fusion on the translation component and weighted interpolation on the rotation component to generate the fusion result. The fusion result is then input into the temporal stabilization process to obtain the pose output. The smoothing intensity of the temporal stabilization process is related to the confidence level, depth, and adjacent frame velocity to adapt to the output stability requirements under different observation reliability and motion states.

[0068] In one embodiment, step S35, which involves temporal stabilization of the fusion result to obtain the pose output, includes: S351: Determine the smoothing intensity of the pose output based on the confidence level, and generate smoothing parameters based on the smoothing intensity.

[0069] Specifically, a normalized quantity is generated based on the confidence level Q, the depth Z, and the velocity v of adjacent frames. ,calculate The normalization range is defined by clip(x,0,1)=min(max(x,0),1), where Q1 and Q2 are the first and second confidence thresholds, and V ref With Z refAs a normalized reference value for velocity and depth, further based on Generate smoothing parameters, where α r With α t The smoothing coefficients for rotation and translation are respectively, and α r,min α r,max α t,min α t,max These are the corresponding upper and lower bounds.

[0070] S352: Perform spherical interpolation smoothing on the rotated part of the fusion result according to the smoothing parameters to obtain a rotationally stable result.

[0071] Specifically, the rotation of the previous frame is represented as a quaternion q. {k 1} The rotation of the fusion result is represented as a quaternion. Based on the rotational smoothing coefficient α in the smoothing parameters r Execute q k =SLERP(q {k 1} , ,α r This is used to complete the spherical interpolation smoothing process, where SLERP is a quaternion spherical linear interpolation function used to achieve continuous interpolation in the rotation space, thereby obtaining rotationally stable results.

[0072] S353: Perform a moving average process on the translation portion of the fusion result based on the smoothing parameter to obtain a translationally stable result.

[0073] Specifically, the translation vector of the previous frame is denoted as t{k} 1} and denote the translation vector of the fusion result as Based on the translation smoothing coefficient α in the smoothing parameters t Execute t k =(1 α t )×t {k 1} +α t × This completes the moving average process, thereby obtaining a translation-stabilized result and ensuring that the translation output remains continuous over time.

[0074] S354: Generate pose output based on rotational and translational stabilization results.

[0075] Specifically, the quaternion q corresponding to the rotational stability result k Convert to rotation matrix R k and the rotation matrix R kWith translational stability result t k Combined to generate pose output (R) k ,t k ), and append a corresponding timestamp to the pose output to form a pose output that can be used by subsequent safety strategies.

[0076] In one embodiment, step S40, namely, executing a safety policy based on the pose output and confidence level to generate corresponding control commands, includes: S41: Compare the confidence level with the preset confidence threshold to determine the safe state.

[0077] Specifically, the confidence level Q is compared with the first confidence threshold Q1 and the second confidence threshold Q2 to determine the confidence level interval assignment, and the confidence level interval assignment is mapped to a safety state among normal state, speed limit and amplitude limit state or interlocking state, thereby providing a state selection basis for the generation of control commands.

[0078] S42: When the safety state is normal, generate control instructions that allow operation.

[0079] Specifically, when the safety status is normal, the operation permission flag is written into the control command and the corresponding control status field is configured, so that the output control command can be parsed by the boom truck controller and used to maintain the normal operation status.

[0080] S43: When the safety state is the speed limit and amplitude limit state, generate the speed limit and amplitude limit based on the pose output and write them into the control command.

[0081] Specifically, when the safety state is a speed-limited and amplitude-limited state, the motion constraint corresponding to the current posture is calculated based on the pose output, and the speed limit V is generated by combining the confidence level Q. max With the limit amount, where V max according to Calculate where C is the safety margin, σ(Q) is the mapping function between the confidence level obtained from offline calibration and the upper bound of the pose error, and τ stop The braking hysteresis parameters are set, and the speed limit and amplitude limit are written into the control command for the boom truck controller to execute.

[0082] S44: When the safety state is interlocked, generate audible and visual alarm commands and interlock commands and write them into the control commands.

[0083] Specifically, when the safety state is the interlocked state, the audible and visual alarm command and the interlock command are written into the control command to trigger the audible and visual alarm and execute the interlock control. At the same time, a status bit that represents the reason for the interlock is written to support subsequent reset judgment and operation record.

[0084] S45: Set a hold time window for the safe state, and output the control command corresponding to the safe state within the hold time window.

[0085] Specifically, a holding time window T is set for the safe state, and a continuous determination is made when the confidence level crosses the threshold instead of immediately switching the safe state. The duration is accumulated by the determination results of the confidence interval of consecutive frames, and the safe state is updated only when the duration exceeds the holding time window T. After the safe state is updated, the control command corresponding to the safe state is continuously output until the exit condition is met and the duration exceeds the holding time window T.

[0086] Example 2 like Figure 2 As shown, based on the same inventive concept as the above embodiments, the present invention also provides a bucket truck visual signboard pose estimation interlocking system, comprising: The image acquisition module is used to acquire preprocessed camera image data, perform sign detection on the preprocessed camera image data, obtain the corner coordinates of the multi-scale redundant coded visual sign board, and generate the corresponding observation set. The pose calculation module is used to perform pose calculation based on the observation set to obtain candidate pose solutions. When the joint calculation fails or the observation is missing, the candidate pose solutions are rolled back and updated to reconstruct the observation set. The confidence fusion module is used to calculate the quality index based on the candidate pose solution and the observation set, generate the corresponding confidence score, and perform temporal stabilization fusion on the candidate pose solution based on the confidence score to obtain the pose output. The interlocking control module is used to execute safety strategies based on the pose output and confidence level to generate corresponding control commands.

[0087] Optionally, the image acquisition module includes: The region localization submodule is used to determine the image region containing multi-scale redundant coded visual signs in the preprocessed camera image data. The sign recognition submodule is used to identify multi-scale redundant coded visual signs within an image region and extract the initial corner coordinates of the visual signs. The Corner Refinement Submodule is used to refine the initial corner coordinates to obtain the final corner coordinates. The observation construction submodule is used to construct observation data containing the association information between corner coordinates and markers based on corner coordinates, and to summarize the observation data to generate an observation set.

[0088] Optionally, the pose calculation module includes: The condition determination submodule is used to determine the joint solution conditions for pose calculation based on the valid observation data in the observation set; The joint solution submodule is used to perform joint solution based on the observation set to obtain the first candidate pose solution when the joint solution conditions meet the preset conditions, and to use the first candidate pose solution as the candidate pose solution. The rollback update submodule is used to perform non-joint solution based on the observation set to obtain a second candidate pose solution when the joint solution conditions are not met or the joint solution fails, and to perform rollback update on the candidate pose solution based on the second candidate pose solution. The missing reconstruction submodule is used to perform re-detection on the preprocessed camera image data based on the candidate pose solution after rollback to update the corner coordinates, generate updated observation data, and reconstruct the observation set based on the updated observation data when observations are missing.

[0089] Optional, the missing reconstruction submodule includes: The prediction generation unit is used to obtain historical candidate pose solutions and determine the predicted pose based on the historical candidate pose solutions. The re-detection limiting unit is used to determine the re-detection region based on the predicted pose, and to perform re-detection on the preprocessed camera image data within the re-detection region. The observation update unit is used to update the corner coordinates of the visual signboard based on the sign re-detection results and generate updated observation data. An integrated update unit is used to reconstruct the observation set based on the updated observation data and update the candidate pose solutions based on the reconstructed observation set.

[0090] Optionally, the confidence fusion module includes: The error calculation submodule is used to calculate the reprojection error based on the candidate pose solution and the observation set to obtain the first quality index. The consistency evaluation submodule is used to determine the inlier rate based on the matching consistency of the observation set, thus obtaining the second quality index; The geometric scoring submodule is used to determine the visual geometric score based on the visual geometric relationships of the observation set, thus obtaining the third quality index; The confidence fusion submodule is used to perform normalization and fusion processing on the first quality indicator, the second quality indicator and the third quality indicator to generate confidence scores. The steady-state fusion submodule is used to fuse candidate pose solutions based on confidence level and perform time-domain stabilization on the fusion result to obtain pose output.

[0091] Optional, the steady-state fusion submodule includes: The parameter generation unit is used to determine the smoothing intensity of the pose output based on the confidence level, and to generate smoothing parameters based on the smoothing intensity. The rotational smoothing unit is used to perform spherical interpolation smoothing on the rotational part of the fusion result according to the smoothing parameters to obtain a rotationally stable result; The translation smoothing unit is used to perform a moving average process on the translation part of the fusion result according to the smoothing parameters to obtain a translation-stable result; The output generation unit is used to generate pose output based on rotational stabilization results and translational stabilization results.

[0092] Optionally, the interlocking control module includes: The status determination submodule is used to compare the confidence level with a preset confidence threshold to determine the safe status. The normal instruction submodule is used to generate control instructions that allow execution when the safety state is normal. The speed limiting and amplitude limiting submodule is used to generate speed limiting and amplitude limiting values ​​based on pose output and write control commands when the safety state is speed limiting and amplitude limiting state. The interlocking alarm submodule is used to generate audible and visual alarm commands and interlocking commands and write them into control commands when the safety state is in the interlocking state. The state holding submodule is used to set a holding time window for the safe state and output control commands corresponding to the safe state within the holding time window.

[0093] Example 3 like Figure 3 As shown, the present invention also provides an electronic device 100 for implementing a method for interlocking the pose estimation of a visual signboard on a boom truck; The electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on at least one processor 102, and at least one communication bus 104.

[0094] The memory 101 can be used to store the computer program 103. The processor 102 implements the steps of the visual signboard pose estimation interlocking method of the boom truck in Embodiment 1 by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101.

[0095] The memory 101 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created based on the use of the electronic device 100 (such as audio data), etc. In addition, the memory 101 may include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other non-volatile solid-state storage device.

[0096] At least one processor 102 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 102 may be a microprocessor or any conventional processor. Processor 102 is the control center of electronic device 100, connecting various parts of electronic device 100 via various interfaces and lines.

[0097] The memory 101 in the electronic device 100 stores multiple instructions to implement a method for interlocking the pose estimation of a visual signboard on a boom lift vehicle, and the processor 102 can execute multiple instructions to achieve the following: Acquire preprocessed camera image data, perform sign detection on the preprocessed camera image data, obtain the corner coordinates of the multi-scale redundant coded visual sign board, and generate the corresponding observation set; Candidate pose solutions are obtained by performing pose calculation based on the observation set. When the joint calculation fails or the observation is missing, the candidate pose solutions are backed up and updated to reconstruct the observation set. The quality index is calculated based on the candidate pose solutions and the observation set, the corresponding confidence score is generated, and the candidate pose solutions are fused in the time domain based on the confidence score to obtain the pose output. Based on the pose output and confidence level, a safety policy is executed to generate corresponding control commands.

[0098] Example 4 If the modules / units integrated in the electronic device 100 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or system capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, and read-only memory (ROM).

[0099] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0100] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0101] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0102] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0103] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for pose estimation interlocking of visual signboards on a boom truck, characterized in that, The method comprises: acquiring pre-processed camera image data, performing mark detection on the pre-processed camera image data, obtaining corner point coordinates of a multi-scale redundant coded visual mark board, and generating a corresponding observation set; performing pose solving based on the observation set to obtain a candidate pose solution, performing rollback updating on the candidate pose solution when joint solving fails or observation is missing, and reconstructing the observation set; calculating a quality index according to the candidate pose solution and the observation set, generating a corresponding confidence, and performing time domain stable fusion on the candidate pose solution based on the confidence to obtain a pose output; performing a safety strategy based on the pose output and the confidence to generate a corresponding control instruction.

2. The method of claim 1, wherein, The method comprises: determining an image region containing the multi-scale redundant coded visual mark board in the pre-processed camera image data; identifying the multi-scale redundant coded visual mark board in the image region and extracting initial corner point coordinates of the visual mark board; performing corner point refinement on the initial corner point coordinates to obtain corner point coordinates; constructing observation data containing corner point coordinates and mark association information based on the corner point coordinates, and generating the observation set by summarizing the observation data.

3. The method of claim 1, wherein, The method comprises: determining a joint solving condition of pose solving based on valid observation data in the observation set; when the joint solving condition meets a preset condition, performing joint solving based on the observation set to obtain a first candidate pose solution, and taking the first candidate pose solution as the candidate pose solution; when the joint solving condition does not meet or the joint solving fails, performing non-joint solving based on the observation set to obtain a second candidate pose solution, and performing rollback updating on the candidate pose solution based on the second candidate pose solution; when the observation is missing, performing re-detection on the pre-processed camera image data based on the rollback updated candidate pose solution to update the corner point coordinates, generating updated observation data, and reconstructing the observation set based on the updated observation data.

4. The method of claim 3, wherein, The method comprises: acquiring a historical candidate pose solution, and determining a predicted pose according to the historical candidate pose solution; determining a re-detection region according to the predicted pose, and performing mark re-detection on the pre-processed camera image data in the re-detection region; updating the corner point coordinates of the visual mark board based on the mark re-detection result, generating updated observation data; reconstructing the observation set based on the updated observation data, and updating the candidate pose solution based on the reconstructed observation set.

5. The method of claim 1, wherein, The method comprises: calculate a re-projection error based on the candidate pose solution and the observation set, to obtain a first quality indicator; determine an inlier rate according to a matching consistency of the observation set, to obtain a second quality indicator; determine a visible geometry score based on a visible geometric relationship of the observation set, to obtain a third quality indicator; perform a normalized fusion processing on the first quality indicator, the second quality indicator and the third quality indicator, to generate the confidence; perform a fusion processing on the candidate pose solution based on the confidence, and perform a time domain stabilization processing on a fusion result, to obtain the pose output.

6. The method of visual marker board pose estimation interlock for a bucket arm vehicle of claim 5, wherein, The time domain stabilization processing on the fusion result to obtain the pose output comprises: determine a smoothing strength of the pose output based on the confidence, and generate a smoothing parameter according to the smoothing strength; perform a spherical interpolation smoothing processing on a rotation part of the fusion result according to the smoothing parameter, to obtain a rotation stabilization result; perform a sliding average processing on a translation part of the fusion result according to the smoothing parameter, to obtain a translation stabilization result; generate the pose output based on the rotation stabilization result and the translation stabilization result.

7. The method of claim 1, wherein, The execution of a safety policy according to the pose output and the confidence to generate a corresponding control instruction comprises: compare the confidence with a preset confidence threshold, to determine a safety state; when the safety state is a normal state, generate a control instruction allowing operation; when the safety state is a speed limit and amplitude limit state, generate a speed limit amount and an amplitude limit amount based on the pose output and write into the control instruction; when the safety state is an interlocking state, generate an audible and light alarm instruction and an interlocking instruction and write into the control instruction; set a holding time window for the safety state, and output the control instruction corresponding to the safety state within the holding time window.

8. A visual signboard pose estimation interlocking system for a boom lift truck, characterized in that, The system comprises: an image acquisition module, configured to acquire preprocessed camera image data, perform landmark detection on the preprocessed camera image data, obtain corner point coordinates of a multi-scale redundant coded visual marker board and generate a corresponding observation set; a pose solution module, configured to perform pose solution based on the observation set to obtain a candidate pose solution, perform back-up update on the candidate pose solution when joint solution fails or observation is missing, and reconstruct the observation set; a confidence fusion module, configured to calculate quality indicators according to the candidate pose solution and the observation set, generate corresponding confidence, and perform time domain stabilization fusion on the candidate pose solution based on the confidence, to obtain a pose output; an interlocking control module, configured to execute a safety policy according to the pose output and the confidence to generate a corresponding control instruction.

9. An electronic device, comprising: The system comprises a processor and a memory, and the processor is configured to execute a computer program stored in the memory to implement the steps of the method for estimating a pose of a visual marker board of a dump truck according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores at least one instruction, and the at least one instruction is executed by the processor to implement the steps of the method for estimating a pose of a visual marker board of a dump truck according to any one of claims 1 to 7.

Images