Steering vision assisting method, device and equipment of non-polar steering wheel and medium

Abstract

The invention discloses a steering vision assisting method, device and equipment for a non-polar steering wheel and a medium. Comprising the following steps: calibrating a center axis datum line, and calibrating a left limit dip angle and a right limit dip angle based on the center axis datum line; hydraulic rod image data are collected based on a preset RGB industrial camera, and visual feature points of the image data are extracted; calculating a homography matrix based on the visual feature points, and calculating a direction vector of the hydraulic rod based on the homography matrix and the physical coordinates of the hydraulic rod; converting the direction vector into an actual steering angle of a non-polar steering wheel, and determining a limit angle percentage based on the actual steering angle, the left limit inclination angle and the right limit inclination angle; and performing visual rendering and early warning on the display terminal based on the limit angle percentage. According to the scheme, the real-time steering angle of the tire can be accurately obtained, when the steering operation reaches the mechanical limit can be pre-judged, visual rendering and early warning are carried out, and operation safety, economical efficiency and efficiency are improved.

Description

Technical Field

[0001] This application relates to the field of mechanical operation vision assistance technology, and more specifically, to a steering vision assistance method, device, equipment and medium for a non-polarized steering wheel. Background Technology

[0002] In the field of construction machinery, loaders, graders, and wheeled bulldozers are widely used in various construction scenarios, and the precision and safety of their operation are crucial. However, these construction machines are usually equipped with non-polarized steering wheels, which, due to the lack of fixed mechanical limits and angle markings, bring many operational and perceptual challenges to the driver.

[0003] On the one hand, the lack of steering angle perception is a major problem. Drivers cannot accurately obtain the real-time steering angle of the tires, making it difficult to accurately judge the actual direction of the equipment's movement or operation, which can easily lead to operational errors in complex working conditions. On the other hand, the lack of awareness of steering limits is also extremely critical. Drivers lack a clear perception of the remaining steering wheel travel and cannot predict when the steering operation will reach its mechanical limits, which increases the risk of equipment loss of control.

[0004] Currently, traditional mechanical rulers cannot adapt to different machine models and loads, and their fixed calibration makes it difficult to meet diverse needs; single-angle sensors can only provide limit warnings and cannot fully perceive steering angles, and their modification costs are high, making large-scale application difficult. Therefore, existing technologies cannot meet the needs of precise steering control for construction machinery. Summary of the Invention

[0005] This application provides a steering vision assistance method, device, equipment, and medium for a non-polarized steering wheel, to at least solve the technical problem in the related art of accurately perceiving the steering angle and limit position of a non-polarized steering wheel.

[0006] According to one aspect of the embodiments of this application, a steering visual assistance method for a non-polarized steering wheel is provided, comprising:

[0007] Calibrate the central axis reference line, and calibrate the left and right limit tilt angles based on the central axis reference line;

[0008] Based on a preset RGB industrial camera, image data of hydraulic rods are acquired, and visual feature points of the image data are extracted.

[0009] The homography matrix is ​​calculated based on the visual feature points, and the direction vector of the hydraulic rod is calculated based on the homography matrix and the physical coordinates of the hydraulic rod.

[0010] The direction vector is converted into the actual steering angle of the non-polarized steering wheel, and the percentage of the limit angle is determined based on the actual steering angle and the left limit tilt angle and the right limit tilt angle.

[0011] Visual rendering and early warning are performed on the display terminal based on the aforementioned extreme angle percentage.

[0012] In one embodiment, after acquiring hydraulic rod image data based on a preset RGB industrial camera, the method further includes:

[0013] Calculate the environmental quality coefficient based on the image data;

[0014] When the environmental quality coefficient is less than a preset threshold, the steering column rotation angle collected by the preset Hall angle sensor is obtained;

[0015] Based on the steering column rotation angle, construct a rotation matrix;

[0016] The direction vector of the hydraulic rod is corrected based on the rotation matrix.

[0017] In one implementation, calculating the environmental quality coefficient based on the image data includes:

[0018] Calculate the variance of the Laplacian operator for a grayscale image;

[0019] Calculate the standard deviation of the image's grayscale values;

[0020] The variance and standard deviation of the Laplace operator are normalized and weighted to obtain the environmental quality coefficient.

[0021] In one implementation, visual rendering and early warning are performed on the display terminal based on the extreme angle percentage, including:

[0022] Based on the extreme angle percentage and the preset warning thresholds for different levels, the corresponding warning level is determined;

[0023] Obtain the warning strategy corresponding to the warning level;

[0024] Based on the aforementioned early warning strategy, visual rendering and device control are performed on the display terminal.

[0025] In one implementation, it further includes:

[0026] Automatically read the model identification code in the electronic control unit of the equipment to determine the machine type, and adaptively adjust the warning thresholds at different levels based on different machine types;

[0027] The working environment is determined based on the environmental quality coefficient, and the warning thresholds at different levels are adaptively adjusted based on different working environments.

[0028] Based on the equipment's operating mode, the warning thresholds and warning strategies at different levels are adaptively adjusted.

[0029] In one implementation, it further includes:

[0030] Acquire acceleration and temperature data collected by the system's sensors;

[0031] Based on the acceleration and temperature data, determine whether the preset recalibration conditions are met;

[0032] If the recalibration conditions are met, the central axis reference line, left limit tilt angle, and right limit tilt angle are recalibrated based on the hydraulic rod position under the preset frame number stable state.

[0033] In one implementation, after acquiring hydraulic rod image data based on a preset RGB industrial camera, the method further includes:

[0034] In foggy environments, a dark channel prior dehazing algorithm is used to correct the image, resulting in preprocessed image data.

[0035] In a high-dust environment, morphological filtering algorithms are used to remove image noise and obtain preprocessed image data.

[0036] According to another aspect of the embodiments of this application, a steering vision assistance device with a non-polarized steering wheel is provided, comprising:

[0037] The calibration module is used to calibrate the central axis reference line and calibrate the left and right limit tilt angles based on the central axis reference line.

[0038] The image acquisition module is used to acquire hydraulic rod image data based on a preset RGB industrial camera and extract visual feature points from the image data;

[0039] The calculation module is used to calculate the homography matrix based on the visual feature points, and to calculate the direction vector of the hydraulic rod based on the homography matrix and the physical coordinates of the hydraulic rod.

[0040] The steering determination module is used to convert the direction vector into the actual steering angle of the non-polarized steering wheel, and determine the percentage of the limit angle based on the actual steering angle and the left limit tilt angle and the right limit tilt angle;

[0041] The visual rendering and early warning module is used to perform visual rendering and early warning on the display terminal based on the extreme angle percentage.

[0042] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described steering vision assistance method for a non-polarized steering wheel through the computer program.

[0043] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, and the computer program is configured to execute the above-described steering vision assistance method for a non-polarized steering wheel when it is run.

[0044] The technical solutions provided in this application embodiment may include the following beneficial effects:

[0045] This application relates to the field of engineering machinery operation assistance technology, and in particular to a steering polarity visual assistance and warning method based on a non-polarized steering wheel. The system can accurately acquire the real-time steering angle of the tires, solving the problem of lack of steering angle perception in traditional non-polarized steering wheel operation and improving operational accuracy. Furthermore, by predicting when the steering operation will reach the mechanical limits, it effectively avoids safety hazards caused by oversteering, significantly improving operational safety. Simultaneously, through visual rendering and warnings, the system provides operators with intuitive operational guidance, reducing the time spent on delicate operations and improving work efficiency.

[0046] Furthermore, through an environmental adaptive mechanism, feature point loss is effectively avoided in harsh environments such as heavy fog and high dust levels, significantly improving the system's environmental adaptability. Simultaneously, the use of multi-sensor fusion technology ensures reliable calculation of the steering angle even in the event of visual failure, greatly enhancing data accuracy.

[0047] Furthermore, through dynamic calibration technology, this solution can adapt to different machine models and loads, improving its adaptability. Secondly, the introduction of a three-level early warning and dynamic threshold mechanism significantly improves the success rate of emergency obstacle avoidance and enhances operational safety. Attached Figure Description

[0048] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0049] Figure 1 This is a flowchart of a steering visual assistance method for a non-polarized steering wheel according to an embodiment of this application;

[0050] Figure 2 This is a schematic diagram of a steering visual assistance method for a non-polar steering wheel according to an embodiment of this application;

[0051] Figure 3 This is a method for determining a direction vector based on image data and sensor data according to an embodiment of this application;

[0052] Figure 4 This is a schematic diagram of a visual warning interface according to an embodiment of this application;

[0053] Figure 5 This is a visual warning interface effect diagram according to an embodiment of this application;

[0054] Figure 6 This is a schematic diagram of a steering vision assistance device for a non-polar steering wheel according to an embodiment of this application;

[0055] Figure 7 This is a schematic diagram of the structure of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0056] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0057] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0058] This application aims to solve the problems of blind spots in steering angle perception and lack of limit warning in the operation of construction machinery with non-polar steering wheels, and to provide a steering polar vision assistance and warning system that can dynamically adapt to different working conditions, integrate multiple sensors, and has a graded warning function, thereby improving operational safety, economy and efficiency.

[0059] The technical terms appearing in the embodiments of this application are explained below:

[0060] Non-polar steering wheel: A steering device without mechanical limit markings that can rotate continuously 360°, suitable for equipment such as loaders that require precise control of steering limits.

[0061] Neutral position: The steering wheel is not turned (steering angle 0°), and the center line of the bucket hydraulic rod coincides with the central axis baseline.

[0062] RGB industrial camera: An industrial-grade imaging device that can capture color images. "RGB" means that it can sense and collect light of the three basic colors of red, green and blue respectively, and present a color image consistent with the real scene by combining these three colors, similar to how the human eye perceives color.

[0063] Perspective projection model: A mathematical model that maps physical points (X,Y,Z) in the three-dimensional world coordinate system to the two-dimensional image coordinate system (u,v) through the camera intrinsic parameter matrix (K) and extrinsic parameter matrix ([R t]).

[0064] The following is a detailed description of the steering vision assistance method for a non-polarized steering wheel according to an embodiment of this application, with reference to the accompanying drawings. Figure 1 As shown, the method mainly includes the following steps:

[0065] S101 calibrates the central axis baseline, and calibrates the left and right limit tilt angles based on the central axis baseline.

[0066] In one implementation, an RGB industrial camera is first installed. Optionally, the camera resolution is ≥720P (1280×720 pixels), frame rate is ≥25fps, protection rating is IP67, and operating temperature is -40℃~85℃. When the equipment is in a neutral position, the camera can clearly capture a complete image of the bucket hydraulic rod. The installation position must avoid obstruction from the cab, and the lens should face the plane of hydraulic rod movement.

[0067] It also includes the installation of a Hall angle sensor with an angle measurement error of ≤1%, an IP69 protection rating, a stainless steel or anodized aluminum housing, and is coaxially fixed with the steering column. The mechanical structure ensures synchronous rotation angle, and the wiring is connected to the device's CAN bus to complete the initial zero-position calibration.

[0068] Furthermore, after the device is powered on, the system automatically performs initialization, loading the camera intrinsic parameter matrix (K) and distortion coefficients (D).

[0069] The operator moves the equipment to the neutral position (steering wheel 0°, bucket hydraulic rod naturally drooping), the system triggers the centerline baseline calibration, the camera acquires an image of the hydraulic rod, and the centerline pixel coordinates are extracted using an edge detection algorithm. The horizontal offset Δx between the centerline and the horizontal centerline of the image is calculated (initial value 0, dynamically updated if the camera shifts: Δx = current centerline x-coordinate - initial x-coordinate).

[0070] Furthermore, the left and right limit tilt angles are calibrated based on the central axis baseline. With the equipment steering wheel fully turned to the left, the angle αL between the centerline of the bucket hydraulic rod and the central axis baseline is recorded, thus obtaining the left limit tilt angle. With the equipment steering wheel fully turned to the right, the angle αR between the centerline of the bucket hydraulic rod and the central axis baseline is recorded, thus obtaining the right limit tilt angle. Auxiliary lines for the left and right limits are also obtained.

[0071] The S102 acquires hydraulic rod image data based on a preset RGB industrial camera and extracts visual feature points from the image data.

[0072] During system operation, an RGB camera acquires image data of the hydraulic rod at 25fps. After image capture, image processing techniques, such as edge detection and feature point extraction algorithms, are used to identify and extract visual feature points from the images. These feature points typically include the edges, corners, or other prominent visual markers of the hydraulic rod, representing its position and orientation in the image. The extracted feature points are then used to calculate the homography matrix to monitor and analyze the motion state of the hydraulic rod.

[0073] S103 calculates the homography matrix based on visual feature points, and then calculates the direction vector of the hydraulic rod based on the homography matrix and the physical coordinates of the hydraulic rod.

[0074] This application, based on a perspective projection model, maps the physical position (Pw(X,Y,Z)) of the bucket hydraulic rod in the three-dimensional world coordinate system to the two-dimensional image coordinate system (Pi(u,v)), as shown in the following formula:

[0075]

[0076] Where u,v are pixel coordinates in the image coordinate system, and K is the camera intrinsic parameter matrix. (Includes focal lengths fx, fy, and principal point coordinates cx, cy); [Rt] is the extrinsic parameter matrix (R is a 3*3 rotation matrix, t is a 3*1 translation vector), used to describe the relative pose of the camera and the world coordinate system; Z is the depth value of the feature point in the world coordinate system, ensuring the linear correspondence between the two-dimensional image coordinates and the three-dimensional physical position.

[0077] The homography matrix H is a special case of the perspective projection model, used to describe the mapping relationship between the world coordinate system and the image coordinate system for the same plane (such as the plane containing the hydraulic rod). The homography matrix H is used to establish the mapping relationship between the plane containing the bucket hydraulic rod (world coordinate system) and the image plane (image coordinate system). Through H, the physical coordinates of the hydraulic rod can be mapped from the world coordinate system to the image coordinate system, thereby achieving accurate monitoring of the hydraulic rod's orientation. The perspective projection model provides a complete mapping process from the three-dimensional world coordinate system to the two-dimensional image coordinate system, and the homography matrix is ​​a special case of this process, used to describe the mapping relationship between the same plane and the two coordinate systems. By combining the use of the homography matrix, the system can efficiently and accurately convert the physical position of the hydraulic rod into pixel coordinates in the image, providing a solid foundation for subsequent visual monitoring and steering angle calculation.

[0078] In one implementation, the homography matrix H = K*[r1r2t];

[0079] Where K is the camera intrinsic parameter matrix, containing parameters such as the camera's focal length and optical center. r1 and r2 are the first two columns of the rotation matrix R, representing the rotation from the world coordinate system to the camera coordinate system. t is the translation vector, representing the translation from the world coordinate system to the camera coordinate system.

[0080] The homography matrix H is calculated using OpenCV's cv2.findHomography function in conjunction with the RANSAC algorithm. The specific steps are as follows:

[0081] Feature point extraction: Extract feature points from the reference image and the current image, and perform matching.

[0082] Calculate the homography matrix: Use the cv2.findHomography function to calculate the homography matrix H, and set the robustness threshold of RANSAC to 3.0 to remove abnormal feature points caused by vibration and occlusion.

[0083] Homography transformation solves the coordinate offset problem caused by camera installation position deviation, enabling the hydraulic rod features of different models and different camera installation angles to be uniformly mapped to the standard image coordinate system.

[0084] Furthermore, based on the homography matrix and the physical coordinates of the hydraulic rod, the direction vector of the hydraulic rod is calculated.

[0085] Specifically, first, the physical coordinates of the hydraulic rod are defined in the world coordinate system. The homography matrix H is then used to map the physical coordinates of the hydraulic rod from the world coordinate system to the camera coordinate system. The mapped direction vector is then normalized to obtain the normalized direction vector of the hydraulic rod.

[0086] In one embodiment, this application provides an environmental adaptive mechanism. After acquiring hydraulic rod image data based on a preset RGB industrial camera, the mechanism further includes calculating an environmental quality coefficient based on the image data; when the environmental quality coefficient is less than a preset threshold, acquiring the steering column rotation angle acquired by a preset Hall angle sensor; constructing a rotation matrix based on the steering column rotation angle; and correcting the direction vector of the hydraulic rod based on the rotation matrix.

[0087] Specifically, the environmental quality coefficient is calculated based on image data, including calculating the variance of the Laplacian operator of the image grayscale; calculating the standard deviation of the image grayscale; and normalizing and weighting the variance and standard deviation of the Laplacian operator to obtain the environmental quality coefficient.

[0088] The environmental quality coefficient θ (θ∈[0,1]) can be calculated using the following formula:

[0089]

[0090] Where θ is closer to 1, the more ideal the environment; θ < 0.3 indicates severe interference. `laplacian_var` represents the variance of the Laplacian operator in the grayscale image, measuring sharpness; a higher value indicates a sharper image, typically ranging from 0 to 2000. `contrast` represents the standard deviation of the grayscale image, measuring contrast; a higher value indicates stronger contrast, typically ranging from 0 to 100.

[0091] Normalization process: Map the resolution to the range of 0 to 1. Map the contrast to the range of 0 to 1.

[0092] Next, a weighted fusion is performed, with each accounting for 50% of the weight. Finally, θ is constrained to [0,1] and truncated by max(0.0,min(1.0,θ)).

[0093] When the environmental quality is poor, this application can activate a multi-sensor fusion strategy, correct the direction vector based on the angle data detected by the Hall sensor, and improve the accuracy of the results.

[0094] Optionally, to address the issue of visual feature failure caused by harsh environments such as heavy fog and high dust levels, the system also includes calculating an environmental quality coefficient to assess the current environmental quality. Based on the assessment results, the feature point selection threshold is dynamically adjusted, lowering the threshold when the environmental quality is poor to enhance anti-interference capabilities. For example, a threshold below 0.3 indicates severe interference, and the feature point selection threshold is dynamically adjusted, relaxing the conditions in harsh environments (gradient threshold reduced from 50 to 30) to avoid losing effective feature points.

[0095] Next, high-gradient feature points are selected to mitigate the effects of blurring and occlusion. If the number of effective feature points is insufficient (less than 5), prediction is performed using historical homography matrices. For example, the homography matrices of the last 3 frames are cached, and the current homography matrix is ​​generated through linear prediction when there are insufficient feature points, thus solving the problem of computational interruption caused by transient occlusion.

[0096] The dynamic calibration algorithm proposed in this application effectively avoids feature point loss in harsh environments such as heavy fog and high dust levels through an environmental adaptive mechanism, significantly improving the system's environmental adaptability. Simultaneously, by employing multi-sensor fusion technology, it can still reliably calculate the steering angle even when vision fails, greatly improving data accuracy. Furthermore, by introducing historical data redundancy, it solves the problem of computational interruption caused by momentary occlusion, ensuring the continuity and stability of the system. The algorithm also features a good compatibility design, performing identically to the original algorithm under normal conditions, with optimization only triggered under special operating conditions, ensuring compatibility with existing equipment and operating conditions. Ultimately, the algorithm significantly reduces the false recognition rate, especially in harsh environments, where the system's robustness is significantly enhanced, providing strong support for the safe and efficient operation of construction machinery.

[0097] S104 converts the direction vector into the actual steering angle of the non-polarized steering wheel, and determines the percentage of the limit angle based on the actual steering angle, the left limit tilt angle, and the right limit tilt angle.

[0098] In one implementation, the direction vector (Vc) of the hydraulic rod in the camera coordinate system is converted into the actual steering angle, which can be done based on the following formula:

[0099]

[0100] Where Vc,x,Vc,y are the x and y direction components of the hydraulic rod direction vector in the camera coordinate system, and arctan2(Vc,y,Vc,x) is the arctangent function in the four quadrants, with an output range of [-π,π]. Convert to angle range [-180°, 180°].

[0101] The final actual steering angle , where a positive value indicates right turn and a negative value indicates left turn, is consistent with the direction of steering wheel operation. This mapping directly links visual features and physical steering angles through trigonometric functions, providing a quantitative basis for calculating the ultimate tilt angle.

[0102] Furthermore, based on the actual steering angle and the left and right limit tilt angles, the limit angle percentage is determined, which is the ratio of the actual steering wheel angle to the limit angle. This is used to measure the relative position of the current steering angle with respect to the steering limits.

[0103] For example, if the actual steering angle θ is 15°, the left limit tilt angle αL is -30°, and the right limit tilt angle αR is 30°, then the percentage of the limit angles is: 15° / 30° = 50%.

[0104] For example, if the actual steering angle θ is -15°, the left limit tilt angle αL is -30°, and the right limit tilt angle αR is 30°, then the percentage of the limit angles is: -15° / -30° = 50%.

[0105] The S105 performs visual rendering and early warning on the display terminal based on the extreme angle percentage.

[0106] In one implementation, a corresponding warning level is determined based on the extreme angle percentage and preset warning thresholds for different levels. A warning strategy corresponding to the warning level is obtained; visual rendering and device control are then performed on the display terminal based on the warning strategy.

[0107] In this embodiment, a three-level warning mechanism is provided. Specifically, when the extreme angle percentage is greater than or equal to 70%, a three-level warning strategy is triggered. The text content is formatted as "[Warning Type]: Turning Approaching Limit (Current Remaining XX°)" in black text on a yellow background with 80% opacity (RGB: 255,220,0), and black text (RGB: 0,0,0) with a font size of 14-16px (adapted to 1080P displays). The specific font and color can be set according to actual conditions.

[0108] When the extreme angle percentage is greater than or equal to 90%, and the equipment is under working load (bucket pressure > 500kPa), a level 2 warning strategy is triggered. The text format is red background with white text: "[Warning]: Steering is about to reach its limit! Correct immediately (XX° remaining)". The background is red with 80% opacity (RGB: 255,0,0), and the text is white (RGB: 255,255,255), accompanied by a 1-2Hz low-frequency buzzer alarm.

[0109] When the limit angle is reached, a level 1 warning strategy is triggered, with the text format "[Danger]: Steering limit reached! System locked", the background is red with 90% opacity (RGB: 255,0,0), the text is white (RGB: 255,255,255), accompanied by a 3-5Hz high-frequency beep, and the steering hydraulic lock is triggered via the CAN bus.

[0110] In an exemplary scenario, Figure 4 This is a schematic diagram of a visual warning interface according to an embodiment of this application, such as... Figure 4 As shown, it can be displayed on the display terminal. Warning messages are displayed below the central notification bar at the top of the device's control interface, such as... Figure 5 As shown, centered left and right (corresponding to) Figure 5The text includes phrases like "Lock activated" and "Please note the left-side view...", displayed as a horizontal floating bar. It covers but does not obscure key control buttons, such as the right-side steering control area and the core data in the left-side parameter charts. The specific display location is not limited in this embodiment and can be set according to actual conditions.

[0111] This application combines dynamic calibration with sensor feedback to adjust steering warning thresholds in real time based on the actual operating environment. The 70% and 90% values ​​are not fixed but rather the result of multiple experimental verifications and adaptation to actual operating environments, taking into account the steering system's load-bearing capacity, the hydraulic system's workload, and safety margins. These values ​​ensure that the operator can perceive danger in advance and react accordingly. Through dynamic monitoring and analysis of the steering angle, the system triggers alarms in stages when approaching its limits. The specific 70% and 90% warning trigger points are calculated based on the hydraulic rod's working range and the equipment's physical limitations; under different loads, the system dynamically adjusts these thresholds.

[0112] In one embodiment, this application proposes a dynamic environment adaptation mechanism to achieve dynamic adaptation to the type of construction machinery, the working environment, and the work tasks.

[0113] Specifically, it enables adaptation to different types of construction machinery. It automatically reads the machine type identification code from the equipment's electronic control unit to determine the machinery type, and adaptively adjusts the warning thresholds at different levels based on the different machinery types.

[0114] Based on the inherent parameters of the equipment's hydraulic system, such as the bucket cylinder stroke and steering pump flow, the machine identification code in the equipment's ECU is automatically read via the CAN bus, such as the unique code for loaders and graders. This triggers the pre-stored machine feature library and automatically adjusts the visual warning threshold. According to the maximum safe angle of the machine's steering system, such as ±45° for loaders and ±30° for graders, the three-level warning threshold is scaled proportionally (70% / 90% → the dynamic ratio of the corresponding machine's safety limit).

[0115] It can also adjust the homography matrix calculation weights and correct the depth value calculation coefficients of the perspective projection model for different machine models' bucket hydraulic rod physical dimensions (such as loader rod length 1.2m, grader rod length 1.5m).

[0116] It can also adapt to different work environments. The work environment is determined based on the environmental quality coefficient, and the warning thresholds at different levels are adaptively adjusted according to different work environments.

[0117] In one implementation, an environment is defined as foggy when the environmental quality coefficient is less than 0.3 and the visibility is less than or equal to 5m. A multi-sensor fusion mode is forcibly enabled, reducing the weight of visual feature points to 30% and increasing the weight of Hall angle sensor data to 70%, while correcting the visual direction vector using a rotation matrix.

[0118] When the environmental quality index is less than 0.5 and the dust concentration is greater than or equal to 10 mg / m³ 3 When the environment is identified as high-dust, morphological filtering enhancement is triggered (the kernel size is dynamically adjusted from 3×3 to 5×5). Simultaneously, the redundancy of the extreme warning threshold is increased (from 70% to 75%) to compensate for visual recognition errors.

[0119] In one implementation, after acquiring hydraulic rod image data based on a preset RGB industrial camera, the method further includes correcting the image using a dark channel prior dehazing algorithm in a foggy environment to obtain preprocessed image data.

[0120]

[0121] Where A is the atmospheric light value and t is the transmittance, the edge sharpness of the hydraulic rod is improved by 40% after correction.

[0122] In high-dust environments, morphological filtering algorithms are used to remove image noise, resulting in preprocessed image data that improves image clarity.

[0123] It can also adapt to different work tasks. Based on the equipment's operating mode, it adaptively adjusts the warning thresholds and warning strategies at different levels.

[0124] In one implementation, bucket pressure sensors and vehicle speed sensors are used to determine operation modes such as precision operations, heavy-duty handling, and rapid relocation.

[0125] When the bucket pressure is ≥80% of the rated value and the vehicle speed is ≤5km / h, the fine operation mode is determined, the three-level warning threshold is tightened (70%→65%, 90%→85%), and the warning trigger response time is shortened (from 200ms→150ms).

[0126] When the bucket pressure is ≤20% of the rated value and the vehicle speed is ≥15km / h, the rapid transfer mode is determined, the threshold is relaxed (70%→75%, 90%→95%), the high-frequency buzzer is turned off and only the visual warning is retained to avoid interfering with the driver's normal operation.

[0127] The dynamic environment adaptation mechanism proposed in this application significantly improves the operational safety, economy, and efficiency of construction machinery under diverse working conditions. By automatically reading the model identification code in the equipment's electronic control unit, the system can accurately identify the type of construction machinery and adaptively adjust the warning threshold accordingly, ensuring that different models operate within their respective safety limits. Regarding operational environment adaptation, the system intelligently switches sensor fusion modes based on the environmental quality coefficient. In foggy or high-dust environments, it enhances image clarity and improves monitoring reliability through algorithm optimization and parameter adjustment. Furthermore, the system flexibly adjusts warning thresholds and strategies according to different tasks. For example, it tightens warning thresholds and shortens response time in a refined operation mode, while relaxing thresholds and optimizing warning methods in a rapid transfer mode. This improves operational efficiency, reduces equipment wear and energy consumption, and enhances the overall performance and adaptability of the system while ensuring safety.

[0128] In one implementation, dynamic calibration data is also included.

[0129] Specifically, the system acquires acceleration and temperature data collected by its sensors; based on the acceleration and temperature data, it determines whether the preset recalibration conditions are met.

[0130] This application proposes an implementation method for determining whether a vision monitoring system for construction machinery meets preset recalibration conditions. The system monitors the operating status of the equipment in real time through sensors. When the detected acceleration exceeds 2g or the temperature change rate exceeds 10℃ / min, it determines that the recalibration conditions are met and immediately triggers the calibration process to ensure the accuracy and reliability of the monitoring system.

[0131] Under the condition that the recalibration is met, the centerline baseline, left limit tilt angle and right limit tilt angle are recalibrated based on the position of the hydraulic rod in the stable state of the preset frame number.

[0132] First, the system recalculates the direction vector of the centerline baseline based on the hydraulic rod position data from the most recent 30 frames under stable conditions to ensure the accuracy of the baseline. Next, to further improve the system's robustness and avoid misjudgments caused by factors such as mechanical deformation, the system adds a safety redundancy of ±1° to the originally calibrated limit tilt angles (αL and αR). This method effectively improves the accuracy and reliability of steering angle monitoring, ensuring the safe operation of construction machinery under various working conditions.

[0133] To facilitate understanding of the methods in the embodiments of this application, the following description is provided in conjunction with the appendix. Figure 2 and 3 Further description.

[0134] like Figure 2As shown, the data acquisition module first uses an RGB industrial camera and a Hall angle sensor to acquire images of the bucket hydraulic rod and output the real-time rotation angle of the steering column, respectively. Next, the image preprocessing unit performs noise reduction and filtering enhancement on the images, while the sensor data parsing unit processes the sensor data.

[0135] In the measurement and processing module, the system evaluates the image quality. If the environmental quality coefficient is greater than 0.3, it performs viewpoint feature extraction and homography matrix calculation to map the physical coordinates of the hydraulic rod to the image coordinate system. If the environmental quality coefficient is not greater than 0.3, it enables multi-sensor fusion and combines Hall sensor data to calibrate the visual direction vector.

[0136] Subsequently, the dynamic calibration unit calibrates the centerline baseline and limit auxiliary lines, and performs adjustments to adapt to different operating conditions. Direction angle mapping is performed based on the hydraulic rod direction vector to obtain the actual steering angle. In the visual rendering and warning module, the system determines the steering state based on the percentage calculation of the limit angle. A corresponding three-level warning strategy is determined based on the steering state. Corresponding warning signals and commands are output through the cab display terminal and CAN bus, completing the entire measurement and warning process.

[0137] like Figure 3 The diagram illustrates the flow of a dynamic calibration algorithm. First, parameters are initialized and input data is gathered, and the environmental quality coefficient θ is calculated. Different feature thresholds are selected based on the value of θ to filter high-precision feature points. If the number of valid points is greater than 5, the homography matrix H is calculated; if it is less than 5 but the historical H matrix is ​​stored for more than two frames, the H matrix is ​​obtained through prediction; otherwise, the most recent H matrix is ​​reused. Next, the calculated H matrix is ​​stored and subjected to coordinate transformation and vector normalization. Finally, the environmental quality coefficient θ is evaluated again. If the condition is met (θ ≥ 0.3), a vector is output; otherwise, the vector is corrected using sensor data, completing the calibration process and outputting the result. The process then continues until the next frame of data is received. This process ensures that the algorithm can accurately calibrate and output reliable measurement results under different environmental conditions.

[0138] Compared to existing technologies, this technical solution offers significant improvements and beneficial effects. Firstly, through dynamic calibration technology, the solution can adapt to different machine models and loads, significantly reducing calibration time and improving adaptability. Secondly, the introduction of a three-level early warning and dynamic threshold mechanism, optimized based on extensive operational data, significantly improves the success rate of emergency obstacle avoidance and enhances operational safety. Furthermore, the modification cost of this technical solution is low; by reusing existing cameras and Hall sensors, the modification cost per unit is controlled within 1000 yuan, greatly reducing the economic burden. In summary, this technical solution has significant advantages in improving the operational safety, economy, and efficiency of construction machinery, providing strong support for the intelligentization and automation of construction machinery.

[0139] According to another aspect of the embodiments of this application, a steering vision assistance device for a non-polarized steering wheel is also provided for implementing the above-described steering vision assistance method for a non-polarized steering wheel. For example... Figure 6 As shown, the device includes:

[0140] The calibration module 601 is used to calibrate the central axis reference line and calibrate the left and right limit tilt angles based on the central axis reference line.

[0141] Image acquisition module 602 is used to acquire hydraulic rod image data based on a preset RGB industrial camera and extract visual feature points from the image data;

[0142] The calculation module 603 is used to calculate the homography matrix based on visual feature points, and to calculate the direction vector of the hydraulic rod based on the homography matrix and the physical coordinates of the hydraulic rod.

[0143] Steering determination module 604 is used to convert the direction vector into the actual steering angle of the non-polarized steering wheel, and determine the percentage of the limit angle based on the actual steering angle, the left limit tilt angle, and the right limit tilt angle.

[0144] The visual rendering and early warning module 605 is used for visual rendering and early warning on the display terminal based on the extreme angle percentage.

[0145] It should be noted that the steering vision assistance device for a non-polarized steering wheel provided in the above embodiments is only illustrated by the division of the above functional modules when executing the steering vision assistance method for a non-polarized steering wheel. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the steering vision assistance device for a non-polarized steering wheel and the steering vision assistance method embodiments provided in the above embodiments belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.

[0146] According to another aspect of the present application, an electronic device corresponding to the steering vision assistance method for a non-polarized steering wheel provided in the foregoing embodiments is also provided, so as to execute the steering vision assistance method for a non-polarized steering wheel described above.

[0147] Please refer to Figure 7 This illustrates a schematic diagram of an electronic device provided by some embodiments of this application. For example... Figure 7 As shown, the electronic device includes: a processor 700, a memory 701, a bus 702, and a communication interface 703. The processor 700, the communication interface 703, and the memory 701 are connected via the bus 702. The memory 701 stores a computer program that can run on the processor 700. When the processor 700 runs the computer program, it executes the steering vision assistance method for a non-polarized steering wheel provided in any of the foregoing embodiments of this application.

[0148] The memory 701 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 703 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0149] Bus 702 can be an ISA bus, PCI bus, or EISA bus, etc. Buses can be divided into address buses, data buses, control buses, etc. Memory 701 is used to store programs. After receiving execution instructions, processor 700 executes the programs. The steering vision assistance method for a non-polarized steering wheel disclosed in any of the aforementioned embodiments of this application can be applied to processor 700, or implemented by processor 700.

[0150] The processor 700 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 700 or by instructions in software form. The processor 700 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 701. Processor 700 reads the information in memory 701 and, in conjunction with its hardware, completes the steps of the above method.

[0151] The electronic device provided in this application embodiment and the steering vision assistance method for a non-polar steering wheel provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they employ, operate, or implement.

[0152] According to another aspect of the present application, a computer-readable storage medium corresponding to the steering vision assistance method for a non-polarized steering wheel provided in the foregoing embodiments is also provided, wherein a computer program (i.e., a program product) is stored thereon, and when the computer program is run by a processor, it executes the steering vision assistance method for a non-polarized steering wheel provided in any of the foregoing embodiments.

[0153] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0154] The computer-readable storage medium provided in the above embodiments of this application and the steering vision assistance method for a non-polar steering wheel provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0155] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0156] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this patent should be determined by the appended claims.

Claims

1. A steering visual assistance method for a non-polarized steering wheel, characterized in that, include: Calibrate the central axis reference line, and calibrate the left and right limit tilt angles based on the central axis reference line; Based on a preset RGB industrial camera, image data of hydraulic rods are acquired, and visual feature points of the image data are extracted. The homography matrix is ​​calculated based on the visual feature points, and the direction vector of the hydraulic rod is calculated based on the homography matrix and the physical coordinates of the hydraulic rod. The direction vector is converted into the actual steering angle of the non-polarized steering wheel, and the percentage of the limit angle is determined based on the actual steering angle and the left limit tilt angle and the right limit tilt angle. Visual rendering and early warning are performed on the display terminal based on the aforementioned extreme angle percentage.

2. The method according to claim 1, characterized in that, After acquiring hydraulic rod image data based on a preset RGB industrial camera, the following is also included: Calculate the environmental quality coefficient based on the image data; When the environmental quality coefficient is less than a preset threshold, the steering column rotation angle collected by the preset Hall angle sensor is obtained; Based on the steering column rotation angle, construct a rotation matrix; The direction vector of the hydraulic rod is corrected based on the rotation matrix.

3. The method according to claim 2, characterized in that, Calculating the environmental quality coefficient based on the image data includes: Calculate the variance of the Laplacian operator for a grayscale image; Calculate the standard deviation of the image's grayscale values; The variance and standard deviation of the Laplace operator are normalized and weighted to obtain the environmental quality coefficient.

4. The method according to claim 1, characterized in that, Visual rendering and early warning on the display terminal based on the aforementioned extreme angle percentage include: Based on the extreme angle percentage and the preset warning thresholds for different levels, the corresponding warning level is determined; Obtain the warning strategy corresponding to the warning level; Based on the aforementioned early warning strategy, visual rendering and device control are performed on the display terminal.

5. The method according to claim 4, characterized in that, Also includes: Automatically read the model identification code in the electronic control unit of the equipment to determine the type of machinery, and adaptively adjust the warning thresholds at different levels based on different machinery types; The working environment is determined based on the environmental quality coefficient, and the warning thresholds at different levels are adaptively adjusted based on different working environments. Based on the equipment's operating mode, the warning thresholds and warning strategies at different levels are adaptively adjusted.

6. The method according to claim 1, characterized in that, Also includes: Acquire acceleration and temperature data collected by the system's sensors; Based on the acceleration and temperature data, determine whether the preset recalibration conditions are met; If the recalibration conditions are met, the central axis reference line, left limit tilt angle, and right limit tilt angle are recalibrated based on the hydraulic rod position under the preset frame number stable state.

7. The method according to claim 1, characterized in that, After acquiring hydraulic rod image data based on a preset RGB industrial camera, the following is also included: In foggy environments, a dark channel prior dehazing algorithm is used to correct the image, resulting in preprocessed image data. In a high-dust environment, morphological filtering algorithms are used to remove image noise and obtain preprocessed image data.

8. A steering visual aid device for a non-polarized steering wheel, characterized in that, include: The calibration module is used to calibrate the central axis reference line and calibrate the left and right limit tilt angles based on the central axis reference line. The image acquisition module is used to acquire hydraulic rod image data based on a preset RGB industrial camera and extract visual feature points from the image data; The calculation module is used to calculate the homography matrix based on the visual feature points, and to calculate the direction vector of the hydraulic rod based on the homography matrix and the physical coordinates of the hydraulic rod. The steering determination module is used to convert the direction vector into the actual steering angle of the non-polarized steering wheel, and determine the percentage of the limit angle based on the actual steering angle and the left limit tilt angle and the right limit tilt angle; The visual rendering and early warning module is used to perform visual rendering and early warning on the display terminal based on the extreme angle percentage.

9. An electronic device, characterized in that, It includes a processor and a memory storing program instructions, the processor being configured to, when executing the program instructions, perform the steering vision assistance method for a non-polarized steering wheel as described in any one of claims 1 to 7.

10. A computer-readable medium, characterized in that, It stores computer-readable instructions that are executed by a processor to implement a steering vision assistance method for a non-polarized steering wheel as described in any one of claims 1 to 7.

Images