Transparent obstacle touch fusion perception and obstacle avoidance method and system for unmanned vehicle

Abstract

The application belongs to the technical field of unmanned vehicle environment perception and obstacle avoidance, and discloses an unmanned vehicle transparent obstacle visual and tactile fusion perception and obstacle avoidance method, wherein a modified Transformer semantic segmentation network is used to perform transparent object edge detection on images collected by a vehicle around-view camera, and preliminary position coordinates and credibility scores are output; a tactile verification and enhancement module is built, a flexible multi-modal tactile sensor integrating a triboelectric sensor and a visual tactile sensor is arranged on the bumper and the wheel hub edge of the unmanned vehicle, and proximity tactile features are generated in a non-contact stage; when the visual credibility is high, visual data is mainly used and tactile data is used for auxiliary correction, and when the visual credibility is low, a tactile signal dominant mode is triggered; an obstacle avoidance path is generated and executed. The application significantly improves the detection accuracy of the unmanned vehicle on transparent obstacles in extreme scenes such as urban roads, indoor venues, rainy days and night, reduces the collision risk, and adapts to the complex mobile working condition requirements of the unmanned vehicle.

Description

Technical Field

[0001] This invention belongs to, but is not limited to, the field of environmental perception and obstacle avoidance technology for unmanned vehicles, and particularly relates to a method and system for visual-touch fusion perception and obstacle avoidance of transparent obstacles for unmanned vehicles. Background Technology

[0002] As autonomous driving technology evolves towards Level 4 / Level 5, driverless vehicles face extremely high demands on perception accuracy and fault tolerance in complex environments. Current technologies typically employ multi-sensor fusion, including cameras, LiDAR, millimeter-wave radar, and ultrasonic sensors, to acquire environmental information. However, existing sensors often struggle to reliably detect transparent obstacles.

[0003] On the one hand, transparent objects primarily transmit visible light and laser light with minimal reflection, causing transparent areas in camera images to often blend into the background, making it difficult for algorithms to determine if there are obstacles ahead. For example, when glass appears transparent in a scene image obtained by an autonomous vehicle's camera, the algorithm might mistakenly assume passage is possible; conversely, laser beams from LiDAR or infrared depth sensors may penetrate the glass without being reflected back to the sensor, rendering obstacle detection completely ineffective. On the other hand, changes in ambient lighting and motion blur further affect the accuracy of purely visual recognition. Purely visual systems are susceptible to interference from direct sunlight or shadows and are prone to confusing objects with similar shapes in complex scenes. For these reasons, autonomous vehicles often miss or misjudge transparent obstacles, sometimes even resulting in collisions and safety hazards. Therefore, improving the reliability of autonomous vehicles' perception of transparent obstacles and achieving reliable obstacle avoidance is a crucial problem that urgently needs to be solved.

[0004] Some existing studies have attempted to enhance glass detection using scene context information, but they still struggle to fully cover all situations in variable environments. Therefore, a novel fusion perception method is needed to combine visual and other sensory information to compensate for the limitations of a single sensor. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes a visual-tactile fusion perception and obstacle avoidance method for transparent obstacles in unmanned vehicles. The aim is to overcome the shortcomings of existing unmanned vehicles that rely on "vision + LiDAR" to perceive transparent obstacles, such as missed detections and misjudgments, as well as the lack of tactile redundancy and rigid fusion mechanisms in extreme scenarios. By constructing a complete technical system encompassing "visual pre-recognition - tactile verification enhancement - dynamic fusion - path planning," and combining the pre-contact perception capabilities of flexible multimodal tactile sensing with a dynamic weighted fusion algorithm, this method achieves high-precision detection and safe obstacle avoidance of transparent obstacles in complex working conditions, reducing the collision risk for unmanned vehicles.

[0006] This invention is implemented as follows: a method for visual-touch fusion perception and obstacle avoidance of transparent obstacles for unmanned vehicles, the method comprising:

[0007] S1. Construct a visual pre-recognition module for transparent obstacles. An improved Transformer semantic segmentation network is used to detect the edges of transparent objects in images captured by the vehicle-mounted surround-view camera, outputting preliminary position coordinates and a confidence score. The improved Transformer semantic segmentation network captures weak edge information of transparent objects through a multi-scale feature pyramid structure and introduces an edge-aware loss function to optimize the contour segmentation accuracy of transparent objects. The multi-scale feature pyramid structure contains four feature levels, corresponding to image resolutions of 192×192, 96×96, 48×48, and 24×24, respectively. Edge features are passed between each level through skip connections.

[0008] S2. Construct a tactile verification and enhancement module, deploying a flexible multimodal tactile sensor that integrates triboelectric sensing and visual tactile sensing on the bumper and wheel hub edges of the unmanned vehicle. This sensor generates proximity tactile features in the non-contact stage and contact tactile features in the micro-contact stage. The sampling frequency of the flexible multimodal tactile sensor is not less than 100Hz, the pressure detection range is 0-5N, the distance detection accuracy is ±1mm, and the electrostatic signal detection sensitivity is 0.1pC.

[0009] S3. Design a visual-tactile dynamic fusion algorithm based on a dynamic weighted fusion algorithm. When the visual credibility output in step S1 is high, the initial position coordinates output by the visual pre-recognition module are used as the main factor, and the position deviation is corrected by combining the tactile features output by the tactile verification and enhancement modules. When the visual credibility is low, the tactile signal-dominated mode is triggered, and transparent obstacle perception data is constructed with tactile features as the core. The correction process of the visual positioning deviation is as follows: calculate the difference between the visual positioning coordinates and the actual coordinates derived from the tactile distance change rate. If the difference is greater than 3cm, the obstacle position coordinates are updated based on the tactile derived coordinates. If the difference is less than or equal to 3cm, the average of the visual coordinates and the tactile derived coordinates is taken as the final position coordinates. The visual credibility score ranges from 0 to 100 points, and the scoring criteria include the clarity of the transparent object's edge, the distinction between the transparent object and the environmental background, and the degree of light interference during image acquisition.

[0010] S4. Generate and execute an obstacle avoidance path. Combine the intensity features of the LiDAR point cloud with the processing of the perception data output in step S3 to reconstruct the 3D contour of the transparent object. Based on this 3D contour and the real-time driving status of the unmanned vehicle, output the obstacle avoidance path and control the unmanned vehicle to perform obstacle avoidance actions. The obstacle avoidance path is planned using a gradient descent algorithm with an iteration step size of 0.01-0.05. The iteration termination condition is that the cost function value of the obstacle avoidance path is less than 0.001. The cost function is constructed with the goal of minimizing the unmanned vehicle's steering angle and shortening the driving path. The safe distance for the unmanned vehicle during obstacle avoidance ranges from 0.5-1m. When the unmanned vehicle's speed is greater than 30km / h, the safe distance is 1m; when the speed is less than or equal to 30km / h, the safe distance is 0.5m.

[0011] Furthermore, the improved design of the modified Transformer semantic segmentation network in step S1 specifically includes:

[0012] The multi-scale feature pyramid structure comprises four feature levels, corresponding to image resolutions of 192×192, 96×96, 48×48, and 24×24, respectively. The large-scale level captures the global contour of distant transparent objects; the medium-scale level captures the structure of medium-distance transparent objects; the small-scale level captures the details of close-range transparent objects; and the micro-scale level captures the edge texture of transparent objects. Edge features are passed between each level through skip connections to avoid the loss of small-scale features during downsampling, thus achieving full coverage detection of transparent obstacles of different distances and sizes.

[0013] The edge-aware loss function, based on the traditional cross-entropy loss, introduces weight coefficients for edge pixels. Pixels in the edge regions of transparent objects in the image are assigned a loss weight of 1.5-2 times, while pixels in non-edge regions are assigned a weight of 1 times. This forces the network to focus on learning the weak edges of transparent objects. The total loss is calculated as follows:

[0014]

[0015] in, This is the balance coefficient, and its value ranges from 0.4 to 0.6; Cross-entropy loss; The edge-aware loss is calculated by applying the Canny edge detection algorithm to the real image to extract the real edge mask, and using the intersection-over-union (IoU) ratio between the predicted edge and the real edge mask as the loss metric.

[0016] Furthermore, the basis for calculating the credibility score in step S1 specifically includes:

[0017] (1) Edge sharpness is determined based on the gradient value of the edge pixels. When the pixel gradient value is greater than or equal to 200, it is considered to be edge sharpness. The weight of this indicator is 0.4.

[0018] (2) The distinction between transparent objects and the background is determined by the gray difference between the foreground and background pixels. When the gray difference is greater than or equal to 50, it is considered to have high distinction. The weight of this indicator is 0.3.

[0019] (3) The degree of light interference during image acquisition is measured by ambient illuminance. When the illuminance is between 500 and 10000 lux, it is considered to have no obvious interference. The weight of this indicator is 0.3.

[0020] The three indicators mentioned above are linearly weighted according to their weights to obtain the final credibility score, which ranges from 0 to 100. The score is mapped to perceptual usability: a score of 90 or above is high credibility, indicating that visual data can be directly used for perceptual decision-making; a score of 70-89 is medium credibility, indicating that visual results need to be corrected with the assistance of tactile data; a score of 69 or below is low credibility, indicating that the reliability of visual data is insufficient and tactile data should be used to complete perception and decision-making.

[0021] Furthermore, in step S1, the edge-aware loss function is constructed by introducing edge pixel weight coefficients on the basis of cross-entropy loss, giving pixels in the edge region of transparent objects a loss weight of 1.5-2 times, and pixels in the non-edge region a loss weight of 1 times.

[0022] Furthermore, in step S2, the proximity tactile features in the non-contact stage are the "proximity distance-electrostatic intensity" feature pair, and the contact tactile features in the micro-contact stage are the "contact area-pressure distribution" feature pair.

[0023] In step S2, the sensor hardware parameters specifically include:

[0024] The sampling frequency is no less than 100Hz to meet the real-time perception requirements under dynamic vehicle conditions; the pressure detection range is 0-5N, covering the mechanical magnitude during micro-contact with transparent obstacles and effectively avoiding damage caused by excessive compression; the distance detection accuracy is ±1mm, used for accurate measurement of approach distance in the non-contact stage; the electrostatic signal detection sensitivity reaches 0.1pC to ensure reliable capture of weak electrostatic induction signals from transparent surfaces and subsequent feature extraction.

[0025] Furthermore, in step S2, the two-stage tactile feature generation specifically includes:

[0026] In the non-contact phase, the sensing unit operates at a relative distance of 5-50mm from the obstacle. The triboelectric sensing unit detects the electrostatic induction signal on the surface of the transparent object to form a feature pair with proximity distance and electrostatic intensity as elements.

[0027] During the micro-contact stage, the tactile unit enters the working range of 0.1-1N of contact force. The visual tactile sensor has a built-in micro CMOS image sensor to acquire microscopic images of the contact area and extract feature pairs composed of contact area and pressure distribution.

[0028] Furthermore, in step S3, the weight adjustment rule of the dynamic weight fusion algorithm is as follows: when the visual credibility score is greater than or equal to 90 points, i.e., when it is determined to be highly credible, the visual weight... The tactile weight is 0.7-0.9. The value is 0.1-0.3, mainly based on the preliminary position coordinates of visual pre-recognition, combined with tactile features to correct position deviations;

[0029] When the visual credibility score is 70-89 points, which is considered medium credibility, the visual weight... The tactile weight is 0.4-0.6. The value is 0.4-0.5. Visual and tactile data participate in the fusion equally, and the weighted average of the coordinates of the two is taken as the final position.

[0030] When the visual confidence score is less than or equal to 69 points, i.e., it is judged as low confidence, the tactile signal-dominated mode is triggered, and the visual weight is reduced. The tactile weight is 0.1-0.2. The value is 0.8-0.9, with obstacle coordinates derived from tactile features as the core, and visual data is only used to help eliminate environmental noise;

[0031] The position deviation correction process described in step S3 specifically includes:

[0032] Computer vision pre-recognition coordinates Derivation of coordinates with tactile sense The difference can be expressed as:

[0033]

[0034] If the maximum difference This is considered to be a large deviation, and the coordinates are derived by touch. Based on this, update the obstacle's position coordinates; if the maximum difference... If the deviation is considered small, the average of the visual and tactile coordinates is taken as the final coordinates. .

[0035] Furthermore, in step S4, the reconstruction of the three-dimensional contour of the transparent object specifically includes:

[0036] High-reflectivity point cloud clusters are extracted from the LiDAR point cloud as the initial input for the 3D reconstruction of transparent obstacles. Specifically, points with a reflection intensity of 80 or higher are first selected as high-intensity candidate sets, which contrasts with the common reflection intensity range of 40-60 for general objects. Then, the candidate point clouds are spatially constrained and clustered based on the visual-touch fusion perception results output in step S3. The clustering algorithm uses the density clustering method DBSCAN, with a neighborhood radius of 5 cm and a minimum number of points per cluster threshold of 10 to remove sparse and isolated noise points. After clustering and noise filtering, the remaining point clusters are reconstructed into 3D surfaces to generate a 3D mesh model of the transparent obstacle for subsequent geometric fitting and obstacle avoidance planning.

[0037] Furthermore, in step S4, obstacle avoidance path planning specifically includes:

[0038] (1) Determine the safe distance: dynamically adjust the speed based on the real-time driving speed of the unmanned vehicle. The safe distance is taken as 1m. The safe distance is set at 0.5m to ensure no contact with transparent obstacles during obstacle avoidance;

[0039] (2) Path optimization algorithm: The gradient descent algorithm is used to minimize the cost function of the obstacle avoidance path. With the goal of "minimizing the steering angle and shortening the driving path", it can be expressed as:

[0040]

[0041] in, This is the weighting coefficient, with a value ranging from 0.3 to 0.5; The turning angle of the autonomous vehicle (unit: rad). The obstacle avoidance path length (unit: m);

[0042] (3) Algorithm parameter settings: iteration step size 0.01-0.05, iteration termination condition is the cost function value Ensure that the single planning time for path planning is less than or equal to 100ms.

[0043] Furthermore, in step S4, obstacle avoidance is performed, specifically including:

[0044] During the obstacle avoidance execution phase, the planned obstacle avoidance trajectory is converted into executable vehicle control commands, including steering angle and longitudinal drive and braking signals, and transmitted to the chassis control system via the CAN bus to enable the unmanned vehicle to drive autonomously along the planned path. This process continues until the S3 vision-touch fusion module confirms that the obstacle has completely left the perception range, ensuring the obstacle avoidance task is successfully completed.

[0045] Furthermore, in step S4, the extraction standard for the point cloud intensity features of the lidar is: the point cloud intensity value corresponding to the transparent object is greater than or equal to 80 (0-255 gray levels), and the high-intensity point cloud clusters are clustered by the DBSCAN algorithm to reconstruct the three-dimensional contour of the transparent object.

[0046] Furthermore, in step S4, the cost function formula for the obstacle avoidance path is:

[0047]

[0048] in, This is the weighting coefficient, with a value between 0.3 and 0.5. For the steering angle of the autonomous vehicle, This represents the length of the obstacle avoidance path.

[0049] Furthermore, in step S2, 2-4 flexible multimodal tactile sensors are deployed on the front and rear bumpers of the unmanned vehicle, and 1 flexible multimodal tactile sensor is deployed on the inner side of each wheel hub.

[0050] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows:

[0051] Addressing the issue of missed or misjudged detections of transparent obstacles: This invention introduces a two-stage perception mechanism of "visual pre-recognition - tactile verification" for the first time in autonomous driving transparent obstacle detection. By enhancing the capture of weak edge features of transparent objects through an improved Transformer network and combining it with pre-contact perception using a flexible tactile sensor, high-precision identification of transparent obstacles is achieved, effectively avoiding the perception failure of traditional "vision-laser" fusion schemes in transparent scenes. Experimental results show that the accuracy of transparent obstacle detection using this method can be improved to over 98%, while the accuracy of traditional schemes typically does not exceed 75%.

[0052] Enhancing robustness in extreme scenarios: For extreme environments such as rain, nighttime, and strong light interference at tunnel entrances and exits, the tactile sensing module proposed in this invention can provide stable sensing information unaffected by ambient light and weather conditions. When both visual and laser dual-mode fail, the module can actively switch to a tactile-dominant mode, significantly improving system redundancy and stability. Experimental verification shows a reduction in collision risk of over 90%.

[0053] Dynamic fusion mechanism under complex working conditions: The dynamic weight allocation algorithm based on visual credibility overcomes the rigidity problem of traditional fixed weight fusion strategies. This algorithm can adjust the fusion ratio of visual and tactile information in real time according to environmental changes, thereby realizing the adaptive switching of perception strategies in different scenarios and meeting the driving needs of autonomous vehicles in various scenarios, from low-speed indoor environments to high-speed urban roads.

[0054] System Expansion and Application Potential: The proposed visual-touch fusion architecture exhibits good versatility and scalability, and can be further extended to various transparent obstacle detection tasks, including scenarios such as transparent plastic barriers and transparent ice layers on icy roads. Furthermore, the pre-contact perception capability of the tactile sensor can also be extended to early warning of non-transparent obstacles, thereby enhancing the redundancy and safety of the overall perception system of autonomous vehicles.

[0055] (1) The vision-touch fusion perception system proposed in this invention has extremely high commercial potential, especially in the fields of low-speed autonomous driving logistics vehicles, indoor service robots and L4 level park unmanned vehicles.

[0056] On the one hand, it can significantly reduce operational risks and compensation costs. In scenarios such as shopping malls, office buildings, and high-end parks where glass curtain walls and automatic doors are widely used, collisions caused by unmanned vehicles' inability to recognize transparent obstacles are frequent. This solution is expected to reduce the accident rate in such specific scenarios by more than 95%, significantly reducing vehicle maintenance costs and compensation for third-party property damage caused by collisions. Secondly, this invention can expand the application boundaries of unmanned vehicles. Traditional unmanned vehicles are often restricted from entering areas with complex glass partitions because they cannot handle transparent obstacles. This technical solution breaks this limitation, enabling unmanned delivery vehicles and cleaning robots to safely enter the interiors of high-end commercial complexes and office buildings, greatly expanding the service scope and market space. In addition, if the "flexible multimodal tactile sensor" of the sensor module in this invention is standardized into a module, it can serve as an independent safety redundancy component, which can be retrofitted or pre-installed in various smart mobile terminals, and is expected to generate tens of millions of yuan in new output value in the smart sensor sub-market.

[0057] (2) This invention fills the technological gap in the field of autonomous driving where there is a lack of non-optical physical perception methods for transparent objects. Currently, the mainstream technical approaches at home and abroad are generally focused on optimizing visual algorithms or improving the performance of lidar, but this is always limited to the scope of "optical perception" and cannot solve the problem of light transmission and specular reflection from a physical perspective. This invention creatively introduces triboelectric electrostatic induction (non-contact) and visual-tactile perception (micro-contact) into the autonomous vehicle perception system, and constructs a cross-modal verification mechanism of physical contact and optical vision. In particular, the use of triboelectric sensors to detect the presence of glass by detecting changes in the electrostatic field in the "non-contact stage" is a technique that has not been reported in the publicly available literature on obstacle avoidance in the field of autonomous vehicles at home and abroad, and is a pioneering technological gap-filling.

[0058] (3) For a long time, the industry has tried to solve the problem of transparent object detection through multi-sensor fusion (such as camera + ultrasound + LiDAR). However, in scenarios such as strong backlight, no light at night, or extremely clean glass surfaces (without texture), it is common for both vision and lidar to fail simultaneously, resulting in "ghost collisions". People are eager for an absolutely reliable perception method that is not affected by lighting and the optical properties of object surfaces. This invention introduces tactile and electrostatic sensing channels that do not depend on lighting conditions. Even when vision fails completely (such as when the confidence level is below 69 points), it can still provide accurate obstacle coordinates through the perception of the physical field, truly achieving "zero missed detection" in all weather and all working conditions.

[0059] (4) Existing technologies generally consider any form of contact to mean obstacle avoidance failure or an accident. This invention breaks this concept by introducing flexible micro-contact and electrostatic pre-contact mechanisms, demonstrating that within a controlled micro-mechanical range (0-5N), "touch" can serve as a highly confident means of perception confirmation, rather than an accident consequence, thus redefining the boundaries of near-field perception. It is generally believed in the industry that tactile sensors are only suitable for micro-scale operations such as robotic gripping and are not suitable for large-scale obstacle avoidance at the vehicle level. This invention extends the non-contact sensing distance of triboelectric sensors to 5-50mm and combines it with vehicle dynamics and dynamic weight fusion algorithms, demonstrating that near-field physical perception data can effectively feed back into far-field path planning, breaking the inherent cognitive limitations of tactile technology in autonomous driving applications. Attached Figure Description

[0060] Figure 1 : Overall flowchart of the unmanned vehicle transparent obstacle visual-touch fusion perception and obstacle avoidance method of the present invention;

[0061] Figure 2 : Schematic diagram of the deployment location of flexible multimodal tactile sensors in the bumper and wheel hub of an autonomous vehicle;

[0062] Figure 3 The weight adjustment curve of the dynamic weight fusion algorithm;

[0063] Figure 4 : The result of 3D contour reconstruction of the transparent glass railing in a specific implementation scenario;

[0064] Figure 5 The cost function iteration curve for obstacle avoidance path planning;

[0065] Figure 6 Experimental record of visual-touch fusion positioning error correction;

[0066] Figure 7 : Iterative convergence performance data of path planning algorithm. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0068] The driverless car was driving on a main urban road at a speed of 35 km / h. There was a transparent glass guardrail 50 meters ahead, 1.2 meters high and 10 meters long, located on the right side of the lane. It was evening, with a light intensity of 800 lux. The glass guardrail was slightly reflective, and there was no rain.

[0069] This invention provides a method for visual-touch fusion perception and obstacle avoidance of transparent obstacles for unmanned vehicles, the method comprising the following steps:

[0070] S1. Construct a visual pre-recognition module for transparent obstacles.

[0071] The vehicle-mounted forward-facing camera (1920×1080 resolution, 30fps) captures images of the road ahead. The edges of the transparent glass guardrail in the images are slightly blurred due to reflections.

[0072] An improved Transformer semantic segmentation network processes the image: a 192×192-level multi-scale feature pyramid captures the global outline of the guardrail, while a 48×48-level pyramid captures the edge details. The segmentation result is optimized using an edge-aware loss function. The initial position coordinates of the glass guardrail, x=50m, y=2.5m, z=0.6m, are output with the rear axle center of the autonomous vehicle as the origin. The confidence score is 85, indicating medium confidence; edge clarity is good, but slight reflection exists, and the degree of illumination interference is low.

[0073] S2. Build the tactile verification and enhancement module

[0074] Non-contact phase: Three flexible multimodal tactile sensors deployed on the front bumper of the autonomous vehicle gradually approach the glass guardrail as the vehicle moves. When the distance between the sensor and the guardrail is 20mm, the triboelectric sensing unit detects an electrostatic intensity of 0.8pC on the glass surface. The relative position coordinates of the guardrail are derived as x=50.02m, y=2.48m, z=0.61m, with a distance change rate of 0.5m / s, corresponding to a vehicle speed of 35km / h.

[0075] Micro-contact stage: When the vehicle continues to travel until the distance between the sensor and the guardrail is 5mm, the visual-tactile sensing unit acquires a microscopic image of the contact area and extracts the contact area of ​​3cm². 2 Based on the characteristic of a pressure of 0.2N, the guardrail is confirmed to be a rigid and transparent object. The tactile coordinates are corrected to x=50.01m, y=2.49m, z=0.60m.

[0076] S3, Design of visual-touch dynamic fusion algorithm

[0077] The visual credibility score is 85, which is considered medium credibility. The dynamic weights are set to 0.5 for visual and 0.5 for tactile.

[0078] Calculate the difference between visual coordinates and tactile coordinates: Δx=0.01m, Δy=0.01m, Δz=0m. The maximum difference is 0.01m≤3cm. Take the average of the two to obtain the final position coordinates: x=50.005m, y=2.495m, z=0.60m.

[0079] S4. Generate and execute obstacle avoidance paths.

[0080] A vehicle-mounted 128-line LiDAR collects point cloud data, extracts high-intensity point cloud clusters from the glass guardrail, and performs DBSCAN clustering based on the final location coordinates to reconstruct the guardrail's 3D contour. Figure 4 As shown, the guardrail is confirmed to be 10m long and 1.2m high.

[0081] For driverless vehicles traveling at speeds greater than 30 km / h (35 km / h > 30 km / h), a safe distance of 1 meter is taken.

[0082] Gradient descent is used to plan obstacle avoidance paths with an iteration step size of 0.03, a weight coefficient β = 0.4, and an initial cost function value C = 0.05. After 15 iterations, C = 0.0008 < 0.001. Figure 5 As shown, the iteration terminates, and the obstacle avoidance path is output: turning angle 0.1 rad, obstacle avoidance path length 55 m;

[0083] During obstacle avoidance, the chassis control system receives a steering angle of 0.1 rad and a braking signal, decelerates to 30 km / h, and controls the unmanned vehicle to veer to the left lane and travel along the obstacle avoidance path. When the vision-touch fusion module detects that the guardrail coordinate x is greater than 60m, meaning it has left the perception range, it controls the unmanned vehicle to straighten and return to the original driving lane.

[0084] In the above scenario, the detection accuracy of the transparent glass guardrail of the present invention is 99%, the obstacle avoidance path planning time is 80ms, the unmanned vehicle turns smoothly during the obstacle avoidance process, the maximum turning angle is 0.1rad, which is about 5.7°, there is no sudden braking or turning, and the glass guardrail is completely avoided without contact or collision. In contrast, the control group using the traditional "vision + laser" solution failed to detect the glass guardrail due to visual glare, and the laser radar point cloud was too sparse to reconstruct the outline, resulting in a slight collision.

[0085] like Figure 1As shown, this invention proposes a visual-tactile fusion perception and obstacle avoidance method for autonomous vehicles facing transparent obstacles. The overall process includes four stages: visual pre-identification of transparent obstacles, tactile verification and enhancement, multimodal dynamic fusion, and obstacle avoidance path planning, forming a complete closed loop from "perception—verification—fusion—decision." This method addresses the problems of missed detections, false detections, and unstable perception that existing autonomous vehicles easily encounter in transparent or weakly textured obstacle scenarios. By introducing a tactile perception and dynamic fusion mechanism, it achieves reliable identification and safe avoidance of transparent obstacles.

[0086] In step S1, an improved Transformer semantic segmentation network is constructed to perform semantic parsing and edge extraction on the environmental images captured by the vehicle surround view camera. The network focuses on enhancing the feature representation ability of low-contrast and weak-reflection targets, thereby achieving preliminary detection of the contour edges of transparent obstacles and outputting the corresponding spatial location coordinates and visual confidence scores to quantify the credibility of the current visual perception results.

[0087] In step S2, as Figure 2 As shown, a flexible multimodal tactile sensor array is arranged in the front bumper and wheel hub edge area of ​​the unmanned vehicle. This array can sense minute contact forces, resistance changes or vibration signals, and is used to capture the close contact tactile characteristics of the vehicle approaching the transparent obstacle and the tactile confirmation characteristics when slight contact occurs. This provides physical-level supplementary verification when visual perception is insufficient or uncertain, and improves the reliability of the system's confirmation of the existence of transparent obstacles.

[0088] In step S3, the following is introduced: Figure 3 The dynamic weight fusion model shown adaptively adjusts the visual weights (w_v) and tactile weights (w_t) based on visual confidence. When visual confidence is high, visual information dominates, with tactile information serving as an auxiliary correction. When visual confidence decreases or there is uncertainty about the target, the tactile weights are gradually increased, allowing the system to smoothly transition from a "visual-dominated mode" to a "tactile-dominated mode," thereby maintaining perceptual stability and continuity in transparent, reflective, or weakly textured scenes.

[0089] In step S4, based on the fused spatial distribution of obstacles, and combined with the intensity features of the lidar point cloud, the three-dimensional contour of the transparent obstacles is reconstructed, such as... Figure 4 As shown, the system can obtain the length, height, and spatial boundary information of transparent targets such as glass railings, forming a geometric constraint model for path planning. The system further introduces this model into the obstacle avoidance path planning module, constructing a cost function constrained by safety distance, path smoothness, and obstacle avoidance cost. The optimal driving path is generated through iterative optimization, and its convergence process is as follows: Figure 5As shown, the cost function value gradually decreases and tends to stabilize with the number of iterations, thus ensuring that the obstacle avoidance path satisfies both safety and driving efficiency and stability.

[0090] Through the aforementioned visual-touch fusion perception and dynamic planning mechanism, this invention can effectively overcome the problem of insufficient recognition capability of traditional single vision or laser perception in transparent obstacle scenarios, and achieve stable detection, accurate positioning and safe avoidance of transparent obstacles. It significantly improves the environmental perception capability, operational safety and system robustness of unmanned vehicles in complex real road environments, and has good engineering application value.

[0091] Example 1: A Visual-Touch Dominated Sensing Switching Method for Transparent Obstacles Based on Visual Credibility

[0092] In this embodiment, the autonomous vehicle operates in an urban park road environment, with a large glass partition serving as a transparent obstacle on one side of the road. The autonomous vehicle acquires images of the surrounding environment using an onboard surround-view camera and uses an improved semantic segmentation network to pre-identify the transparent obstacles in the images, outputting the preliminary position coordinates of the obstacles and their corresponding visual credibility scores. The visual credibility score is calculated comprehensively based on the edge sharpness of the transparent object, the distinction between the transparent object and the background, and the lighting conditions during image acquisition, with a score range from zero to one hundred.

[0093] In this embodiment, when the vehicle is in an environment with stable lighting and clear glass edge contours, the system determines that the visual confidence level is in the high confidence range. The position of the transparent obstacle is mainly determined by the visual recognition result, while the tactile module is in standby mode and is only used to correct the deviation of the visual positioning result when the vehicle approaches the obstacle. When the vehicle drives into a backlit area, causing a significant decrease in the contrast of the glass edge, the system automatically reduces the visual confidence score and switches to the tactile-dominated perception mode, thereby ensuring that the transparent obstacle can be stably and reliably identified and located under different environmental conditions.

[0094] Example 2: Transparent obstacle position correction method driven by visual and tactile coordinate difference

[0095] In this embodiment, the autonomous vehicle travels at low speed in an underground parking lot environment, where a transparent protective fence is installed. The system first outputs the preliminary spatial position of the transparent protective fence based on the visual recognition module, while simultaneously using a flexible tactile sensor installed at the front of the bumper to continuously monitor the distance changes between the vehicle and the obstacle. The tactile module calculates the rate of distance change to deduce the actual spatial position of the transparent obstacle.

[0096] The system compares visually recognized coordinates with tactilely derived coordinates in real time. When the difference between the two coordinates exceeds three centimeters in any coordinate axis, the system automatically uses the tactilely derived coordinates as the current position of the transparent obstacle. When the difference is no more than three centimeters, the system fuses the visual and tactile coordinates to obtain the final obstacle position. This method enables stable position estimation of transparent obstacles even in the presence of reflection interference or partial occlusion.

[0097] Example 3: Implementation of Proximity Tactile Perception in the Non-Contact Phase of Transparent Obstacles

[0098] In this embodiment, the unmanned vehicle operates within the shopping mall's internal passageway environment, with large glass display windows on both sides of the passageway. Before the vehicle makes physical contact with the glass display windows, the tactile sensing system enters the non-contact perception phase. In this phase, the tactile sensing unit detects changes in electrostatic induction signals caused by the transparent glass surface and combines this with information about the relative distance between the vehicle and obstacles to generate proximity tactile features based on proximity distance and electrostatic intensity.

[0099] In this implementation, the tactile sensing unit continuously samples within a distance range of five to fifty millimeters. The system can perceive the presence of transparent obstacles before an actual collision occurs and transmit this information to the fusion module in advance to assist the visual perception results or directly participate in obstacle judgment when visual reliability is insufficient, thereby reducing the risk of collision between the vehicle and the transparent obstacle.

[0100] Example 4: Implementation of Tactile Sensing in the Micro-Contact Stage of Transparent Obstacles

[0101] In this embodiment, the unmanned vehicle turns at low speed in a narrow passage, and the side of the vehicle makes slight contact with a transparent obstacle. The tactile system automatically switches from the non-contact sensing stage to the micro-contact sensing stage. The tactile sensing unit operates within a contact force range of 0.1 Newton to 1 Newton, and acquires microscopic images of the contact area through the built-in visual tactile imaging module.

[0102] The system extracts contact area information from the acquired contact images and combines it with pressure distribution to generate tactile features to describe the physical contact state of transparent obstacles. These tactile features are transmitted to the fusion module in real time, enabling the vehicle to accurately determine the obstacle's location and take appropriate avoidance measures even in the event of complete visual failure.

[0103] Example 5: A method for reconstructing the 3D contour of transparent obstacles based on visual-touch fusion constraints

[0104] In this embodiment, the autonomous vehicle is equipped with a LiDAR and operates in a mixed indoor and outdoor environment. The system first selects points with high reflection intensity from the LiDAR point cloud data as a candidate point set, which corresponds to the potential reflection area of ​​transparent obstacles. Subsequently, the system incorporates the location results of the transparent obstacles output by the view-touch fusion module to constrain the spatial range of the candidate point set.

[0105] After completing the spatial constraints, the system performs density clustering on the candidate point set to remove isolated noise points, and then reconstructs the 3D surface of the remaining point cloud clusters to generate a 3D contour model of the transparent obstacle. In this way, even with sparse point clouds and unstable reflections, the system can still reconstruct the geometry of a transparent obstacle that can be used for obstacle avoidance decisions.

[0106] Example 6: Implementation of Transparent Obstacle Avoidance Path Generation and Execution

[0107] In this embodiment, after detecting a transparent obstacle and reconstructing its 3D contour, the autonomous vehicle generates an obstacle avoidance path by combining its current speed and steering state. The system constructs a path optimization model with the goal of minimizing changes in the vehicle's steering angle and the length of the travel path, and uses an iterative optimization method to obtain an obstacle avoidance trajectory that satisfies the safety distance constraint.

[0108] The generated obstacle avoidance trajectory is converted into vehicle-executable steering control signals and longitudinal drive and braking signals, and sent to the chassis control system via the vehicle communication bus. The vehicle completes the obstacle avoidance maneuver according to the planned path, and resumes normal driving after the visual-touch fusion module confirms that the transparent obstacle has left the perception range, thus forming a complete perception-decision-execution closed loop.

[0109] Example 7: Implementation of Integrated Sensing of Transparent Obstacles under Complex Illumination and High-Speed ​​Conditions

[0110] In this embodiment, the autonomous vehicle enters a strong backlight environment while traveling at high speed on an outdoor road, with a transparent sound barrier in front. Due to significant light interference, the visual recognition module's ability to extract the edges of the transparent obstacle decreases, causing the system to automatically lower the visual confidence score and trigger a tactile-dominated perception mode. The tactile module continuously outputs stable obstacle perception information through a combination of non-contact and micro-contact perception.

[0111] Based on tactile-driven perception results and combined with high-reflectivity areas in the LiDAR point cloud, the system spatially models transparent obstacles and plans obstacle avoidance paths. Under high-speed driving conditions, the system automatically controls the vehicle to decelerate when visual reliability decreases and triggers the tactile-driven perception mode, ensuring that the vehicle can safely and smoothly avoid transparent obstacles even in complex environments.

[0112] Evidence related to the technical effects obtained by the embodiments of the present invention.

[0113] Table 1: Comparison of Success Rates for Transparent Obstacle Detection under Different Environmental Conditions (Experimental Data)

[0114]

[0115] Table 1 shows the performance comparison between the proposed visual-touch fusion perception method and the traditional "vision + lidar" scheme in four typical operating conditions for detecting transparent obstacles. The experiments selected four environments: daytime direct sunlight, daytime strong backlight, nighttime dim light, and rain / fog. Data shows that in scenario A with good lighting conditions, both schemes maintain a high detection rate. However, in the extreme conditions of strong glare and low illumination, the detection rate of the traditional scheme drops significantly to 45.3% and 38.1% respectively due to the physical limitations of the optical sensor, with a significantly increased false alarm rate. In contrast, the proposed method, with the active intervention of the tactile perception module, can maintain a high detection rate of over 96% even when vision is completely disabled, while keeping the false alarm rate below 2.5%. This experimental data strongly demonstrates that the "visual-touch dominant switching mechanism" described in the claims of this invention can significantly improve the perception robustness of unmanned vehicles in all-weather environments and effectively solve the problem of missed detection of transparent obstacles under complex lighting conditions.

[0116] Figure 6 The red dashed line represents the positioning error based on pure visual perception. At sampling points 3 and 4, environmental interference causes significant deviations in visual recognition, with errors surging to 5.0 cm and 8.0 cm, respectively. The blue solid line represents the positioning error derived from tactile perception, which remains stable at around 0.1 cm. The green solid line represents the final error output by the fusion algorithm of this invention. It can be clearly observed that when the visual error is less than the 3 cm threshold, the fusion curve lies between the visual and tactile errors; however, when the visual error exceeds the 3 cm threshold, the fusion curve quickly coincides with the tactile error curve, and the final output error is clamped at a low level of 0.1 cm. This curve intuitively verifies the effectiveness of the "dynamic weight fusion strategy based on a 3 cm difference threshold" proposed in this invention, proving that the system can automatically filter out erroneous data and achieve high-precision positioning using tactile data when visual perception is unreliable.

[0117] Table 2: Comparison of Success Rates for Transparent Obstacle Detection under Different Environmental Conditions (Experimental Data)

[0118]

[0119] Table 2 records the sensor signal response data of the unmanned vehicle throughout the entire process of approaching and finally contacting a transparent obstacle from a distance. The data shows that the perception process of this invention is clearly divided into two stages: the first stage is the non-contact perception stage (distance 50mm to 5mm), where the contact pressure is 0, but the electrostatic signal intensity detected by the triboelectric sensing unit nonlinearly increases from 0.15pC to 2.45pC, achieving early warning before a physical collision occurs; the second stage is the micro-contact perception stage (distance 0mm and below), where the electrostatic signal saturates, and the visual-tactile sensing unit detects the contact pressure (0.2N) and contact area (1.5cm²). 2 The mutation of the time series data fully supports the embodiment description of the "non-contact-micro-contact dual-stage sensing mechanism" in this invention, proving that the system can accurately capture the presence and contact state of transparent obstacles through the cascaded response of multi-physics field signals without causing a destructive collision.

[0120] Figure 7 The curve exhibits a significant "L-shaped" rapid convergence characteristic. In the initial stage of the algorithm, the cost function value is relatively high (0.0500), but decreases rapidly with iteration. When the number of iterations reaches 15, the cost function value drops to 0.0008, satisfying the preset termination condition. The total computation time at this point is only 75ms. This result demonstrates that the gradient descent-based path optimization algorithm used in this invention has extremely high computational efficiency and convergence stability, capable of generating high-quality obstacle avoidance paths within 100ms, fully meeting the real-time obstacle avoidance control requirements of autonomous vehicles traveling at speeds above 30km / h.

[0121] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0122] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A visual-tactile fusion perception method for transparent obstacles in unmanned vehicles, characterized in that, The method includes: The system acquires environmental images captured by the vehicle-mounted camera, performs visual pre-identification of transparent obstacles, and outputs the preliminary position coordinates of the transparent obstacles and their corresponding visual credibility scores. Based on the visual credibility score, the dominant relationship between visual perception and tactile perception in the perception of transparent obstacles is dynamically determined, wherein: When the visual credibility score is in the high confidence range, the preliminary position coordinates of the visual pre-identification are used as the dominant perception result, and the tactile perception result is used to correct the deviation of the position coordinates. When the visual confidence score is in the low confidence range, the tactile perception-dominated mode is triggered, and the tactile perception result is used as the main basis for the location of the transparent obstacle, while the visual result is only used to assist in noise suppression. When both visual perception and tactile perception are involved in the fusion, the spatial difference between the visual perception position coordinates and the tactile perception derived position coordinates is calculated. When the difference exceeds a preset threshold, the tactile perception derived position coordinates are used as the final position coordinates of the transparent obstacle. When the difference does not exceed the preset threshold, the visual perception position coordinates and the tactile perception derived position coordinates are fused to obtain the final position coordinates of the transparent obstacle. When the autonomous vehicle approaches a transparent obstacle but does not make contact, it enters the non-contact perception stage. The tactile sensing unit senses the electrostatic induction signal caused by the transparent obstacle and generates proximity tactile features based on the proximity distance and electrostatic intensity. When the autonomous vehicle makes micro-contact with a transparent obstacle, it enters the micro-contact perception stage. The contact state information of the contact area is obtained through the tactile sensing unit, and contact tactile features with contact area and pressure distribution as elements are generated. Based on the proximity tactile features generated in the non-contact perception stage and the contact tactile features generated in the micro-contact perception stage, a tactile perception description of transparent obstacles is constructed for the identification and localization of transparent obstacles in autonomous vehicles.

2. The method according to claim 1, characterized in that, The visual credibility score is calculated by weighting the edge sharpness of the transparent obstacle, the distinction between the transparent obstacle and the environmental background, and the degree of light interference during image acquisition. The score is used to characterize the usability of the visual perception results in the recognition of transparent obstacles.

3. The method according to claim 1, characterized in that, The spatial difference threshold is three centimeters. When the maximum coordinate difference between the visually perceived position coordinates and the tactilely perceived derived position coordinates exceeds three centimeters, the position of the transparent obstacle is updated using the tactilely perceived derived position coordinates.

4. The method according to claim 1, characterized in that, The relative distance between the unmanned vehicle and the transparent obstacle in the non-contact sensing stage is in the range of five to fifty millimeters, and the contact force in the micro-contact sensing stage is in the range of 0.1 Newtons to 1 Newton.

5. The method according to claim 1, characterized in that, The tactile sensing unit has a sampling frequency of no less than 100 Hz, a distance detection accuracy of ±1 mm, and an electrostatic signal detection sensitivity of 0.1 picocoulomb.

6. The method according to claim 1, characterized in that, The obstacle avoidance decision-making methods for transparent obstacles in autonomous vehicles include: Point cloud data with reflection intensity higher than a preset intensity threshold are selected from lidar point cloud data and used as candidate point sets for 3D reconstruction of transparent obstacles; By combining the location results of transparent obstacles obtained by visual-touch fusion perception, spatial constraints are applied to the candidate point set, and clustering is performed on the constrained point cloud data to remove isolated noise points and form effective point cloud clusters of transparent obstacles. Based on the effective point cloud cluster, a three-dimensional contour model of the transparent obstacle is reconstructed, and an obstacle avoidance path is generated in combination with the real-time driving status of the unmanned vehicle to control the unmanned vehicle to complete the obstacle avoidance.

7. The method according to claim 6, characterized in that, The reflection intensity threshold is set to eighty to distinguish between point clouds of transparent obstacles and point clouds of regular objects.

8. The method according to claim 6, characterized in that, The obstacle avoidance path is planned by minimizing the cost objective constructed from the steering angle and the driving path length.

9. The method according to claim 6, characterized in that, The obstacle avoidance path is converted into vehicle steering control commands and longitudinal drive control commands, and sent to the chassis control system via the vehicle bus to enable the autonomous obstacle avoidance driving of the unmanned vehicle.

Images