Vehicle bottom detection robot local path planning method based on cross-modal collaborative learning

Abstract

The invention relates to the technical field of path planning, in particular to a vehicle bottom detection robot local path planning method based on cross-modal collaborative learning. The method comprises the following steps: carrying out data preprocessing on acquired multi-modal sensing data; according to the preprocessed data and based on a cross-modal collaborative modeling mechanism, constructing a multi-modal fusion perception feature map; mapping the multi-modal fusion perception feature map into a local path navigation map based on path planning; selecting an optimal path in the local path navigation map based on a path generation mechanism; and performing optimal path optimization based on a state error estimation mechanism. And the functions of deep fusion of multi-modal data, dynamic mapping of spatial geometry and semantic categories, modal confidence self-adjustment, execution feedback correction and the like are realized. Experimental verification shows that the success rate of the path planning task can be increased to 95.3% from 78.6% in a typical vehicle bottom complex scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of path planning, in particular to a local path planning method for vehicle bottom detection robot based on cross-modal collaborative learning. BACKGROUND

[0002] With the wide application of intelligent detection equipment in security inspection, emergency inspection, dangerous area exploration and other fields, the vehicle bottom detection robot, as an automatic, non-contact and high-precision detection means, is gradually replacing the traditional manual and fixed equipment, and becomes the key equipment to improve the efficiency and safety of vehicle chassis security inspection. However, the vehicle bottom environment has complex characteristics such as typical space limitation, non-uniform illumination, strong surface reflection and frequent structural occlusion, which makes the existing robot system face severe challenges in path planning and perception stability.

[0003] At present, the mainstream vehicle bottom path planning method is mostly based on visual SLAM, laser radar navigation or obstacle avoidance algorithm based on geometric model. Such methods often rely on single modal input, and it is difficult to maintain stable operation when there is multi-source interference, incomplete sensor data or degraded perception quality. For example, the laser radar may have distance jump in front of the structure occlusion or vehicle bottom metal reflection surface, and the visual image may easily lose edge information in low light or backlight environment. The defects of such single modal perception architecture limit the robustness and continuity of path planning.

[0004] On the other hand, most of the existing path planning systems directly input the perception results into traditional local planning algorithms such as DWA and Teb, ignoring the importance of semantic information to path decision, resulting in path selection too dependent on geometric distance and lack of contextual understanding ability of environment structure and detection target. For example, when the robot travels in the narrow space of the vehicle bottom, distance-based obstacle avoidance may make it unable to identify the functional difference between "vehicle edge" and "ordinary obstacle", resulting in misjudgment, path shock or misentry into dead angle.

[0005] In summary, it is the key to promote the vehicle bottom detection robot to realize stable navigation, efficient perception and intelligent obstacle avoidance to construct a local path planning system that supports multi-modal collaborative perception, has modal failure adaptive ability, and fuses semantic structure mapping and path optimization strategy. Therefore, the present application proposes a local path planning method for vehicle bottom detection robot based on cross-modal collaborative learning, which fuses computer vision, embedded perception, graph neural mapping and adaptive motion control and other technologies, and constructs a complete, intelligent and deployable robot path planning system for actual scenes. SUMMARY

[0006] In order to solve the above-mentioned problems, the present application provides a local path planning method for vehicle bottom detection robot based on cross-modal collaborative learning.

[0007] In a first aspect, the present application provides a local path planning method for a car bottom detection robot based on cross-modal collaborative learning, which adopts the following technical solution:

[0008] A local path planning method for a car bottom detection robot based on cross-modal collaborative learning, comprising:

[0009] Obtaining multi-modal perception data;

[0010] Data preprocessing is performed on the obtained multi-modal perception data;

[0011] Constructing a multi-modal fusion perception feature map based on the preprocessed data and a cross-modal collaborative modeling mechanism;

[0012] Mapping the multi-modal fusion perception feature map to a local path navigation map based on path planning;

[0013] Selecting an optimal path in the local path navigation map based on a path generation mechanism;

[0014] Optimizing the optimal path based on a state error estimation mechanism.

[0015] Further, the data preprocessing of the obtained multi-modal perception data includes laser radar point cloud projection and depth completion fusion, wherein first, each three-dimensional point is subjected to an external parameter transformation and perspective projection, and all point clouds are mapped to a sparse depth map M LiDAR , wherein the pixel position (u k ,v k ) stores the corresponding For each empty pixel position (i,j) in the image, the present application searches for the nearest non-empty point (u k ,v k ) within a window of a certain radius r and assigns its depth, and then performs weighted fusion of the interpolation completion map and the depth camera image I D , which is expressed as:

[0016]

[0017] where p k =(x k ,y k ,z k ) T is the position of the laser radar point in the radar coordinate system, T is the transpose; R and t are the rigid body transformation parameters (rotation and translation) of the laser radar to the camera; is the three-dimensional coordinates of the point in the camera coordinate system; K is the camera intrinsic matrix (including focal length and principal point offset); (u k ,v k) is the pixel coordinate of the point projected to the image space; is the depth value of the point under the camera view.

[0018] Further, the data preprocessing of the acquired multi-modal perception data further comprises modal normalization processing and numerical scale unification, wherein each modal image A standardization operation is performed to make its mean value 0 and standard deviation 1, denoted as:

[0019]

[0020] wherein I RGB ,I IR is the image of the RGB camera, and the image of the infrared camera; I i (h, w) is the pixel value of the i-th modal image at position (h, w); is the mean value of the multi-modal full image; is the standard deviation of the image; H is the height of the image; W is the width of the image; and ε is a numerical stability factor to prevent the denominator from being zero. is the normalized image, and the numerical distribution is standard normal, which ensures that the expression weight of each modal in the subsequent network is balanced, avoiding the phenomenon of single modal dominance.

[0021] Further, the data preprocessing of the acquired multi-modal perception data further comprises introducing an edge saliency enhancement operation on the normalized image to enhance the local structural features of the vehicle bottom structure. First, a Sobel edge gradient map is calculated, denoted as:

[0022]

[0023] wherein, is the partial derivative along the x direction, representing the rate of change of image intensity in the horizontal direction; x is the horizontal coordinate of the image, representing the position of the pixel in the image width direction; is the partial derivative along the y direction, representing the rate of change of image intensity in the vertical direction; y is the vertical coordinate of the image, representing the position of the pixel in the image width direction; and then the original image is fused by weighting to obtain wherein λ is an edge saliency fusion factor; E i is the edge map, enhancing the structural information; is the modal image after edge enhancement.

[0024] Further, the multi-modal fusion perception feature map is constructed according to the preprocessed data and based on the cross-modal collaborative modeling mechanism, comprising inputting the tensor F in ∈R H×W×5Respectively constructed as an image modal branch and a structure modal branch, deep semantic embeddings of each type of modal are extracted through a double-flow neural network, and an initial expression with clear structure is provided for subsequent modal collaboration, wherein RGB image + infrared image are merged as image modal input: F img =F in [:,:,0:4]∈R H×W×4 , wherein [:,:,0:4] represents extracting the first 4 channels, i.e., R, G, B, and IR; H, W represent the height and width dimensions of the image; the laser radar fused depth map is a structure modal input, and finally two lightweight convolutional neural networks are input to extract modal features, denoted as:

[0025] F′ img =CNN img (F img ),F′ geo =CNN geo (F geo )

[0026] , wherein CNN img is a three-layer convolution-normalization-ReLU network; CNN geo is a two-layer light convolution network; F′ img ∈R H ′×W′×C is an image modal feature map; F′ geo ∈R H′×W′×C is a structure modal feature map.

[0027] Further, the construction of the multi-modal fusion perception feature map according to the preprocessed data and based on the cross-modal collaborative modeling mechanism further includes introducing a cross-modal attention guidance enhancement mechanism, using the information advantage of the image modal to guide the structure modal to strengthen the expression in the spatial position, constructing a more semantically consistent fusion representation, then constructing an attention matrix, applying the attention weight matrix to the image modal feature value, and then adopting a channel fusion residual strategy to fuse the information of the two structure modals, and the cross-modal guidance mechanism is represented as:

[0028] Q=φ q (F′ img ),K=φ k (F′ geo ),V=φ v (F′ img )

[0029] , wherein φ q () is a query vector; φ k () is a key vector; φ v () is a value vector, and the three are respectively the features after linear mapping of query, key, and value.

[0030] Furthermore, the multimodal fusion perception feature map is constructed based on the preprocessed data and the cross-modal collaborative modeling mechanism, and the modal confidence perception mechanism is introduced to dynamically evaluate the confidence of the two types of modalities, and the final output features are adjusted proportionally to ensure that the dominant features come from the reliable modal path. First, the output features of the two paths are globally averaged and pooled GAP: g img =GAP(F′ img ),g geo =GAP(F′ geo ), and then through a small fully connected layer and Sigmoid activation to get the confidence: w img =σ(W1·g img ),w geo =σ(W2·g geo )W1, W2 are the corresponding weight matrices, and the weighting coefficients are obtained after normalization: The final fusion output features are:

[0031] Furthermore, the multimodal fusion perception feature map is mapped into a local path navigation map based on path planning, including mapping the multimodal fusion perception feature map to a local path navigation map based on the multimodal feature map F multi Design a lightweight semantic segmentation head and output the category probability map P sem ∈R H′×W′×K Then, the category with the highest probability for each pixel is selected as the semantic label to form a semantic mask in, Indicates that the position with the largest probability of the kth category is selected as the predicted category, k∈{1,...,K}; M sem ∈R H′·W′ is the final pixel-level semantic annotation map, based on the obtained semantic label map M sem ∈R H ′×W′ And the semantic probability map P sem ∈R H′×W′×K Fusion of structural categories and classification confidence information to construct a highly expressive local traffic cost map C nav ∈R H′×W′ , and introduces a cost construction function based on category probability weighting, which integrates the semantic category cost value and confidence information to calculate the pass cost of each pixel.

[0032] Furthermore, the optimal path is selected in the local path navigation map based on the path generation mechanism, including first performing a trajectory simulation guided by multimodal perception, assuming that the current position of the robot is s0 = (x0, y0, θ0), where x0, y0 represent the two-dimensional coordinates of the current robot in the cost map plane; θ0 represents the current robot heading angle in radians, and sampling multiple linear velocity and angular velocity combinations (v i ,ωi ) performing trajectory simulation of fixed time T for each pair of combination, simulation step is Δt, trajectory sequence is obtained; each trajectory is mapped into the cost map plane, combined with the passing cost map C nav and semantic mask map M sem Scoring the path:

[0033]

[0034] Wherein, C nav (τ i ) represents the passing cost value of the current position; M sem (τ i ) represents the semantic value of the current position; I obs is the obstacle penalty function; λ1, λ2 are weight coefficients; G(v i ,ω i ) is the trajectory score value.

[0035] Further, the optimal path optimization based on the state error estimation mechanism comprises introducing a state error estimation mechanism to detect whether the trajectory deviation has exceeded the control tolerance range, defining the current robot actual state Wherein represents the actual position and orientation of the t-th step; the position deviation error Δd t and the orientation deviation error Δθ t , when Δd t > ε d or Δθ t > ε θ and continues to occur for more than K steps, it is determined that the current path is invalid, and the reconstruction process is entered; further introducing a control instruction deviation constraint to judge the state stability of the bottom execution system, defining the actual instruction fed back by the current actuator, and calculating the control deviation metric, if Δu t > ε u and continues for M steps, it is determined that the control is mismatched, M is the maximum allowed continuous mismatch step number, once any of the following conditions is triggered: ① Δd t > ε d or Δθ t > ε θ for K steps; ② Δu t > ε u for M steps, the cost map C nav and the semantic mask map M sem are triggered to re-plan the trajectory.

[0036] In a second aspect, a local path planning system for a robot for detecting the bottom of a vehicle based on cross-modal collaborative learning comprises:

[0037] The data acquisition module is configured to acquire multi-modal perception data;

[0038] a preprocessing module configured to perform data preprocessing on the acquired multi-modal perception data;

[0039] a fusion perception module configured to construct a multi-modal fusion perception feature map according to the preprocessed data and based on a cross-modal collaborative modeling mechanism;

[0040] a local path planning module configured to map the multi-modal fusion perception feature map to a local path navigation map based on path planning;

[0041] an optimal path module configured to select an optimal path in the local path navigation map based on a path generation mechanism;

[0042] an optimization module configured to optimize the optimal path based on a state error estimation mechanism.

[0043] In a third aspect, the present application provides a computer-readable storage medium, wherein a plurality of instructions are stored, the instructions being adapted to be loaded and executed by a processor of a terminal device to implement the local path planning method for a car bottom detection robot based on cross-modal collaborative learning.

[0044] In a fourth aspect, the present application provides a terminal device, comprising a processor and a computer-readable storage medium, the processor being configured to implement instructions; and the computer-readable storage medium being configured to store a plurality of instructions, the instructions being adapted to be loaded and executed by the processor to implement the local path planning method for a car bottom detection robot based on cross-modal collaborative learning.

[0045] In summary, the present application has the following beneficial technical effects:

[0046] Compared with the navigation mode of the existing path planning system relying on single modal perception and ignoring semantic factors, the present application has achieved significant breakthroughs in structural modeling, modal collaboration, explainability and actual control response.

[0047] Through modular design, the present system realizes deep fusion of multi-modal data, dynamic mapping of spatial geometry and semantic categories, modal confidence self-adjustment and execution feedback correction, etc. Experimental verification shows that the present method can improve the success rate of path planning tasks from 78.6% to 95.3% in typical car bottom complex scenes, and the average reasoning time of the diagnosis and feedback mechanism is controlled at 0.48 seconds, meeting the edge deployment requirements of car bottom robots.

[0048] In addition, through the semantic structure mapping and explainability output mechanism, a causal explanation consistency score of up to 0.91 is achieved, significantly enhancing the system's ability to explain obstacle identification, path deviation sources and regional planning errors, and being suitable for intelligent navigation tasks in various car bottom scenes, with high practical promotion value and system deployment ability. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 is a schematic diagram of a vehicle bottom detection robot local path planning method based on cross-modal collaborative learning according to embodiment 1 of the present application.

[0050] Figure 2 is a task execution success rate and explanation factor score comparison chart according to embodiment 1 of the present application.

[0051] Figure 3 is a missed detection rate and false alarm rate comparison chart according to embodiment 1 of the present application.

[0052] Figure 4 is a path fitting error and reasoning time comparison chart according to embodiment 1 of the present application.

[0053] Figure 5 is a multi-dimensional performance index normalized radar chart according to embodiment 1 of the present application. DETAILED DESCRIPTION

[0054] The present application will be further described in detail below with reference to the accompanying drawings.

[0055] Embodiment 1

[0056] Referring to Figure 1 , the vehicle bottom detection robot local path planning method based on cross-modal collaborative learning of the present embodiment aims to solve the following key problems in the path planning technology of existing vehicle bottom detection robots in narrow, unstructured, high light interference environments: first, relying only on visual or radar information, it cannot stably extract environmental features in the face of occlusion, heat interference, low light, etc., leading to path failure or repeated planning; second, there is a lack of collaborative mechanism between the perception stage and path generation, and after local error perception, it cannot be compensated or re-planned at the path level; third, existing multi-modal systems have not established a modal confidence feedback mechanism, and cannot dynamically adjust the input weight according to the perception quality, and cannot cope with sudden scenes such as infrared failure and depth map jump; fourth, the path is generated only according to the obstacle position or grid cost, and cannot understand semantic concepts such as "vehicle body", "structural components", "road surface material", etc., and cannot achieve fine navigation; fifth, although some deep reinforcement learning and three-dimensional mapping methods have accuracy, they are structurally long and have poor real-time performance, and are difficult to land on a mobile robot platform, therefore, the present application solves the problem of path planning failure in complex vehicle bottom environments by constructing a cross-modal feature collaborative learning network + semantic enhanced path mapping mechanism + modal confidence feedback control module + edge deployable lightweight path control algorithm, systematically solves the problem of path planning failure in complex vehicle bottom environments, realizes stable, robust and explainable planning output in dynamic, complex and variable scenes, and has high engineering application prospect and cross-field expansion value.

[0057] Specifically, the following contents are included:

[0058] (1) Multimodal perception data acquisition and standardization preprocessing module

[0059] In the actual operational scenarios of underbody inspection robots, the inspection system is often deployed in extremely harsh environments, such as closed and narrow underbody cavities, highly reflective metal structures, surfaces covered with oil, or areas of low illumination and obstruction. To achieve stable and high-precision local path planning, the robotic system must have comprehensive perception capabilities of this complex environment. Therefore, it must rely on the collaborative work of multimodal sensors: RGB cameras capture visible texture information, infrared cameras supplement the identification of heat source areas, depth cameras provide spatial contours, and lidar is used to locate obstacle geometry. Although these modalities are complementary, due to different sampling mechanisms, they are highly heterogeneous in terms of spatial alignment, data density, brightness response scale, and edge structure information.

[0060] Directly inputting these modal data into downstream networks without unified preprocessing will lead to multiple problems: spatial misalignment between modalities can cause confusion in target recognition; significant differences in the dynamic ranges of different modalities can lead to information imbalance; sparse lidar information lacks detail in key areas; and blurred obstacle boundaries weaken the structural recognition capabilities of subsequent models. Therefore, before path planning, a comprehensive perception preprocessing mechanism must be introduced to perform spatial transformation, depth densification, brightness normalization, and edge enhancement on multi-source information. This allows the construction of a perception input tensor with unified scale, spatial alignment, significant structure, and modal synergy, providing a solid data foundation for subsequent cross-modal fusion and dynamic path reasoning.

[0061] 1) LiDAR point cloud projection and depth completion fusion

[0062] To map the sparse LiDAR point cloud data to the pixel space of the RGB image so that it can be aligned with images of other modalities, the present invention first performs extrinsic parameter transformation and perspective projection on each 3D point:

[0063]

[0064] Among them, p k =(x k ,y k ,z k ) T is the position of the lidar point in the radar coordinate system, T is the transpose; R and t are the rigid body transformation parameters (rotation and translation) from the lidar to the camera; is the three-dimensional coordinate of the point in the camera coordinate system; K is the camera internal parameter matrix (including focal length and principal point offset); (u k ,v k ) is the pixel coordinate of the point projected into the image space; It is the depth value of the point from the camera's perspective.

[0065] By the above transformation, all point clouds can be mapped into a sparse depth map M LiDAR where the pixel position (u k ,v k ) stores the corresponding Since the point cloud is sparse, the missed pixels will be empty. To obtain a dense depth map, the present application designs a dense completion algorithm based on nearest neighbor interpolation, and the completion logic is as follows:

[0066] For each empty pixel position (i,j) in the image, the present application searches for the nearest non-empty point (u k ,v k ) within a window of a certain radius r, and assigns its depth:

[0067]

[0068] where r is the interpolation radius, controlling the interpolation smoothness; ‖(i,j)-(u k ,v k )‖2 represents the Euclidean distance between the two points; if ‖(i,j)-(u k ,v k )‖2≤r, it is considered that the point (u k ,v k ) is the available nearest neighbor point, and its depth value can be used to complete (i,j); if there is no hit point in the neighborhood, it remains 0.

[0069] Subsequently, the present application performs weighted fusion of the interpolation completion map and the depth camera image I D :

[0070]

[0071] where α∈[0,1] is the fusion coefficient; is the fused depth map with complete structure. This result not only improves the robustness of the depth map in the reflective area, but also improves the spatial resolution of the scene edge structure.

[0072] 2) Modal normalization processing and numerical scale unification

[0073] The numerical dynamic range of multi-modal images is very different, and direct input into the network will cause learning bias. Therefore, the standardization operation is performed on each modal image so that its mean is 0 and its standard deviation is 1, and the processing formula is as follows:

[0074]

[0075] where IRGB I IR RGB camera, infrared camera image; I i (h, w) is the pixel value of the i-th modal image at position (h, w); is the mean of the multi-modal full image; is the standard deviation of the image; H is the height of the image; W is the width of the image; ε is a numerical stability factor to prevent the denominator from being zero; is the normalized image, the numerical distribution is standard normal, which ensures that each modality has balanced expression weight in the subsequent network, avoiding the phenomenon of single modality dominance.

[0076] 3) Structure edge enhancement

[0077] Because there are a large number of slits, wire grooves, contour joints, bolt protrusions and other details in the vehicle structure, the visual response is weak and easy to be eliminated by the difference between modalities. In order to enhance these local structures, the present application introduces edge saliency enhancement operation on the normalized image.

[0078] First, calculate the Sobel edge gradient map:

[0079]

[0080] where, is the partial derivative along the x direction, which represents the rate of change of image intensity in the horizontal direction (x axis); x is the horizontal coordinate of the image, which represents the position of the pixel in the width direction of the image; is the partial derivative along the y direction, which represents the rate of change of image intensity in the vertical direction (y axis); y is the vertical coordinate of the image, which represents the position of the pixel in the width direction of the image.

[0081] And then weighted fusion with the original image:

[0082]

[0083] where λ is the edge saliency fusion factor; E i is the edge map, which enhances the structure information; is the modal image after edge enhancement; this step can significantly improve the recognition ability of the subsequent network for obstacle boundaries and connecting components.

[0084] 4) Multi-modal input tensor splicing and output construction

[0085] All processed modal images are spliced into a unified input tensor:

[0086]

[0087] Wherein, the tensor comprises: three channels of RGB; one channel of infrared; one channel of depth-laser fusion map.

[0088] Final output F in ∈R H×W×5 Has the following characteristics: all modalities complete geometric mapping and pixel-level registration; all are standard normal distribution, which helps stable model training; edge information is significant, which facilitates structure discrimination; modalities are friendly to fusion: it is convenient to construct modal attention mechanism and semantic alignment representation for subsequent modules.

[0089] (2) Cross-modal collaborative perception feature modeling module

[0090] When the vehicle bottom detection robot performs a local path planning task, it faces complex perception scenarios such as severe changes in light, high-reflectivity materials, small structural gaps, and low-visibility occlusions. Although the previous module provides an input tensor F in ∈R H×W×5 But to achieve stable and robust path planning perception, it is still necessary to further break down the modal barrier at the feature level, construct a fusion representation that is sensitive to fine-grained structural information and has modal redundancy fault tolerance. Therefore, this module designs a cross-modal collaborative modeling mechanism based on double-flow feature extraction + modal guidance enhancement + confidence dynamic fusion, which constructs a perception feature map that stably expresses the semantic and depth profile of the complex structure of the vehicle bottom, providing high-quality input for semantic graph generation and path cost mapping.

[0091] 1) Double-flow feature extraction network design

[0092] In the vehicle bottom detection task, the system needs to process both image modalities (RGB images, infrared images) and structural modalities (depth maps, laser radar projection maps). The two types of modalities have different information density and physical properties, such as RGB providing high-frequency texture details, infrared images sensing heat profiles, depth maps providing geometric position information, and radar point clouds reflecting spatial structures. If a unified backbone network is used to directly process all channels, the semantic conflict and dynamic weight change between modalities will make it difficult for the model to effectively focus on useful features, and instead be misled by interfering modalities. Therefore, in order to avoid feature pollution caused by direct fusion of different modalities, the input tensor F in ∈R H×W×5 is constructed as an image modality branch and a structural modality branch, respectively. Through a double-flow neural network, the deep semantic embedding of each type of modality is extracted, and a clear structure is provided for the initial expression of subsequent modal collaboration.

[0093] RGB image (3 channels) + infrared image (1 channel) are merged into an image modality input:

[0094] F img =F in[:,:,0:4]∈R H×W×4

[0095] where [:,:,0:4] means extracting the first 4 channels (i.e. R, G, B, IR); H, W means the height and width dimension of the image.

[0096] The laser radar fused depth map (1 channel) is the structural modality input:

[0097] F geo = F in [:,:,4:5]∈R H×W×1

[0098] where [:,:,4:5] means extracting the 4th channel (Depth).

[0099] Two lightweight convolutional neural networks are respectively input to extract modality features:

[0100] F′ img = CNN img (F img ), F′ geo = CNN geo (F geo )

[0101] where CNN img is a three-layer convolution-normalization-ReLU module for extracting edge texture and color structure features in RGB+IR; CNN geo is a two-layer light convolutional network for extracting depth profile, distance boundary and other structural expressions

[0102] ; F′ img ∈R H′×W′×C is the image modality feature map, focusing on texture, heat source and edge clues; F′ geo ∈R H′×W′×C is the structural modality feature map, focusing on obstacle profile, spatial concave-convex changes; H′ = H / 4, W′ = W / 4 is the size after downsampling; C is the number of channels. This step ensures that different modalities are extracted stable high-dimensional representation without coupling interference, and provides a basic structure for subsequent attention fusion mechanism.

[0103] 2) Image-guided structural modality cross-modal enhancement

[0104] The double-flow feature of the previous stage retains the modal independence, but there is a semantic offset between the two types of feature spaces, and the structural modal often has difficulty in expressing complete boundaries and context, especially in areas where the light is severely blocked or the radar point cloud is sparse. In order to improve the collaborative expression ability between modalities, the application introduces a "cross-modal attention guidance enhancement mechanism", which uses the information advantage of the image modal to guide the structural modal to strengthen the expression in the spatial position, and constructs a more semantically consistent fusion representation.

[0105] The cross-modal guidance mechanism is designed as follows:

[0106] Q = φ q (F img ), K = φ k (F geo ), V = φ v (F img )

[0107] Wherein, φ q () is a query vector; φ k () is a key vector; φ v () is a value vector

[0108] The three are respectively the features after linear mapping of query, key and value, and then the attention matrix is constructed:

[0109]

[0110] Wherein, · is a dot product operation; d is an embedding dimension; The scaling coefficient prevents numerical explosion; softmax is normalized along the 2nd dimension.

[0111] Subsequently, the attention weight matrix is applied to the image modal feature value:

[0112] F att = A·V

[0113] Then a channel fusion residual strategy is adopted to fuse the information of the two structural modalities:

[0114] F fused = Conv 1×1 (F geo +F att )

[0115] This step can effectively improve the expression ability of the structural modal in weak texture areas and enhance the structural consistency between it and the image modal.

[0116] 3) Modal confidence-aware fusion

[0117] In real vehicle-underground environment, some modalities may fail due to occlusion, water stains, high-temperature reflection, etc. In order to enhance the fault tolerance of the entire system to abnormal modalities, this module introduces a "modality confidence awareness mechanism" to dynamically evaluate the confidence of two types of modalities and adjust the final output features in proportion to ensure that the dominant features come from reliable modal paths.

[0118] First, the output features of the two paths are respectively subjected to global average pooling (GAP):

[0119] g img =GAP(F′ img ),g geo =GAP(F′ geo )

[0120] Then, the confidence is obtained through a small fully connected layer and Sigmoid activation:

[0121] w img =σ(W1·g img ),w geo =σ(W2·g geo )

[0122] Where W1, W2 are the corresponding weight matrices.

[0123] The weighted coefficients are obtained after normalization:

[0124]

[0125] The final fused output feature is:

[0126]

[0127] The final output feature F multi ∈R H′×W′×C has the ability of image texture guidance, depth geometry robustness and dynamic adaptation to modality degradation, providing spatial expression consistent, semantic accurate and fault-tolerant multi-modal joint features for the next module "semantic map construction and path cost encoding".

[0128] (3) Semantic map construction and path cost encoding module

[0129] The vehicle-underground area not only has complex structure and narrow path, but also has a large number of heterogeneous surface materials (such as metal plates, asphalt, brick surfaces, etc.), and various obstacles (such as oil pipes, supports, protruding bolts) are densely distributed therein. Therefore, after multi-modal fusion, it is difficult to construct a navigation structure suitable for path planning with only original features. This module aims to construct a semantic map and encode path cost based on the multi-modal fusion feature F multiThe mapping is a local path map with high semantic resolution, which clearly depicts the passable area, obstacle and structure position of the vehicle body, and introduces a semantic value coding strategy for path planning, dynamically constructs a local navigation map, and provides an environment modeling basis for downstream trajectory generation.

[0130] 1) Region discrimination segmentation based on semantic features

[0131] To realize the fine discrimination of "ground type" and "obstacle type" in the scene, the application is based on the multi-modal feature map F multi Design a lightweight semantic segmentation head to output a class probability map P sem ∈R H′×W′×K ;

[0132] P sem =Softmax(Conv 1×1 (F multi ))

[0133] Where K represents the number of all semantic categories (such as cement pavement, metal bottom plate, brick surface, obstacle, car, truck, etc.); Conv 1×1 () is a channel compression convolution layer, which is used to compress the feature dimension C to the semantic category number K; Softmax is a normalized semantic category probability distribution for each pixel point; P sem (i,j,k) represents the probability of the pixel point (i,j) belonging to the kth class.

[0134] Then, select the class with the maximum probability of each pixel point as the semantic label to form a semantic mask map:

[0135]

[0136] Where, M sem ∈R H′·W′ is the final pixel-level semantic labeling map, each pixel is an integer class label, which is used to locate the environmental elements and exclude the non-passable area; The segmentation map realizes the spatial region semantic analysis, and provides a semantic basis for subsequent path value estimation.

[0137] 2) Construction of local navigation cost map

[0138] In the actual vehicle bottom detection task, even if there is pixel-level semantic annotation, the system still needs to be further converted into a "passing cost" expression to be used for path optimization calculation. Traditional path planning methods rely on simple obstacle detection (such as obstacle grid map of laser radar) to generate passing cost map, but ignore the influence of semantic information in the scene on path cost. For example, "wheel cover" and "metal plate" are similar in geometric structure, but the former is not passable, and the latter can pass slowly; "asphalt" is more slippery and has higher priority than "brick joint". Therefore, it is difficult to fully express complex ground materials, structural states and passing preferences using only geometric features for mapping, which limits the intelligence and adaptability of path generation strategies.

[0139] The module is based on the obtained semantic label map M sem ∈R H′×W′ and the semantic probability map P sem ∈R H′×W′×K fuse the structure category and classification confidence information to construct a local passing cost map C nav ∈R H′×W′ .

[0140] A cost construction function based on category probability weighting is introduced to fuse semantic category value and confidence information to calculate the passing cost of each pixel. The cost map is defined as follows:

[0141]

[0142] where (i,j) is the spatial pixel coordinate on the image; ω k the cost value of the kth category (such as a higher weight for impassable areas); P sem the probability of pixel (i,j) belonging to category k; C nav the passing cost value of the pixel (i,j), which is in the range of [0,1], and the larger the value, the higher the passing cost.

[0143] (4) Local path generation and control execution module

[0144] After completing the semantic map construction and passing cost map generation in module (3), the system obtains the passability evaluation (through the cost map C nav ) and regional semantic attributes (semantic mask map M sem) extraction. However, only map information cannot directly guide the robot movement, and these static environment features must be further converted into continuous and executable path trajectories, and the speed control signal is output for the robot executor to call. Due to the limited space under the vehicle, the dense structure, and the irregular obstacles, simply relying on the traditional DWA trajectory sampling strategy is easy to cause path non-smoothness or local trapping, so the application designs a path generation mechanism based on kinematic trajectory simulation + multi-modal perception cost joint scoring. The mechanism takes the current position of the robot as the starting point, simulates a plurality of candidate trajectories, scores and filters the optimal path in the semantic map and the cost map, and outputs the first frame of motion instruction to the bottom controller.

[0145] 1) Multi-modal perception guided trajectory simulation and scoring mechanism

[0146] If motion trajectory simulation is not performed, and the path is directly predicted by the gradient method or the heuristic method, the system is difficult to evaluate the interaction between the trajectory and the complex obstacle structure, and the path is easy to produce "seemingly reasonable but collision". The trajectory simulation method is to evaluate the path feasibility in advance under the real motion model through forward prediction, which is a key step to generate a robust path.

[0147] Let the current position of the robot be s0=(x0, y0, θ0), wherein x0 and y0 represent the two-dimensional coordinates of the current robot in the cost map plane; θ0 represents the current robot orientation angle, unit is radian, and is defined as the included angle between the vehicle body advancing direction and the horizontal axis.

[0148] The application samples a plurality of linear velocity and angular velocity combinations (v i ,ω i ) to perform trajectory simulation of a fixed time T for each pair of combinations, the simulation step is Δt, and the following trajectory sequence is obtained:

[0149]

[0150] The calculation method of the trajectory point is as follows:

[0151]

[0152] Wherein, v i is the linear velocity sampling value; ω i is the angular velocity sampling value; Δt is the simulation time of each step; θ0+ω i tΔt represents the orientation angle of the trajectory at time t; τ i represents the trajectory path simulated after executing the control pair (v i ,ω i ) from the current state.

[0153] Each trajectory will be mapped into the cost map plane, combined with the passing cost map C navwith semantic mask M sem Score the path:

[0154]

[0155] where C nav (τ i ) represents the passing cost value of the current position; M sem (τ i ) represents the semantic value of the current position; I obs is the obstacle penalty function, outputting 1 when the current position is an obstacle, otherwise 0; λ1, λ2 are weight coefficients, regulating the influence degree of passing cost and semantic penalty; G(v i ,ω i ) is the trajectory score value, the smaller the better;

[0156] 2) Trajectory optimization and bottom control instruction output

[0157] Among all the sampled trajectories, select the one with the smallest score G(v i ,ω i ) as the current execution path, and denote the optimal speed pair as Its corresponding trajectory is

[0158]

[0159] where represents the selection of the smallest execution path.

[0160] Take its first frame control as the output command:

[0161]

[0162] where u cmd is the linear and angular velocity control pair of the robot output at the current time step; τ * is the optimal trajectory path sequence. The control command u cmd output by module (4) is directly used as the running instruction for the bottom motor controller, and the path sequence τ * output at the same time will be passed into module (5) for execution offset monitoring and trajectory re-planning judgment. If the system detects that the trajectory deviation is high or the real-time semantic value surges, it will trigger module (5) to perform emergency path re-planning, thus constructing a complete closed-loop control system.

[0163] (5) Dynamic feedback mechanism and path reconstruction module

[0164] In the vehicle bottom detection task, the robot needs to accurately execute the path trajectory τ * and control sequence u cmd, to complete the obstacle avoidance and chassis area sampling tasks. However, due to the extremely complex environment under the vehicle, such as the visual drift caused by the reflection of metal surfaces, the change of wheel speed caused by slopes, and the frequent occurrence of slip caused by non-rigid terrain (such as soft asphalt), the robot often deviates from the planned trajectory in actual driving state. Without a feedback mechanism to continuously monitor the trajectory execution state, the original path will soon fail, and in severe cases, it will cause collision or detection failure.

[0165] The module is based on the deviation perception mechanism of the planned path and the execution state, combined with the stability analysis mechanism of the control instruction, to dynamically judge whether the path reconstruction needs to be triggered, and to call the module (3) (4) to generate a new path, to realize the system-level closed-loop control and improve the robust adaptive ability.

[0166] 1) Trajectory state consistency monitoring and deviation determination mechanism

[0167] The trajectory output by module (4) Describes the ideal motion state of the robot in the local planning period T (i.e. (x t , y t , θ t ) represents the expected position and attitude angle at step t). But due to the accumulation of execution errors or environmental disturbances, the actual driving trajectory often deviates from the target point, so a state error estimation mechanism must be introduced to detect whether the trajectory deviation has exceeded the control tolerance range.

[0168] Define the actual state of the current robot obtained by odometry (odometry) as:

[0169]

[0170] Where, represents the actual execution position and orientation at step t;

[0171] Construct the position deviation error Δd t and the orientation deviation error Δθ t

[0172]

[0173] When Δd t > ε d or Δθ t > ε θ occurs and lasts for more than K steps, it is determined that the current path is invalid and needs to enter the reconstruction process.

[0174] Where, Δd t is the spatial deviation error; Δθ t is the angle deviation error; ε d is the maximum allowed position deviation threshold; ε θ ​is the maximum allowable heading deviation threshold; K is the step size of the continuous allowable deviation.

[0175] 2) Control stability monitoring and modal fault judgment mechanism

[0176] Even if the position deviation is not significant, if the actual execution of the control command deviates significantly from the planned value (such as slippage, speed loss, or wheel synchronization), the path planning may fail. Therefore, it is necessary to further introduce control command deviation constraints to determine the state stability of the underlying execution system.

[0177] The planning control sequence output by module (4) is:

[0178]

[0179] Among them, v t ,ω t are the planned linear velocity and angular velocity of step t.

[0180] The actual instructions that define the current actuator feedback are:

[0181]

[0182] in, Feedback value for step t;

[0183] The control deviation measure is:

[0184]

[0185] Among them, α v ,α ω is the control deviation weighting parameter, Δu t It is an indicator of control deviation.

[0186] If Δu t >ε u If the control is mismatched and lasts for M steps, it is judged as control mismatch, where M is the maximum number of allowed continuous mismatch steps.

[0187] 3) Path reconstruction and module-level closed-loop feedback process

[0188] Once any of the following conditions is triggered: ①Δd t >ε d or Δθ t >ε θ Continue for K steps; ②Δu t >ε u Continue for M steps, then call the cost graph C output by module (3) nav and semantic mask M sem , triggering module (4) to replan the trajectory.

[0189] Clear the current control instruction cache:

[0190]

[0191] Update the cost map under the current position:

[0192]

[0193] Path generation:

[0194]

[0195] Wherein, P() is the trajectory calculation function of the path generation module (4); τ *new is the newly generated trajectory sequence; is the updated control command; is the current robot state; all newly generated trajectories will be re-input into the low-level control module for tracking.

[0196] Finally, the module (5) outputs the updated trajectory sequence τ *new and the control command stream constitute an execution feedback loop for the module (4). Through position state perception + control deviation monitoring + path regeneration mechanism, the adaptive closed-loop control of various execution disturbances and perception abnormalities in the complex scene under the vehicle is realized, which greatly enhances the system execution stability, fault tolerance ability and intelligent autonomy level.

[0197] Experimental verification

[0198] In order to comprehensively verify the path planning accuracy, perception robustness and execution stability of the application in real complex vehicle bottom scene, a multi-modal perception and navigation dataset of typical vehicles (sedan, SUV, light truck, bus) in a closed vehicle bottom environment is constructed. The dataset collects 540 scene samples distributed in 10 different light conditions, surface material types and obstacle distribution configurations in the vehicle bottom real scene environment, covering various working conditions: RGB and infrared images collected from low lighting, reflection and stain shielding environment; depth map and sparse laser radar point cloud reflecting the vehicle bottom structure and spatial concave-convex distribution; each group of data is labeled with "structure category label" (such as bolt, oil pipe, chassis edge), "surface material type" (asphalt, brick surface, composite board) and "obstacle masking degree"; record the robot execution trajectory, monitor the position drift, target deviation and trajectory termination reason. According to the navigation task execution result, four types of execution state labels are set: ① complete path task (valid trajectory); ② modal misperception leads to deviation; ③ obstacle collision abort; ④ perception degradation failure.

[0199] To evaluate the performance of the system, the following five categories of mainstream path perception and planning methods are compared: ① ResNet-LSTM: based on image texture feature extraction + LSTM controller, no structure perception ability; ② Mono-LiDAR-DWA: based only on laser radar grid map and classic dynamic window algorithm to plan path; ③ RGB-Only YOLO-DWA: uses YOLO to extract obstacle area, and combines cost map to plan path; ④ GraphFusion-DWA: graph neural network models semantic structure relationship, embedded in DWA path calculation; ⑤ the method of the application: multi-modal collaborative perception + double-flow semantic modeling + dynamic semantic navigation cost + adaptive trajectory feedback re-planning

[0200] All methods are evaluated on the same test set, and the comparison indicators include: task execution success rate ACC, i.e. the proportion of complete execution of planning and navigation; the missed detection rate MR, i.e. the proportion of missed detection caused by perception error in path failure; false alarm rate FAR, i.e. the proportion of false alarm obstacles / ground leading to path termination; path deviation mean square error Path-Dev(cm), reflecting the fitting accuracy of trajectory to the ground semantic structure; the average time of single path generation and control execution Time(s); explanation factor and obstacle / region annotation consistency score Interp-Score(0-1).

[0201] Table 1 Comparison of data of different methods in five indicators

[0202]

[0203] Since traditional models such as ResNet-LSTM and single-mode LiDAR-DWA method cannot provide structured semantic factor explanation and modal fault tolerance mechanism, they are evaluated as not applicable (N / A) in the dimension of "explanation accuracy" (Interp-Score). Specifically, ResNet-LSTM only constructs a path control sequence through image information, and its output path only depends on texture regions, which cannot be combined with structure semantic categories for causal explanation; Mono-LiDAR-DWA completely depends on sparse point cloud information to generate a grid map, and has a high failure probability in identifying obstacles in a shaded or reflective environment, lacks perception confidence regulation and modal redundancy mechanism, and cannot provide any semantic judgment or explanation of the cause of navigation failure.

[0204] Due to the large difference in the measurement methods of the six indicators, they cannot be directly displayed in the radar chart, therefore, in the visualization of the indicator evaluation, the following normalization processing is performed: for the positive indicators (ACC, Interp-Score), the original values are kept; for the negative indicators (MR, FAR, Path-Dev, Time), the "1-original value normalization" method is used to convert them into the "the higher the better" form, and the numerical semantics of the radar chart is unified. Considering that the ResNet-LSTM multi-indicators are at the lowest level, the "paranoia compression coefficient epsilon = 0.05" is set, and all the indicator values are compressed to the range [epsilon, 1-epsilon], so that the visible structure is retained in the radar chart, and the single-point or straight-line degradation effect is avoided.

[0205] The experimental results are shown in Table 1, Figure 2 , Figure 3 , Figure 4 , Figure 5 It can be seen that under the complex real vehicle bottom environment, the traditional method has obvious limitations. ResNet-LSTM has no deep modeling capability, and the navigation success rate in the area densely covered with obstacles is only 78.6%, and the path drift mean square error is as high as 10.4 cm, which belongs to the method with the highest misjudgment rate and deviation. Although Mono-LiDAR-DWA has certain navigation ability in clean space, it performs poorly in areas with stains and blurred edges, with a miss rate and a false alarm rate as high as 15.7% and 14.1%, respectively. RGB-YOLO-DWA improves the passable area recognition ability by using the target detection network to assist path calculation, and the accuracy is improved to 86.1%, but the false alarm is high (11.2%) under the condition of heat interference and reflection, and the explanation score is only 0.54, which is difficult to meet the requirements of task-level causal understanding.

[0206] The GraphFusion-DWA method introduces graph structure learning and semantic relationship modeling, so that the path planning is more consistent with the environment structure, the accuracy is improved to 90.4%, the explanation score is 0.77, and the path deviation is 6.1 cm, but due to its high complexity process of two-stage graph construction + path optimization, the processing time is longer, reaching 0.72 seconds, which is not conducive to the real-time response of the narrow scene in the vehicle bottom.

[0207] In contrast, the method of the present application achieves systematic advantage breakthroughs in multiple core dimensions: by means of multi-modal collaborative perception, modal confidence feedback, semantic navigation cost graph and adaptive trajectory control mechanism, the navigation success rate is as high as 95.3%, the miss rate and false alarm rate are compressed to 4.2% and 5.0%, the path deviation is controlled to 4.4 cm, and the processing delay is only 0.48 s, which is the best among all methods. The explanation consistency score is as high as 0.91, which is much higher than that of the splicing model (0.54) and the graph structure method (0.77), fully reflecting the comprehensive ability of the method in complex scene semantic understanding and execution feedback.

[0208] In summary, the method not only achieves superiority in path accuracy and perception robustness, but also maintains optimal performance in real-time, explainability and execution stability, fully verifying its wide adaptability and engineering deployment potential in multi-modal path planning tasks facing complex scenarios under the vehicle.

[0209] Embodiment 2

[0210] The embodiment provides a local path planning system for a vehicle bottom detection robot based on cross-modal collaborative learning, comprising:

[0211] A data acquisition module configured to acquire multi-modal perception data;

[0212] A preprocessing module configured to perform data preprocessing on the acquired multi-modal perception data;

[0213] A fusion perception module configured to construct a multi-modal fusion perception feature map according to the preprocessed data and based on a cross-modal collaborative modeling mechanism;

[0214] A local path planning module configured to map the multi-modal fusion perception feature map to a local path navigation map based on path planning;

[0215] An optimal path module configured to select an optimal path in the local path navigation map based on a path generation mechanism;

[0216] An optimization module configured to optimize the optimal path based on a state error estimation mechanism.

[0217] A computer-readable storage medium having a plurality of instructions stored therein, the instructions being adapted to be loaded and executed by a processor of a terminal device, and implementing a local path planning method for a vehicle bottom detection robot based on cross-modal collaborative learning.

[0218] A terminal device comprising a processor and a computer-readable storage medium, the processor being configured to implement instructions, and the computer-readable storage medium being configured to store a plurality of instructions, the instructions being adapted to be loaded and executed by the processor, and implementing a local path planning method for a vehicle bottom detection robot based on cross-modal collaborative learning.

[0219] The above are preferred embodiments of the present application, and are not intended to limit the protection scope of the present application, therefore: any equivalent changes made in the structure, shape, principle of the present application should be covered within the protection scope of the present application.

Claims

1. A local path planning method for a vehicle underbody inspection robot based on cross-modal collaborative learning, characterized in that: include: Acquire multimodal perception data; Perform data preprocessing on the acquired multimodal perception data; Construct a multimodal fusion perception feature map based on the preprocessed data and the cross-modal collaborative modeling mechanism; Mapping the multimodal fusion perception feature map into a local path navigation map based on path planning; Select the optimal path in the local path navigation graph based on the path generation mechanism; Optimal path optimization is performed based on state error estimation mechanism.

2. The local path planning method for a vehicle underbody inspection robot based on cross-modal collaborative learning according to claim 1 is characterized in that: The data preprocessing of the acquired multimodal perception data includes lidar point cloud projection and depth completion fusion, wherein each 3D point is first subjected to external parameter transformation and perspective projection, and all point clouds are mapped into a sparse depth map M LiDAR , where the pixel position (u k ,v k ) is stored at the corresponding For each empty pixel position (i, j) in the image, the present invention searches for the nearest non-empty point (u k ,v k ), and assign its depth, and then use the interpolation to complete the graph With the depth camera image I D Perform weighted fusion, expressed as: Among them, p k =(x k ,y k ,z k ) T is the position of the lidar point in the radar coordinate system, T is the transpose; R and t are the rigid body transformation parameters (rotation and translation) from the lidar to the camera; is the three-dimensional coordinate of the point in the camera coordinate system; K is the camera internal parameter matrix (including focal length and principal point offset); (u k ,v k ) is the pixel coordinate of the point projected into the image space; It is the depth value of the point from the camera's perspective.

3. The local path planning method for a vehicle underbody inspection robot based on cross-modal collaborative learning according to claim 2 is characterized in that: The data preprocessing of the acquired multimodal perception data also includes modal normalization processing and numerical scale unification, wherein each modal image Perform standardization operation to make its mean 0 and standard deviation 1 for processing, which is expressed as: Among them, I RGB ,I IR is the RGB camera, infrared camera image; I i (h,w) is the pixel value of the i-th modality image at position (h,w); is the multimodal full image mean; is the image standard deviation; H is the height of the image; W is the width of the image; ε is the numerical stability factor to prevent the denominator from being zero; For normalized images, the numerical distribution is standard normal, ensuring that the expression weight of each modality in the subsequent network is balanced and avoiding the dominance of a single modality.

4. The local path planning method for a vehicle underbody inspection robot based on cross-modal collaborative learning according to claim 3 is characterized in that: The data preprocessing of the acquired multimodal perception data also includes introducing an edge saliency enhancement operation on the normalized image to enhance the local structural features of the vehicle bottom structure. First, the Sobel edge gradient map is calculated, which is expressed as: in, yes The partial derivative along the x direction represents the rate of change of the image intensity in the horizontal direction; x is the horizontal coordinate of the image, which represents the position of the pixel in the width direction of the image; yes The partial derivative along the y direction represents the rate of change of the image intensity in the vertical direction; y is the ordinate of the image, which represents the position of the pixel in the width direction of the image, and then weighted fusion with the original image is Among them, λ is the edge significance fusion factor; E i For edge maps, enhance structural information; is the modal image after edge enhancement.

5. The local path planning method for a vehicle underbody inspection robot based on cross-modal collaborative learning according to claim 4 is characterized in that: The multimodal fusion perception feature map is constructed based on the preprocessed data and the cross-modal collaborative modeling mechanism, including the input tensor F in ∈R H×W×5 They are constructed into image modality branches and structural modality branches respectively. The deep semantic embedding of each modality is extracted through a two-stream neural network, and a clear initial expression is provided for subsequent modality collaboration. Among them, the RGB image + infrared image is combined into the image modality input: F img =F in [:,:,0:4]∈R H×W×4 , where [:,:,0:4] means extracting the first four channels, namely R, G, B, IR; H, W represent the image height and width dimensions; the lidar fused depth map is used as the structural modal input, and finally two lightweight convolutional neural networks are input to extract modal features, which can be expressed as: F′ img =CNN img (F img ),F′ geo =CNN geo (F geo ) Among them, CNN img It is a three-layer convolution-normalization-ReLU network; CNN geo is a two-layer light convolutional network; F′ img ∈R H′×W′×C is the image modality feature map; F′ geo ∈R H′×W′×C It is the structural modal characteristic diagram.

6. The local path planning method for a vehicle underbody inspection robot based on cross-modal collaborative learning according to claim 5 is characterized in that: The method constructs a multimodal fusion perception feature map based on the preprocessed data and the cross-modal collaborative modeling mechanism, and also includes introducing a cross-modal attention guidance enhancement mechanism, utilizing the information advantage of the image modality to guide the structural modality to strengthen the expression in spatial position, and construct a more semantically consistent fusion representation. Then, the attention matrix is ​​constructed, the attention weight matrix is ​​applied to the image modality eigenvalues, and the channel fusion residual strategy is adopted to fuse the two structural modal information. The cross-modal guidance mechanism is expressed as: Q=φ q (F′ img ),K=φ k (F′ geo ),V=φ v (F′ img ) Among them, φ q () is the query vector; φ k () is the key vector; φ v () is the value vector, and the three are the features after linear mapping of query, key, and value respectively.

7. The local path planning method for a vehicle underbody inspection robot based on cross-modal collaborative learning according to claim 6, characterized in that: The method of constructing a multimodal fusion perception feature map based on the preprocessed data and the cross-modal collaborative modeling mechanism also includes introducing a modal confidence perception mechanism to dynamically evaluate the confidence of the two types of modalities, and proportionally adjust the final output features to ensure that the dominant features come from a reliable modal path. First, the output features of the two paths are globally averaged and pooled respectively: img =GAP(F′ img ),g geo =GAP(F′ geo ), and then through a small fully connected layer and Sigmoid activation to get the confidence: w img =σ(W1·g img ),w geo =σ(W2·g geo )W1, W2 are the corresponding weight matrices, and the weighting coefficients are obtained after normalization: The final fusion output features are:

8. The local path planning method for a vehicle underbody inspection robot based on cross-modal collaborative learning according to claim 7 is characterized in that: The method maps the multimodal fusion perception feature map to a local path navigation map based on path planning, including mapping the multimodal fusion perception feature map to a local path navigation map based on the multimodal feature map F multi Design a lightweight semantic segmentation head and output the category probability map P sem ∈R H′×W′×K , then, select the category with the highest probability for each pixel as the semantic label to form a semantic mask in, Indicates that the position with the largest probability of the kth category is selected as the predicted category, k∈{1,...,K}; M sem ∈R H′·W′ is the final pixel-level semantic annotation map, based on the obtained semantic label map M sem ∈R H′×W′ And the semantic probability map P sem ∈R H′×W′×K Fusion of structural categories and classification confidence information to construct a highly expressive local traffic cost map C nav ∈R H′×W′ , and introduces a cost construction function based on category probability weighting, which integrates the semantic category cost value and confidence information to calculate the pass cost of each pixel.

9. The local path planning method for a vehicle underbody inspection robot based on cross-modal collaborative learning according to claim 8, characterized in that: The optimal path is selected in the local path navigation map based on the path generation mechanism, including first performing a trajectory simulation guided by multimodal perception, assuming that the current position of the robot is s0 = (x0, y0, θ0), where x0, y0 represent the two-dimensional coordinates of the current robot in the cost map plane; θ0 represents the current robot heading angle in radians, and samples multiple linear velocity and angular velocity combinations (v i ,ω i ) Perform trajectory simulation for a fixed time T on each pair of combinations, with a simulation step of Δt, to obtain a trajectory sequence; each trajectory is mapped to the cost map plane, combined with the pass cost map C nav and semantic mask M sem Score the path: Among them, C nav (τ i ) represents the current location's pass cost value; M sem (τ i ) indicates the semantic value of the current position; I obs is the obstacle penalty function; λ1,λ2 are weight coefficients; G(v i ,ω i ) is the trajectory score value.

10. The local path planning method for a vehicle underbody inspection robot based on cross-modal collaborative learning according to claim 9, characterized in that: The optimal path optimization based on the state error estimation mechanism includes introducing a state error estimation mechanism to detect whether the trajectory offset has exceeded the control tolerance range and define the current actual state of the robot. in Indicates the actual execution position and direction of step t; Structural position offset error Δd t and the orientation offset error Δθ t , when Δd appears t >ε d or Δθ t >ε θ If this continues for more than K steps, the current path is deemed invalid and the reconstruction process is started. The control instruction deviation constraint is further introduced to judge the state stability of the underlying execution system, define the actual instruction fed back by the current actuator, and calculate the control deviation metric. If Δu t >ε u If the control mismatch lasts for M steps, M is the maximum number of mismatch steps allowed. Once any of the following conditions is triggered: ①Δd t >ε d or Δθ t >ε θ Continue for K steps; ②Δu t >ε u Continue for M steps, then call the cost graph C nav and semantic mask M sem , triggering trajectory replanning.

Images