A path planning method and system based on multi-modal joint integration

Abstract

The application discloses a path planning method and system based on multi-modal joint integration, which automatically identifies and excludes the influence of dynamic objects by using image processing and deep learning technology, and ensures that the SLAM and VIO systems only match stable static background objects. The technical scheme is as follows: first, the image feature point matching technology is used to preliminarily screen and match the feature points in the input image. Then, a deep learning network is introduced for fine segmentation and instance recognition. The deep learning network can accurately distinguish between static objects and dynamic objects, and accurately identify the existence of dynamic objects in complex scenes. After identifying the dynamic objects, intelligent exclusion strategies are adopted to exclude these dynamic objects from the feature matching process, ensuring that the SLAM and VIO systems only perform feature matching and positioning calculation based on stable static background objects, avoiding the negative impact of dynamic objects on positioning accuracy, and improving the stability and applicability of the system in complex dynamic environments.

Description

Technical Field

[0001] This invention relates to a path planning method, specifically to a path planning method and system based on multimodal joint integration. Background Technology

[0002] With the rapid development of robotics and autonomous navigation, Simultaneous Localization and Mapping (SLAM) technology has become one of the core technologies in these fields. Over the past decade, SLAM technology has made significant progress and has been widely applied in autonomous driving, drone navigation, industrial robots, and virtual / augmented reality, among other areas. SLAM technology enables autonomous navigation and localization by continuously updating location information and building environmental maps through sensor inputs (such as cameras and LiDAR) without relying on external positioning systems (such as GPS).

[0003] However, with the increasing complexity of real-world applications and the growing demand for high-precision positioning, traditional SLAM technology faces increasing challenges. To address these challenges, researchers have begun combining Visual-Inertial Odometry (VIO) with SLAM technology, proposing Visual-Inertial Simultaneous Localization and Mapping (VI-SLAM). VI-SLAM is an enhanced algorithm that integrates visual sensor data with inertial measurement unit (IMU) data, building upon traditional SLAM. The IMU provides high-frequency acceleration and angular velocity information, enabling short-term position and attitude estimation based on inertial data even when visual information is lacking, obstructed, or under poor lighting conditions. This high-frequency data complements the low-frequency characteristics of visual information, effectively reducing error accumulation. Simultaneously, visual data provides long-term global correction for the IMU, avoiding errors caused by inertial drift and allowing the system to maintain high positioning accuracy over extended periods.

[0004] While SLAM and VIO technologies can provide high-precision localization and mapping in static environments, the presence of dynamic objects in the real world is a significant problem. Dynamic objects, such as pedestrians and vehicles, can interfere with the matching of image feature points, causing SLAM systems to mistakenly treat dynamic objects as static backgrounds, leading to errors in position estimation and map updates. This interference from dynamic objects is particularly severe in complex urban environments, significantly reducing the robustness and accuracy of the system.

[0005] Therefore, how to solve the problem of interference from dynamic objects in order to improve the applicability and positioning accuracy of SLAM and VI-SLAM systems in complex dynamic environments is an urgent problem to be solved in the industry. Summary of the Invention

[0006] The following provides a brief overview of one or more aspects to offer a basic understanding of them. This overview is not an exhaustive summary of all conceived aspects, nor is it intended to identify key or decisive elements of all aspects, nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form to prepare for the more detailed descriptions that follow.

[0007] The purpose of this invention is to solve the above problems and provide a path planning method and system based on multimodal joint integration. By utilizing image processing and deep learning techniques, the method automatically identifies and eliminates the influence of dynamic objects, thereby ensuring that SLAM and VIO systems only match stable static background objects, thus improving the overall performance and positioning accuracy of the system.

[0008] The technical solution of this invention is as follows: This invention discloses a path planning method based on multimodal joint integration, the method comprising:

[0009] Step S1: Image preprocessing and preliminary screening of dynamic regions, including image preprocessing, extraction of image feature points, and matching the image at time t-1 with the image at time t to preliminarily screen out dynamic regions;

[0010] Step S2: Based on loose coupling, perform online calibration and initialize the motion state matrix;

[0011] Step S3: Region fine segmentation and instance recognition, including after the motion state matrix is ​​initialized, the subsequent images from the vehicle camera are fed into the SAM network as input to obtain the region fine segmentation output, which is then input into the CLIP network to obtain the instance ID corresponding to the refined region. The output result is saved, and the targets at time t-1 and time t+1 are compared. If there are targets with the same static target instance ID, the process proceeds to step S4. If there are no targets with the same static target instance ID, the instantaneous inertial measurement unit data is used as the motion data of the frame to proceed to step S6.

[0012] Step S4: Based on the target with the matched static target instance ID, group and map them, and finally calculate the average rotation matrix;

[0013] Step S5: Fuse the average rotation matrix with the data from the n inertial measurement units before and after the frame to obtain the final state matrix, where n is a natural number greater than or equal to 2;

[0014] Step S6: Save the state matrix and draw the global route.

[0015] According to an embodiment of the path planning method based on multimodal joint integration of the present invention, step S1 further includes:

[0016] Step S11: Preprocess the captured image, including statistical analysis of the frequency distribution of image pixel values, and improve contrast and visibility by equalization processing to make the pixel distribution more uniform.

[0017] Step S12: Use Shi-Tomasi to detect feature points in the image;

[0018] Step S13: Match the feature points of the image at time t-1 with the image at time t, and then estimate the movement of the image feature points in consecutive frames by analyzing the movement of pixels in the image sequence, thereby tracking the key points and finally obtaining the dynamic area heat map relative to the car.

[0019] According to an embodiment of the path planning method based on multimodal joint integration of the present invention, step S2 further includes:

[0020] Step S21: Combine the sensing information from the inertial measurement unit to perform loosely coupled online calibration and align information including velocity and scale;

[0021] Step S22: Initialize the motion state matrix of the object.

[0022] According to an embodiment of the path planning method based on multimodal joint integration of the present invention, step S3 further includes:

[0023] Step S31: Input the image to be segmented;

[0024] Step S32: Use an image encoder to extract high-dimensional feature representations of the input image and output an image feature map;

[0025] Step S33: Encode the user-provided prompts into feature representations using a prompt encoder, and output the prompt features, wherein the user-provided prompts include points, boxes, and text descriptions;

[0026] Step S34: Feature fusion, which involves fusing image features and cue features together through feature concatenation or attention mechanisms, and outputting fused features to prepare for subsequent mask decoding;

[0027] Step S35: Using a mask decoder based on the Transformer architecture, a segmentation mask is generated and output according to the fusion features. In the segmentation mask, the value of each pixel represents the probability that the pixel belongs to the target object.

[0028] Step S36: Perform optimization operations on the segmentation mask to generate the final refined segmentation result, i.e., the refined segmentation mask. The optimization operations include thresholding and morphological operations.

[0029] Step S37: Combine the segmentation mask with the original image to extract the image region of the target object;

[0030] Step S38: Input the image region of the target object into an image-text model to generate the image's feature vector;

[0031] Step S39: Match the feature vector of the image with the feature vector of the predefined text label, use a similarity metric to find the most similar text label and output it.

[0032] According to an embodiment of the path planning method based on multimodal joint integration of the present invention, in step S32, the image encoder uses a pre-trained visual model to realize feature extraction, wherein the visual model is a multi-scale visual model based on CrossVIT, which extracts and fuses information through image patches of different sizes to improve the detection capability of targets at different scales.

[0033] According to an embodiment of the path planning method based on multimodal joint integration of the present invention, step S32 further includes:

[0034] Step S321: Obtain a one-dimensional vector representation by encoding with a block encoder and output it;

[0035] Step S322: Obtain category vectors through a category encoder to integrate image information for classification;

[0036] Step S323: Obtain a position vector through position encoding to express the position information of the image patch in the original image;

[0037] Step S324: Extract features and aggregate information from the input sequence using a Transformer encoder.

[0038] According to an embodiment of the path planning method based on multimodal joint integration of the present invention, step S34 further includes:

[0039] Step S341: Project the image and text features into the same embedding space through a linear projection layer for feature alignment;

[0040] Step S342: By contrastive learning loss function, matching text image features are brought closer in the embedding space, while non-matching features are moved further away.

[0041] According to an embodiment of the path planning method based on multimodal joint integration of the present invention, step S4 further includes:

[0042] Step S41: Calculate the covariance matrix according to the matching target;

[0043] Step S42: Obtain the rotation matrix using singular value decomposition;

[0044] Step S43: Calculate the mean of the rotation matrix for all targets.

[0045] According to an embodiment of the path planning method based on multimodal joint integration of the present invention, step S41 further includes:

[0046] Step S411: Obtain the centroid through the acquired matching feature points and decentroid them;

[0047] Step S412: Calculate the centroid-decentered covariance matrix. The covariance matrix represents the changing trend of matching point pairs between consecutive frames and is used to infer the rotation relationship between consecutive frames.

[0048] According to an embodiment of the path planning method based on multimodal joint integration of the present invention, step S43 further includes:

[0049] Step S431: For each target, convert the rotation matrix into a quaternion;

[0050] Step S432: Calculate a weighted average of all quaternions to obtain an average quaternion;

[0051] Step S433: Convert the quaternion into a rotation matrix form to obtain the average rotation matrix.

[0052] According to an embodiment of the path planning method based on multimodal joint integration of the present invention, in step S5, the average rotation matrix obtained in step S4 is used together with the integral of at least two inertial measurement unit data before and after the frame to calculate the observed motion matrix. After one weighting, the motion state matrix of the object is updated by the weighted motion state value to obtain the predicted motion matrix. The predicted value at time t+1 and the actual value at time t+1 in the predicted motion state matrix are weighted again to obtain the output value.

[0053] This invention also discloses a path planning system based on multimodal joint integration, the system comprising:

[0054] The dynamic region preliminary screening module is configured for image preprocessing and dynamic region preliminary screening, including image preprocessing, extracting image feature points, and performing a match between the image at time t-1 and the image at time t to initially screen out dynamic regions;

[0055] The motion state matrix initialization module is configured for loosely coupled online calibration and performs motion state matrix initialization.

[0056] The region fine segmentation and instance recognition module is configured such that after the motion state matrix is ​​initialized, subsequent images from the vehicle-mounted camera are fed into the SAM network as input to obtain the region fine segmentation output, which is then input into the CLIP network to obtain the instance ID corresponding to the refined region. The output result is saved, and the targets at time t-1 and time t+1 are compared. If there are targets with the same static target instance ID, the module enters the average rotation matrix calculation module. If there are no targets with the same static target instance ID, the instantaneous inertial measurement unit data is used as the motion data of the frame and entered into the global route drawing module.

[0057] The average rotation matrix calculation module is configured to group targets based on the matched static target instance IDs, perform mapping, and finally calculate the average rotation matrix.

[0058] The state matrix acquisition module is configured to fuse the average rotation matrix with the data from n inertial measurement units before and after the frame to obtain the final state matrix, where n is a natural number greater than or equal to 2.

[0059] The global route drawing module is configured to save the state matrix and draw the global route.

[0060] The present invention also discloses a computer system for path planning, including a memory, a processor, and program instructions stored in the memory that are executable by the processor, wherein the processor executes the program instructions to implement the steps of the path planning method based on multimodal joint integration as described above.

[0061] The present invention also discloses a computer-readable storage medium for path planning, characterized in that it stores program instructions executable by a processor to implement the steps of the path planning method based on multimodal joint integration as described above.

[0062] The present invention also discloses a computer program product, comprising a computer program that, when executed by a processor, implements the steps of the path planning method based on multimodal joint integral as described above.

[0063] Compared with existing technologies, this invention offers the following advantages: It aims to solve the interference problem caused by dynamic objects to SLAM (Simultaneous Localization and Mapping) and VIO (Visual Inertial Odometry) systems, thereby improving positioning accuracy and system applicability in complex dynamic environments. To this end, this invention proposes an innovative method that cleverly integrates image processing and deep learning technologies to achieve automatic identification and exclusion of dynamic objects.

[0064] Specifically, this invention first utilizes image feature point matching technology to perform preliminary screening and matching of feature points in the input image. Subsequently, a deep learning network is introduced for refined segmentation and instance recognition. This deep learning network, after extensive training, can accurately distinguish between static and dynamic objects, thereby precisely identifying the presence of dynamic objects in complex scenes.

[0065] Upon identifying dynamic objects, this invention employs an intelligent exclusion strategy. The system excludes these dynamic objects from the feature matching process, ensuring that SLAM and VIO systems perform feature matching and localization calculations solely based on stable, static background objects. This strategy effectively avoids the negative impact of dynamic objects on localization accuracy, while simultaneously improving the system's stability and applicability in complex dynamic environments.

[0066] Through the above technical solutions, this invention not only significantly improves the positioning accuracy of SLAM and VIO systems, but also expands their application scope in complex dynamic environments, providing more reliable and accurate positioning and navigation support for fields such as robotics, autonomous driving, and drone navigation.

[0067] In detail, the main advantages of this invention compared to conventional technology are:

[0068] 1. Multimodal data fusion and IMU data correction:

[0069] Compared to traditional SLAM (Simultaneous Localization and Mapping) technology, this invention introduces a visual fusion component, achieving deep integration of visual sensor and IMU (Inertial Measurement Unit) data. Through the combined effect of multimodal data, IMU data offsets are effectively corrected, significantly improving the system's positioning accuracy and stability. This fusion strategy not only enhances the system's robustness but also enables it to perceive and respond more accurately in complex environments.

[0070] 2. Application of Deep Learning Algorithms in Dynamic Object Recognition:

[0071] Compared to traditional V-SLAM (Visual Simultaneous Localization and Mapping) technology, this invention innovatively employs deep learning algorithms to perform more refined segmentation and recognition of dynamic and static objects in images. This improvement enables the system to calculate the vehicle's motion state more accurately, avoiding interference from dynamic objects in the localization results. Through the introduction of deep learning technology, the system of this invention exhibits higher localization accuracy and applicability in complex dynamic environments.

[0072] 3. Unsupervised target region segmentation and splitting capability:

[0073] Compared to traditional supervised object detection algorithms, the algorithm structure proposed in this invention can still segment and decompose target regions even in unsupervised environments. This characteristic greatly reduces the algorithm's dependence on labeled data, improving its flexibility and generalization ability. Within the framework of unsupervised learning, this invention can adaptively handle various complex scenarios, further enhancing the system's practicality and reliability. Attached Figure Description

[0074] The above-described features and advantages of the present invention will be better understood after reading the following detailed description of embodiments of the present disclosure in conjunction with the accompanying drawings. In the drawings, components are not necessarily drawn to scale, and components having similar related characteristics or features may have the same or similar reference numerals.

[0075] Figure 1 A flowchart of an embodiment of the path planning method based on multimodal joint integration of the present invention is shown.

[0076] Figure 2 It shows Figure 1 The diagram shows the model architecture of the multi-scale visual model in the method embodiment shown.

[0077] Figure 3 A schematic diagram of an embodiment of the path planning system based on multimodal joint integration of the present invention is shown. Detailed Implementation

[0078] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. It should be noted that the aspects described below with reference to the accompanying drawings and specific embodiments are merely exemplary and should not be construed as limiting the scope of protection of the present invention in any way.

[0079] Figure 1 The flowchart of an embodiment of the path planning method based on multimodal joint integration of the present invention is shown. Please refer to... Figure 1 The implementation steps of the method in this embodiment are described in detail below.

[0080] Step S1: Image preprocessing and preliminary screening of dynamic regions.

[0081] The image is preprocessed to extract feature points. The image at time t-1 is matched with the image at time t, and the vehicle itself is used as a reference frame to initially screen out dynamic regions.

[0082] Step S1 further includes the following processing procedures:

[0083] Step S11: Preprocess the image captured by the camera, including statistical analysis of the frequency distribution of image pixel values, and equalization processing to make the pixel distribution more uniform, improve contrast and visibility, so as to facilitate subsequent feature extraction and reduce brightness changes between consecutive frames.

[0084] Step S12: Extract image feature points from the image based on the Shi-Tomasi corner detection algorithm.

[0085] Step S12 further includes the following processing procedures:

[0086] To detect feature points in an image using Shi-Tomasi, for the input image, the gradients in the x and y directions of each pixel are first calculated, yielding derivatives in these two directions. These derivatives represent the rates of change in the x and y directions. For each pixel, an autocorrelation matrix is ​​constructed by weighted summation of the gradients in the x and y directions. Then, the two eigenvalues ​​of this autocorrelation matrix are calculated, and the smaller eigenvalue is used as the response value R for the corner point. Finally, all R values ​​are statistically analyzed, and the largest R value is calculated by multiplying it by a value k, where k is a set response value, a threshold factor used to select feature points. This factor influences the number and quality of detected feature points. If the response value R is greater than the threshold, it is considered an image feature point.

[0087] Step S13: Feature point matching between the image at time t-1 and the image at time t refers to calculating the Euclidean distance between the feature point sets in the images at time t-1 and time t respectively. If a matching point exists within the set distance, it is considered a successful match. Then, by analyzing the movement of pixels in the image sequence, the movement of image feature points in consecutive frames is estimated, thereby tracking key points and finally obtaining a dynamic area heat map relative to the vehicle.

[0088] Specifically, each matching step involves matching the image feature points obtained in step S12 with the image feature points obtained from feature detection in the previous frame. For each image feature point in the previous frame, each image feature point in the current frame is traversed one by one, and the Euclidean distance between them is calculated. The image feature point with the smallest distance is selected as the matching point. If the distance between the matching points is less than a set threshold, then the matching point is a valid matching point; otherwise, the matching point is discarded. The matching formula is as follows: Where x1 and x2 are the x-coordinates (horizontal coordinates) of the preceding and following frames, respectively; and y1 and y2 are the y-coordinates (vertical coordinates) of the preceding and following frames, respectively. After a successful match, the feature points of that image are added to the tracking sequence.

[0089] Based on the historical motion information of image feature points in the tracking sequence, and taking advantage of the characteristic that image feature points move relatively little in local regions within consecutive frames, motion vectors are calculated by combining temporal and spatial gradients. This yields the estimated predicted position of the image feature points in the next frame, allowing for matching near the predicted position in the next frame. If a valid match is found, tracking continues and the latest motion information is updated; otherwise, the match is discarded.

[0090] The key points tracked are image feature points, which are certain local regions in an image that can be stably and uniquely detected and described. Image feature points are generally located in salient areas of the image (such as corners, edges, spots, etc.), and these regions remain relatively unchanged under image scaling, rotation, or lighting changes.

[0091] The generation of dynamic region heatmaps involves calculating a motion vector for each image feature point to represent its displacement between frames. If the motion vector of an image feature point is less than a defined threshold, it is considered a static point; otherwise, it is considered a dynamic point, indicating that it belongs to a dynamic object. These dynamic points are then selected as dynamic regions in the image.

[0092] Step S2: Loosely coupled online calibration and motion state matrix initialization.

[0093] Step S2 further includes the following processing:

[0094] Step S21: Combine the sensing information from the IMU (Inertial Measurement Unit) to perform loosely coupled online calibration, aligning information such as velocity and scale. Loose coupling refers to processing image data and IMU data independently before fusing them. Online calibration refers to adjusting parameters in real time during system operation to ensure data accuracy.

[0095] Step S22: Initialize the motion state matrix of the object. The motion state matrix has the shape of n*m, where n corresponds to the time and m corresponds to the state information at different times, including but not limited to the motion speed, motion direction and other motion information.

[0096] Step S3: Refined region segmentation and instance identification.

[0097] After initializing the motion state matrix, subsequent images from the vehicle-mounted camera are fed into the SAM (SegmentAnything Model) network as input to obtain refined region segmentation output (corresponding to subsequent steps S31-S37). Then, the images are input into the CLIP (Contrastive Language-Image Pre-training) network (corresponding to subsequent steps S38-S39) to obtain the instance IDs corresponding to the refined regions. The output results are saved, and the targets at time t-1 and time t+1 are compared. If there are targets with the same static target instance IDs, the process proceeds to step S4. If there are no targets with the same instance IDs, the instantaneous IMU data is used as the motion data for that frame, and the process proceeds to step S6.

[0098] Step S3 further includes the following processing procedures.

[0099] Step S31: Input the image to be segmented.

[0100] Step S32: Use an image encoder to extract high-dimensional feature representations of the input image and output an image feature map.

[0101] Image encoders typically use pre-trained vision models, such as Vision Transformer (ViT) or high-performance convolutional neural networks (CNN), to perform feature extraction.

[0102] refer to Figure 2 As shown, in this embodiment, the visual model is a multi-scale visual model based on CrossVIT, which extracts and fuses information through image patches of different sizes to improve the detection capability of targets at different scales.

[0103] The training of a multi-scale visual model includes the following process.

[0104] Step 1: For the input image, first use the DS2N (Deep Separable Convolutional Bottleneck unit with 2times) module, then the DSP3 module, and then use the DS2N module again to extract features and obtain the feature map.

[0105] in:

[0106] LeakyReLU (Leaky Linear Rectified Function) is used as an activation function to perform a non-linear transformation on the input data to enhance the expressive power of the model. Compared with the traditional ReLU, LeakyReLU allows negative values ​​of the input to pass through at a smaller proportion, thereby alleviating the problem of neuron death.

[0107] DSCBS (Depthwise Separable Convolutional Block with BatchNormalization and SILU) is a combined module of depthwise separable convolution (DSC), batch normalization (BN), and SILU (Sigmoid-weighted Linear Unit).

[0108] Bottleunit is a bottleneck unit that uses a combination of two DSCBS modules;

[0109] DSP3 (Depthwise Separable Convolutional Pyramid with 3 levels) is a module that extracts features from the original feature map at three different depths and then concatenates them.

[0110] DS2N is a module that extracts features from the original feature map using N (at least 2) Bottleunits and extracts features using DSCBS once, and then concatenates them together.

[0111] The Add operation adds the output of the main branch and the output of the skip links to form a residual map. This mechanism can alleviate the gradient vanishing problem and promote feature fusion to improve the model's feature extraction capabilities.

[0112] The second step: After feature extraction in the first step, the output feature map is divided into small and large windows. For example, a feature map may be divided into 8*8 blocks or 16*16 blocks. That is, small and large patches are used to extract targets of different sizes more accurately.

[0113] Step 3: Input the ViT (Vision Transformer) network (see steps S321-S324)

[0114] Step 4: Finally, use `cat(Concatenate, concatenation operation)` to output the final result.

[0115] Step S32 is further refined into the following processing procedure.

[0116] Step S321: Obtain a one-dimensional vector representation by encoding with a block encoder and output it. Specifically, first divide the image into P*P (e.g., 16*16) blocks, and then flatten each block into a one-dimensional vector.

[0117] Step S322: Obtain category vectors through a category encoder to integrate image information for classification.

[0118] Specifically, a learnable image patch embedding vector is used as a classification label and added to the front of the image patch sequence.

[0119] Step S323: Obtain the position vector through position encoding to express the position information of the image patch in the original image.

[0120] Specifically, a learnable location embedding vector is assigned to each image patch to ensure that the location information of each patch is preserved. The location embedding vector is then added element-wise to the image patch embedding vector to complete the encoding.

[0121] Step S324: Perform feature extraction and information aggregation on the input sequence (i.e., the encoding output in step S323) using the Transformer encoder.

[0122] Specifically, the process involves: first, layer normalization is performed to reduce range fluctuations compared to batch normalization, and then normalization is applied within the sample. Feature extraction is then performed using a multi-head attention mechanism. Finally, a residual structure is used for the final output. These steps are repeated multiple times to extract features. Finally, a fully connected layer is used with a non-linear activation function for the final output.

[0123] Step S33: Encode the user-provided prompts (such as dots, boxes, and text descriptions) into feature representations using a prompt encoder, and output the prompt features.

[0124] The specific implementation includes:

[0125] Hint: Use the geometric encoding method to encode the position of the point as a feature. Here, the input is the center point of each dynamic region in step S1.

[0126] Text hint: Use a pre-trained text encoder to encode text descriptions as features.

[0127] Step S34: Feature fusion, that is, fusing image features and cue features together through feature splicing or attention mechanism, outputting fused features to prepare for subsequent mask decoding.

[0128] Step S34 further includes the following processing procedure.

[0129] Step S341: Project the image text features into the same embedding space through a linear projection layer for feature alignment.

[0130] Step S342: The contrastive learning loss function (InfoNCE) is used to bring matching text image features closer together in the embedding space, while those that do not match are moved further away.

[0131] In step S342, based on RINCE and ReLICv2, the innovative RCILoss (Robust Causal Invariant Loss) is introduced. RINCE emphasizes the robustness of negative sample selection, while ReLIC2 enhances the robustness and generalization of the representation.

[0132] The advantages of RCILoss are:

[0133] Robustness Enhancement: The negative sample selection strategy of RINCE is incorporated to enhance the model's robustness to data imbalance and noise.

[0134] Local structure preservation: Borrowing from the ideas of ReLICv2, this ensures that the model can capture local structure, making it suitable for scenarios with complex local information.

[0135]

[0136] RCILoss=αL1+βL2

[0137] Where the sum of a and b is 1, the default values ​​in the early stages of training are 0.8 and 0.2, and the values ​​of a and b are dynamically adjusted according to the training effect; q represents the control parameter, which adjusts the behavior of the loss function;

[0138] S represents the similarity score of the feature vectors.

[0139] δ+ represents the similarity score of positive samples.

[0140] δi- represents the similarity score of the i-th negative sample, that is, the degree of similarity between the target feature and the negative sample;

[0141] K represents the number of negative samples;

[0142] x i The input features for the current sample,

[0143] X i + For the current sample, a positive sample

[0144] P(x i ) represents the set of positive samples.

[0145] α is a weight hyperparameter used to balance the impact of the joint distribution loss of positive samples on the total loss;

[0146] P(x i ;x i+ ) represents the joint probability distribution used to quantify x. i and x i + Relationship;

[0147] The β-weight hyperparameter is used to control the contribution of KL divergence to the total loss.

[0148] P(x i The sample xi distribution,

[0149] P(x + Positive sample distribution,

[0150] DKL is the KL divergence, used to measure p(x) i ) and p(x + The difference between ).

[0151] Step S35: Using a mask decoder based on the Transformer architecture, a segmentation mask is generated and output based on the fusion features. In the segmentation mask, the value of each pixel represents the probability that the pixel belongs to the target object.

[0152] Step S36: Perform optimization operations on the segmentation mask (such as thresholding, morphological operations, etc.) to generate the final refined segmentation result, i.e., the refined segmentation mask.

[0153] Step S37: Combine the segmentation mask with the original image to extract the image region of the target object. This includes: using the segmentation mask to extract the image region of the target object from the original image, for example, by accurately extracting the target region through mask binarization and morphological operations.

[0154] Step S38: Input the image region of the target object into an image-text model to generate the feature vector of the image.

[0155] Step S39: Match the feature vector of the image with the feature vector of the predefined text label, use a similarity metric (such as cosine similarity) to find the most similar text label and output it.

[0156] Step S3 is used to segment static and dynamic objects in an unsupervised manner.

[0157] Step S4: Based on the target of the matched instance ID, group and map them, and finally calculate the average rotation matrix, which is the motion state at this time.

[0158] Step S4 further includes the following processing procedures.

[0159] Step S41: Calculate the covariance matrix according to the matching target.

[0160] Step S41 further includes:

[0161] Step S411: Obtain the centroid through the acquired matching feature points and decentroid them;

[0162] Step S412: Calculate the centroid-decentered covariance matrix. The covariance matrix represents the changing trend of matching point pairs between frames, which can be used to infer the rotation relationship between frames.

[0163] Step S42: Obtain the rotation matrix by using SVD (Singular Value Decomposition).

[0164] Step S42 further includes:

[0165] Step S421: Perform singular value decomposition using the covariance matrix obtained in step S41. The formula is as follows: U and V are orthogonal matrices, ε is a diagonal matrix containing the singular values ​​of the covariance matrix, and H represents the covariance matrix.

[0166] H=UεV T

[0167] Step S422: After SVD decomposition, calculate the rotation matrix R. The formula is as follows, representing the rotation matrix between the previous and next frames: R = VU T .

[0168] Step S43: Calculate the mean of the rotation matrix for all targets.

[0169] Step S43 further includes:

[0170] Step S431: For each target, convert the rotation matrix into a quaternion for better computation;

[0171] Step S432: Calculate a weighted average of all quaternions to obtain an average quaternion;

[0172] Step S433: Convert the quaternion into a rotation matrix form to obtain the average rotation matrix.

[0173] Step S5: Fuse the average rotation matrix with the n IMU data before and after the frame to obtain the final state matrix, where the value of n is greater than or equal to 2.

[0174] The average rotation matrix obtained in step S4 is used together with the integrals of at least two IMU data points before and after the frame to calculate the observed motion matrix. After one weighting, the motion state matrix of the object is updated using the weighted motion state values ​​to obtain the predicted motion matrix. The predicted value at time t+1 and the actual value at time t+1 in the predicted motion state matrix are then weighted again to obtain the output value, as shown in the following formula:

[0175] Status ti =α·imu ti +β·img ti +γ·predict ti

[0176] Where α, β, and γ are weight hyperparameters, and their sum is 1; imu represents the calculated value of imu at time ti; img represents the calculated image value at time ti; predict represents the Kalman filter prediction value at time ti; and Status... ti This represents the final motion state matrix.

[0177] Step S6: Save the state matrix and draw the global route.

[0178] That is, save the current motion state and combine it with the historical motion trajectory to draw the global motion state.

[0179] Figure 3 The principle of an embodiment of the path planning system based on multimodal joint integration of the present invention is shown. Please refer to... Figure 3 The system in this embodiment includes the following modules: dynamic region preliminary screening module, motion state matrix initialization module, region fine segmentation and instance recognition module, average rotation matrix calculation module, state matrix acquisition module, and global route drawing module.

[0180] The dynamic region preliminary screening module is configured for image preprocessing and dynamic region preliminary screening. This includes image preprocessing, extracting image feature points, and performing a match between the image at time t-1 and the image at time t to initially screen out dynamic regions. The specific processing of this module is as follows... Figure 1 The steps S1 shown are the same and will not be repeated here.

[0181] The motion state matrix initialization module is configured for loosely coupled online calibration and performs motion state matrix initialization. The specific processing of this module is related to... Figure 1 The steps shown in step S2 are the same and will not be repeated here.

[0182] The region fine-grained segmentation and instance recognition module is configured such that, after the motion state matrix initialization is complete, subsequent images from the vehicle-mounted camera are fed into the SAM network as input to obtain the region fine-grained segmentation output. This output is then fed into the CLIP network to obtain the instance IDs corresponding to the refined regions. The output results are saved, and the targets at time t-1 and time t+1 are compared. If targets with the same static target instance ID exist, the module proceeds to the average rotation matrix calculation module. If no targets with the same static target instance ID exist, the instantaneous inertial measurement unit data is used as the motion data for that frame and fed into the global route drawing module. The specific processing of this module is as follows... Figure 1 The steps shown in step S3 are the same and will not be repeated here.

[0183] The average rotation matrix calculation module is configured to group targets based on their matched static target instance IDs, perform mapping, and finally calculate the average rotation matrix. The specific processing of this module is similar to... Figure 1 Step S4 is the same as shown, and will not be repeated here.

[0184] The state matrix acquisition module is configured to fuse the average rotation matrix with data from n inertial measurement units before and after the frame to obtain the final state matrix, where n is a natural number greater than or equal to 2. The specific processing of this module is as follows... Figure 1 Step S5 is the same as shown, and will not be repeated here.

[0185] The global route drawing module is configured to save the state matrix and draw the global route. The specific processing of this module is... Figure 1 Step S6 is the same as shown, and will not be repeated here.

[0186] Furthermore, this invention also discloses a computer system for path planning, the system including a memory, a processor, and program instructions stored in the memory that are executable by the processor, wherein the processor executes the program instructions to implement, for example... Figure 1 The steps of an embodiment of the path planning method based on multimodal joint integration are shown below.

[0187] Furthermore, this invention also discloses a computer-readable storage medium for path planning, which stores program instructions executable by a processor for performing tasks such as... Figure 1 The steps of an embodiment of the path planning method based on multimodal joint integration are shown below.

[0188] Furthermore, this invention also discloses a computer program product, including a computer program, which, when executed by a processor, implements as follows: Figure 1 The steps of an embodiment of the path planning method based on multimodal joint integration are shown below.

[0189] Although the methods described above are illustrated and depicted as a series of actions for the sake of simplicity, it should be understood and appreciated that these methods are not limited by the order of the actions, as some actions may occur in a different order and / or concurrently with other actions from the illustrations and descriptions herein or not illustrated and described herein but which may be understood by those skilled in the art, according to one or more embodiments.

[0190] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps are described above in a generalized manner in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the invention.

[0191] The various illustrative logic blocks, modules, and circuits described in conjunction with the embodiments disclosed herein can be implemented or performed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but in alternatives, it may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration.

[0192] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of both. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor such that the processor can read and write information to / from the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and storage medium may reside as discrete components in the user terminal.

[0193] In one or more exemplary embodiments, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functionality may be stored or transmitted as one or more instructions or code on or through a computer-readable medium. A computer-readable medium includes both computer storage media and communication media, encompassing any medium that facilitates the transfer of a computer program from one location to another. A storage medium may be any available medium accessible to a computer. By way of example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and is accessible to a computer. Any connection is also legitimately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of a medium. As used in this article, disk and disc include compact discs (CDs), laser discs, optical discs, digital multi-purpose discs (DVDs), floppy disks, and Blu-ray discs. Disks typically reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of these should also be included within the scope of computer-readable media.

[0194] The prior description of this disclosure is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not intended to be limited to the examples and designs described herein, but should be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A path planning method based on multimodal joint integration, characterized in that, The methods include: Step S1: Image preprocessing and preliminary screening of dynamic regions, including image preprocessing, extraction of image feature points, and matching the image at time t-1 with the image at time t to preliminarily screen out dynamic regions; Step S2: Based on loose coupling, perform online calibration and initialize the motion state matrix; Step S3: Region fine segmentation and instance recognition, including after the motion state matrix is ​​initialized, the subsequent images from the vehicle camera are fed into the SAM network as input to obtain the region fine segmentation output, which is then input into the CLIP network to obtain the instance ID corresponding to the refined region. The output result is saved, and the targets at time t-1 and time t+1 are compared. If there are targets with the same static target instance ID, the process proceeds to step S4. If there are no targets with the same static target instance ID, the instantaneous inertial measurement unit data is used as the motion data of the current image frame to proceed to step S6. Step S4: Based on the target with the matched static target instance ID, group and map them, and finally calculate the average rotation matrix; Step S5: Fuse the average rotation matrix with the data from the n inertial measurement units before and after the frame to obtain the final state matrix, where n is a natural number greater than or equal to 2; Step S6: Save the state matrix and draw the global route.

2. The path planning method based on multimodal joint integration according to claim 1, characterized in that, Step S1 further includes: Step S11: Preprocess the captured image, including statistical analysis of the frequency distribution of image pixel values, and improve contrast and visibility by equalization processing to make the pixel distribution more uniform. Step S12: Use Shi-Tomasi to detect feature points in the image; Step S13: Match the feature points of the image at time t-1 with the image at time t, and then estimate the movement of the image feature points in consecutive frames by analyzing the movement of pixels in the image sequence, thereby tracking the key points and finally obtaining the dynamic area heat map relative to the car.

3. The path planning method based on multimodal joint integration according to claim 1, characterized in that, Step S2 further includes: Step S21: Combine the sensing information from the inertial measurement unit to perform loosely coupled online calibration and align information including velocity and scale; Step S22: Initialize the motion state matrix of the object.

4. The path planning method based on multimodal joint integration according to claim 1, characterized in that, Step S3 further includes: Step S31: Input the image to be segmented; Step S32: Use an image encoder to extract high-dimensional feature representations of the input image and output an image feature map; Step S33: Encode the user-provided prompts into feature representations using a prompt encoder, and output the prompt features, wherein the user-provided prompts include points, boxes, and text descriptions; Step S34: Feature fusion, which involves fusing image features and cue features together through feature concatenation or attention mechanisms, and outputting fused features to prepare for subsequent mask decoding; Step S35: Using a mask decoder based on the Transformer architecture, a segmentation mask is generated and output based on the fusion features. In the segmentation mask, the value of each pixel represents the probability that the pixel belongs to the target object. Step S36: Perform optimization operations on the segmentation mask to generate the final refined segmentation result, i.e., the refined segmentation mask. The optimization operations include thresholding and morphological operations. Step S37: Combine the segmentation mask with the original image to extract the image region of the target object; Step S38: Input the image region of the target object into an image-text model to generate the image's feature vector; Step S39: Match the feature vector of the image with the feature vector of the predefined text label, use a similarity metric to find the most similar text label and output it.

5. The path planning method based on multimodal joint integration according to claim 4, characterized in that, In step S32, the image encoder uses a pre-trained visual model to extract features. The visual model is a multi-scale visual model based on CrossVIT, which extracts and fuses information through image patches of different sizes to improve the detection capability of targets at different scales.

6. The path planning method based on multimodal joint integration according to claim 5, characterized in that, Step S32 further includes: Step S321: Obtain a one-dimensional vector representation by encoding with a block encoder and output it; Step S322: Obtain category vectors through a category encoder to integrate image information for classification; Step S323: Obtain a position vector through position encoding to express the position information of the image patch in the original image; Step S324: Extract features and aggregate information from the input sequence using a Transformer encoder.

7. The path planning method based on multimodal joint integration according to claim 4, characterized in that, Step S34 further includes: Step S341: Project the image and text features into the same embedding space through a linear projection layer for feature alignment; Step S342: By contrastive learning loss function, matching text image features are brought closer in the embedding space, while non-matching features are moved further away.

8. The path planning method based on multimodal joint integration according to claim 1, characterized in that, Step S4 further includes: Step S41: Calculate the covariance matrix according to the matching target; Step S42: Obtain the rotation matrix using singular value decomposition; Step S43: Calculate the mean of the rotation matrix for all targets.

9. The path planning method based on multimodal joint integration according to claim 8, characterized in that, Step S41 further includes: Step S411: Obtain the centroid through the acquired matching feature points and decentroid them; Step S412: Calculate the centroid-decentered covariance matrix. The covariance matrix represents the changing trend of matching point pairs between consecutive frames and is used to infer the rotation relationship between consecutive frames.

10. The path planning method based on multimodal joint integral according to claim 8, characterized in that, Step S43 further includes: Step S431: For each target, convert the rotation matrix into a quaternion; Step S432: Calculate a weighted average of all quaternions to obtain an average quaternion; Step S433: Convert the quaternion into a rotation matrix form to obtain the average rotation matrix.

11. The path planning method based on multimodal joint integral according to claim 1, characterized in that, In step S5, the average rotation matrix obtained in step S4 is combined with the integral of at least two inertial measurement unit data before and after the frame to calculate the observed motion matrix. After one weighting, the motion state matrix of the object is updated by the weighted motion state value to obtain the predicted motion matrix. The predicted value at time t+1 and the actual value at time t+1 in the predicted motion state matrix are weighted again to obtain the output value.

12. A path planning system based on multimodal joint integration, characterized in that, The system includes: The dynamic region preliminary screening module is configured for image preprocessing and dynamic region preliminary screening, including image preprocessing, extracting image feature points, and performing a match between the image at time t-1 and the image at time t to initially screen out dynamic regions; The motion state matrix initialization module is configured for loosely coupled online calibration and performs motion state matrix initialization. The region fine segmentation and instance recognition module is configured such that after the motion state matrix is ​​initialized, subsequent images from the vehicle-mounted camera are fed into the SAM network as input to obtain the region fine segmentation output, which is then input into the CLIP network to obtain the instance ID corresponding to the refined region. The output result is saved, and the target at time t-1 and time t+1 is compared. If there is a target with the same static target instance ID, it enters the average rotation matrix calculation module. If there is no target with the same static target instance ID, the instantaneous inertial measurement unit data is used as the motion data of the current image frame to enter the global route drawing module. The average rotation matrix calculation module is configured to group targets based on the matched static target instance IDs, perform mapping, and finally calculate the average rotation matrix. The state matrix acquisition module is configured to fuse the average rotation matrix with the data from n inertial measurement units before and after the frame to obtain the final state matrix, where n is a natural number greater than or equal to 2. The global route drawing module is configured to save the state matrix and draw the global route.

13. A computer system for path planning, characterized in that, It includes a memory, a processor, and program instructions stored in the memory that are executable by the processor, wherein the processor executes the program instructions to implement the steps of the path planning method based on multimodal joint integration as described in any one of claims 1 to 11.

14. A computer-readable storage medium for path planning, characterized in that, It stores program instructions executable by a processor to implement the steps of the path planning method based on multimodal joint integration as described in any one of claims 1 to 11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the path planning method based on multimodal joint integral as described in any one of claims 1 to 11.

Images