Path planning method and system based on multi-modal joint integration

Abstract

The invention discloses a path planning method and system based on multi-modal joint integration, and the method employs an image processing and deep learning technology to automatically recognize and eliminate the influence of a dynamic object, and guarantees that SLAM and VIO systems are only matched with a stable static background object. According to the technical scheme, firstly, feature points in an input image are preliminarily screened and matched through the image feature point matching technology; and then introducing a deep learning network to carry out refined segmentation and instance identification. The deep learning network can accurately distinguish a static object from a dynamic object, and accurately identify the existence of the dynamic object in a complex scene. After the dynamic objects are identified, the dynamic objects are eliminated from the feature matching process by adopting an intelligent elimination strategy, so that the SLAM and VIO systems are ensured to perform feature matching and positioning calculation only based on stable static background objects, the negative influence of the dynamic objects on the positioning precision is avoided, and the stability and applicability of the system in a complex dynamic environment are improved.

Description

Technical Field

[0001] The present invention relates to a path planning method, and in particular to a path planning method and system based on multi-modal joint integration. Background Art

[0002] With the rapid development of robotics and autonomous navigation, Simultaneous Localization and Mapping (SLAM) technology has become one of the core technologies in these fields. In the past decade, SLAM technology has made significant progress and has been widely used in many fields such as autonomous driving, drone navigation, industrial robots, and virtual / augmented reality. SLAM technology can continuously update position information and build environmental maps through sensor input (such as cameras, lidar, etc.) without relying on external positioning systems (such as GPS), thereby achieving autonomous navigation and positioning.

[0003] However, with the increasing complexity of actual application scenarios and the increasing demand for high-precision positioning, the challenges faced by traditional SLAM technology are also increasing. In order to meet these challenges, researchers began to combine visual-inertial odometer (VIO) with SLAM technology and proposed visual-inertial simultaneous localization and mapping (VI-SLAM) technology. VI-SLAM technology is an enhanced algorithm that combines the inertial data of visual sensors and inertial measurement units (IMUs) based on traditional SLAM technology. IMU sensors can provide high-frequency acceleration and angular velocity information, so that when visual information is missing, blocked or lighting conditions are poor, short-term position and attitude estimation can be performed based on inertial data. This high-frequency data complements the low-frequency characteristics of visual information and effectively reduces the accumulation of errors. At the same time, visual data can also provide long-term global correction for IMUs, avoid errors caused by inertial drift, and enable the system to maintain high positioning accuracy over a long period of time.

[0004] Although SLAM and VIO technologies can provide high-precision positioning and map construction in static object environments, the existence of dynamic objects in the real world is an issue that cannot be ignored. Dynamic objects, such as pedestrians and vehicles, will interfere with the matching of image feature points, causing the SLAM system to mistakenly treat dynamic objects as static backgrounds, which in turn causes position estimation errors and map update errors. Especially in complex urban environments, the interference of dynamic objects is particularly serious, greatly reducing the robustness and accuracy of the system.

[0005] Therefore, how to solve the problem of dynamic object interference in order to improve the applicability and positioning accuracy of SLAM and VI-SLAM systems in complex dynamic environments is a problem that needs to be solved urgently in the industry. Summary of the invention

[0006] A brief summary of one or more aspects is given below to provide a basic understanding of these aspects. This summary is not an exhaustive overview of all conceived aspects, and is neither intended to identify the key or critical elements of all aspects nor to define the scope of any or all aspects. Its only purpose is to give some concepts of one or more aspects in a simplified form as a prelude to a more detailed description that will be given later.

[0007] The purpose of the present invention is to solve the above problems and provide a path planning method and system based on multimodal joint integration, which uses image processing and deep learning technology to automatically identify and eliminate the influence of dynamic objects, thereby ensuring that SLAM and VIO systems only match stable static background objects to improve the overall performance and positioning accuracy of the system.

[0008] The technical solution of the present invention is: the present invention discloses a path planning method based on multi-modal joint integration, the method comprising:

[0009] Step S1: image preprocessing and preliminary screening of dynamic areas, including preprocessing the image, extracting image feature points, matching the image at time t-1 with the image at time t, and preliminary screening of dynamic areas;

[0010] Step S2: based on loosely coupled online calibration, initialization of motion state matrix is ​​performed;

[0011] Step S3: fine-grained regional segmentation and instance recognition, including after the motion state matrix is ​​initialized, the subsequent on-board camera image is input into the SAM network as input to obtain the fine-grained regional segmentation output, which is then input into the CLIP network to obtain the instance ID corresponding to the refined region, save the output result, compare the targets at time t-1 and time t+1, if there is a target with the same static target instance ID, proceed to step S4, if there is no target with the same static target instance ID, use the instantaneous inertial measurement unit data as the motion data of the frame to proceed to step S6;

[0012] Step S4: Mapping is performed in groups according to the targets with the matched static target instance IDs, and finally the average rotation matrix is ​​calculated;

[0013] Step S5: fusing the average rotation matrix with the n inertial measurement unit data before and after the frame to obtain the final state matrix, where n is a natural number greater than or equal to 2;

[0014] Step S6: Save the state matrix and draw the global route.

[0015] According to an embodiment of the path planning method based on multi-modal joint integration of the present invention, step S1 further comprises:

[0016] Step S11: pre-processing the captured image, including statistically analyzing the frequency distribution of image pixel values, and making the pixel distribution uniform through equalization to improve contrast and visibility;

[0017] Step S12: Use Shi-Tomasi to detect image feature points;

[0018] Step S13: Match the feature points of the image at time t-1 with the image at time t, and then estimate the movement of the image feature points in consecutive frames by analyzing the movement of pixels in the image sequence, so as to track the key points and finally obtain a dynamic area heat map relative to the car.

[0019] According to an embodiment of the path planning method based on multi-modal joint integration of the present invention, step S2 further comprises:

[0020] Step S21: combining the sensor information of the inertial measurement unit, performing online calibration based on loose coupling, and aligning information including speed and scale;

[0021] Step S22: Initialize the motion state matrix of the object.

[0022] According to an embodiment of the path planning method based on multi-modal joint integration of the present invention, step S3 further comprises:

[0023] Step S31: input the image to be segmented;

[0024] Step S32: using an image encoder to extract a high-dimensional feature representation of the input image and output an image feature map;

[0025] Step S33: encoding the prompt provided by the user into a feature representation through a prompt encoder, and outputting a prompt feature, wherein the prompt provided by the user includes a point, a box, and a text description;

[0026] Step S34: feature fusion, that is, fusing the image features and the prompt features together through feature splicing or attention mechanism, and outputting the fused features to prepare for subsequent mask decoding;

[0027] Step S35: Generate and output a segmentation mask based on the fusion features through a mask decoder based on the Transformer architecture. In the segmentation mask, the value of each pixel represents the probability that the pixel belongs to the target object.

[0028] Step S36: Optimizing the segmentation mask to generate a final refined segmentation result, that is, a refined segmentation mask, wherein the optimization operation includes thresholding and morphological operations;

[0029] Step S37: combining the segmentation mask with the original image to extract the image area of ​​the target object;

[0030] Step S38: inputting the image region of the target object into an image-text model to generate a feature vector of the image;

[0031] Step S39: Match the feature vector of the image with the feature vector of the predefined text tag, use a similarity metric to find the most similar text tag and output it.

[0032] According to an embodiment of the path planning method based on multimodal joint integration of the present invention, in step S32, the image encoder uses a pre-trained visual model to realize feature extraction, wherein the visual model is a multi-scale visual model based on CrossVIT, which extracts and fuses information through image patches of different sizes to improve the detection capability of targets of different scales.

[0033] According to an embodiment of the path planning method based on multi-modal joint integration of the present invention, step S32 further includes:

[0034] Step S321: Obtain a one-dimensional vector representation by encoding through a block encoder and output;

[0035] Step S322: obtaining a category vector through a category encoder so as to integrate image information for classification;

[0036] Step S323: obtaining a position vector by position coding to express the position information of the image block in the original image;

[0037] Step S324: Perform feature extraction and information aggregation on the input sequence through the Transformer encoder.

[0038] According to an embodiment of the path planning method based on multi-modal joint integration of the present invention, step S34 further includes:

[0039] Step S341: projecting the image text features into the same embedding space through a linear projection layer for feature alignment;

[0040] Step S342: By contrasting the learning loss function, matching text image features are brought closer in the embedding space, while mismatched ones are moved further away.

[0041] According to an embodiment of the path planning method based on multi-modal joint integration of the present invention, step S4 further comprises:

[0042] Step S41: Obtaining covariance matrices according to the matching targets respectively;

[0043] Step S42: Obtain a rotation matrix using singular value decomposition;

[0044] Step S43: Calculate the mean of the rotation matrices of all targets.

[0045] According to an embodiment of the path planning method based on multi-modal joint integration of the present invention, step S41 further includes:

[0046] Step S411: obtaining the centroid through the obtained matching feature points, and removing the centroid;

[0047] Step S412: Calculate the de-centroided covariance matrix, where the covariance matrix represents the change trend of the matching point pairs of the previous and next frames, and is used to infer the rotation relationship between the previous and next frames;

[0048] According to an embodiment of the path planning method based on multi-modal joint integration of the present invention, step S43 further includes:

[0049] Step S431: for each target, convert the rotation matrix into a quaternion;

[0050] Step S432: performing weighted averaging on all quaternions to obtain an average quaternion;

[0051] Step S433: Convert the quaternion into a rotation matrix to obtain an average rotation matrix.

[0052] According to an embodiment of the path planning method based on multimodal joint integration of the present invention, in step S5, the average rotation matrix obtained in step S4 and the integral of at least two inertial measurement unit data before and after the frame are used to jointly calculate the observation motion matrix. After weighting once, the motion state matrix of the object is updated by the weighted motion state value to obtain the predicted motion matrix, and the predicted value at time t+1 and the true value at time t+1 in the predicted motion state matrix are weighted again as the output value.

[0053] The present invention also discloses a path planning system based on multi-modal joint integration, the system comprising:

[0054] The dynamic area preliminary screening module is configured as image preprocessing and dynamic area preliminary screening, including image preprocessing, image feature point extraction, and preliminary screening of dynamic areas by matching the image at time t-1 with the image at time t.

[0055] A motion state matrix initialization module is configured as an online calibration based on loose coupling and performs motion state matrix initialization;

[0056] The region refinement segmentation and instance recognition module is configured to, after the motion state matrix is ​​initialized, send the subsequent on-board camera image as input to the SAM network to obtain the region refinement segmentation output, and then input it to the CLIP network to obtain the instance ID corresponding to the refined region, save the output result, compare the targets at time t-1 and time t+1, if there is a target with the same static target instance ID, enter the average rotation matrix calculation module, if there is no target with the same static target instance ID, use the instantaneous inertial measurement unit data as the motion data of the frame to enter the global route drawing module;

[0057] The average rotation matrix calculation module is configured to group and map the targets according to the matched static target instance IDs, and finally calculate and find the average rotation matrix;

[0058] A state matrix acquisition module is configured to fuse the average rotation matrix with n inertial measurement unit data before and after the frame to obtain a final state matrix, where n is a natural number greater than or equal to 2;

[0059] A global route drawing module is configured to save the state matrix and draw the global route.

[0060] The present invention also discloses a computer system for path planning, comprising a memory, a processor, and program instructions stored in the memory for execution by the processor, wherein the processor executes the program instructions to implement the steps of the path planning method based on multi-modal joint integration as described above.

[0061] The present invention also discloses a computer-readable storage medium for path planning, characterized in that it stores program instructions executable by a processor to implement the steps of the path planning method based on multi-modal joint integration as described above.

[0062] The present invention also discloses a computer program product, including a computer program, which implements the steps of the path planning method based on multi-modal joint integration as described above when executed by a processor.

[0063] Compared with the prior art, the present invention has the following beneficial effects: The present invention aims to solve the interference problem caused by dynamic objects to SLAM (Simultaneous Localization and Mapping) and VIO (Visual Inertial Odometry) systems, thereby improving the positioning accuracy and system applicability in complex dynamic environments. To this end, the present invention proposes an innovative method that cleverly combines image processing and deep learning technology to achieve automatic recognition and exclusion of dynamic objects.

[0064] Specifically, the present invention first uses image feature point matching technology to perform preliminary screening and matching of feature points in the input image. Subsequently, a deep learning network is introduced to perform refined segmentation and instance recognition. After extensive training, the deep learning network can accurately distinguish between static objects and dynamic objects, thereby accurately identifying the presence of dynamic objects in complex scenes.

[0065] After identifying dynamic objects, the present invention adopts an intelligent exclusion strategy. The system will exclude these dynamic objects from the feature matching process, ensuring that the SLAM and VIO systems only perform feature matching and positioning calculations based on stable static background objects. This strategy effectively avoids the negative impact of dynamic objects on positioning accuracy, while improving the stability and applicability of the system in complex dynamic environments.

[0066] Through the above technical scheme, the present invention not only significantly improves the positioning accuracy of SLAM and VIO systems, but also expands their application scope in complex dynamic environments, providing more reliable and accurate positioning and navigation support for robots, autonomous driving, drone navigation and other fields.

[0067] In detail, the main advantages of the present invention compared to the conventional technology are:

[0068] 1. Multimodal data fusion and IMU data correction:

[0069] Compared with the traditional SLAM (Simultaneous Localization and Mapping) technology, this invention introduces the visual fusion part to achieve the deep integration of visual sensor and IMU (Inertial Measurement Unit) data. Under the joint action of multimodal data, the data offset of IMU is effectively corrected, thereby significantly improving the positioning accuracy and stability of the system. This fusion strategy not only enhances the robustness of the system, but also enables the system to perceive and respond more accurately in complex environments.

[0070] 2. Application of deep learning algorithms and dynamic object recognition:

[0071] Compared with the traditional V-SLAM (Visual Simultaneous Localization and Mapping) technology, the present invention innovatively adopts a deep learning algorithm to perform more precise segmentation and identification of dynamic and static objects in the image. This improvement enables the system to more accurately calculate the motion state of the vehicle and avoid the interference of dynamic objects on the positioning results. Through the introduction of deep learning technology, the system of the present invention shows higher positioning accuracy and applicability in complex dynamic environments.

[0072] 3. Unsupervised target area segmentation and splitting capabilities:

[0073] Compared with the traditional supervised target detection algorithm, the algorithm structure proposed in the present invention can still segment and split the target area under unsupervised conditions. This feature greatly reduces the algorithm's dependence on labeled data and improves the algorithm's flexibility and generalization ability. Under the framework of unsupervised learning, the present invention can adaptively handle various complex scenarios, further improving the practicality and reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The above features and advantages of the present invention can be better understood after reading the detailed description of the embodiments of the present disclosure in conjunction with the following drawings. In the drawings, the components are not necessarily drawn to scale, and components with similar related properties or features may have the same or similar reference numerals.

[0075] Figure 1 A flow chart of an embodiment of a path planning method based on multi-modal joint integration of the present invention is shown.

[0076] Figure 2 Shows Figure 1 A schematic diagram of the model architecture of a multi-scale visual model in the method embodiment is shown.

[0077] Figure 3 A schematic diagram of an embodiment of a path planning system based on multi-modal joint integration of the present invention is shown. DETAILED DESCRIPTION

[0078] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. Note that the aspects described below in conjunction with the accompanying drawings and specific embodiments are only exemplary and should not be construed as limiting the scope of protection of the present invention in any way.

[0079] Figure 1 The flow chart of an embodiment of the path planning method based on multi-modal joint integration of the present invention is shown. Figure 1 , the implementation steps of the method of this embodiment are described in detail as follows.

[0080] Step S1: Image preprocessing and preliminary screening of dynamic areas.

[0081] The image is preprocessed to extract feature points. The image at time t-1 is matched with the image at time t. The vehicle itself is used as a reference system to preliminarily screen out the dynamic area.

[0082] Step S1 further includes the following processing steps:

[0083] Step S11: pre-process the image captured by the camera, including statistically analyzing the frequency distribution of image pixel values, and making the pixel distribution uniform through equalization to improve contrast and visibility, so as to facilitate subsequent feature extraction and reduce brightness changes between consecutive frames.

[0084] Step S12: extracting image feature points in the image based on the Shi-Tomasi corner detection algorithm.

[0085] Step S12 further includes the following processing steps:

[0086] Use Shi-Tomasi to detect image feature points. For the input image, first calculate the gradient of each pixel in the x and y directions to obtain the derivatives in the two directions, which represent the rate of change in the x and y directions. For each pixel, an autocorrelation matrix is ​​constructed by weighted summing the gradients on x and y, and then the two eigenvalues ​​of the autocorrelation matrix are calculated, and the smaller eigenvalue is used as the response value R of the corner point. Finally, all R values ​​are counted, and the maximum R value and k value are calculated as the threshold, where the k value is the set response value, which is the threshold scale factor when selecting feature points, and is used to determine the scale factor of the selection threshold of the response value, which affects the number and quality of detected feature points. If the response value R is greater than the threshold, it is regarded as an image feature point.

[0087] Step S13: matching the feature points of the image at time t-1 with the image at time t means calculating the Euclidean distance through the image feature point sets in the image at time t-1 and the image at time t, respectively. If there are matching points within the set distance, the match is considered successful. Then, by analyzing the movement of pixels in the image sequence, the movement of the image feature points in continuous frames is estimated, thereby tracking the key points, and finally obtaining a dynamic area heat map relative to the vehicle.

[0088] Specifically, one match is to match the image feature points obtained in step S12 with the image feature points obtained by feature detection in the previous frame. For each image feature point in the previous frame, each image feature point in the current frame is traversed one by one, and the Euclidean distance between them is calculated. The image feature point with the smallest distance is found as the matching point. If the distance of the matching point is less than the set threshold, the matching point is a valid matching point, otherwise the matching point is eliminated. The matching formula is: Among them, x1 and x2 are the x coordinates of the previous and next frames, that is, the horizontal coordinates on the image; y1 and y2 are the y coordinates of the previous and next frames, that is, the vertical coordinates on the image. After the match is successful, the feature point of the image is added to the tracking sequence.

[0089] According to the historical motion information of the image feature points in the tracking sequence, the feature points in the image move less in the local area in the continuous frames, and the motion vector are calculated by combining the temporal gradient and the spatial gradient, so as to obtain the estimated predicted position of the image feature points in the next frame, so that the next frame can be matched near the predicted position. If there are valid matching points, continue to track and update the latest motion information, if there are no valid matching points, remove them.

[0090] The key points tracked are image feature points, which are local areas in the image that can be stably and uniquely detected and described. Image feature points are generally in the salient areas of the image (such as corners, edges, spots, etc.), which remain relatively unchanged when the image is scaled, rotated or the illumination changes.

[0091] The generation of dynamic area heat map is to calculate a motion vector for each image feature point to represent the displacement of the previous and next frames. If the motion vector of the image feature point is less than the defined threshold, it is a static point, otherwise it is a dynamic point, indicating that it belongs to a dynamic object. These dynamic points are selected as the dynamic area in the image.

[0092] Step S2: Loosely coupled online calibration and motion state matrix initialization.

[0093] Step S2 further includes the following processing:

[0094] Step S21: Combine the sensor information of IMU (Inertial Measurement Unit) and perform loosely coupled online calibration to align speed, scale and other information. Loose coupling means that the image data and IMU data are processed independently and then fused. Online calibration means adjusting parameters in real time during system operation to ensure data accuracy.

[0095] Step S22: Initialize the motion state matrix of the object. The motion state matrix has a form of n*m, where the n axis corresponds to the moment, and m corresponds to the state information at different moments, including but not limited to the motion speed, motion direction, and other motion information.

[0096] Step S3: Region refinement segmentation and instance recognition.

[0097] After initializing the motion state matrix, the subsequent image from the on-board camera is sent as input to the SAM (SegmentAnything Model) network to obtain the refined regional segmentation output (corresponding to the subsequent steps S31-S37), and then input into the CLIP (Contrastive Language-Image Pre-training) network (corresponding to the subsequent steps S38-S39) to obtain the instance ID corresponding to the refined area, save the output result, compare the targets at time t-1 and time t+1, if there is a target with the same static target instance ID, go to step S4, if there is no target with the same instance ID, use the instantaneous IMU data as the motion data of the frame and go to step S6.

[0098] Step S3 further includes the following processing steps.

[0099] Step S31: input the image to be segmented.

[0100] Step S32: Use an image encoder to extract a high-dimensional feature representation of the input image and output an image feature map.

[0101] Image encoders usually use pre-trained vision models such as Vision Transformer (ViT) or high-performance convolutional neural networks (CNN) for feature extraction.

[0102] refer to Figure 2 As shown, in this embodiment, the visual model is a multi-scale visual model based on CrossVIT, which extracts and fuses information through image patches of different sizes to improve the detection capability of targets of different scales.

[0103] The training of the multi-scale vision model includes the following process.

[0104] Step 1: For the input image, first pass it through the DS2N (Deep Separable Convolutional Bottleneck unit with 2 times) module, then through the DSP3 module, and then through the DS2N module once more to extract features to obtain the feature map.

[0105] in:

[0106] LeakyReLU (leaky linear rectifier function), as an activation function, performs nonlinear transformation on the input data to enhance the expressiveness of the model. Compared with traditional ReLu, LeakyReLU allows negative input values ​​to pass at a smaller ratio, thereby alleviating the problem of neuron death;

[0107] DSCBS (Depthwise Separable Convolutional Block with BatchNormalization and SILU) is a combination module of depthwise separable convolution (DSC), batchnorm (BN, Batch Normalization), and SILU (Sigmoid-weighted Linear Unit);

[0108] Bottleunit is a bottleneck unit using 2 DSCBS modules;

[0109] DSP3 (Depthwise Separable Convolutional Pyramid with 3levels) is a module that extracts features from the original feature map at three different depths and then splices them together;

[0110] DS2N is a module that extracts features from the original feature map using N (at least 2) Bottleunits and one DSCBS, and then concatenates them.

[0111] The Add operation adds the output of the main branch and the output of the jump link to form a residual map. Using this mechanism can alleviate the gradient vanishing problem and promote feature fusion to improve the model's feature extraction capability.

[0112] Step 2: After extracting features in the first step, the output feature map is cut into windows of different sizes, for example, a feature map is divided into 8*8 blocks, or a feature map is divided into 16*16 blocks. That is, small patches and large patches are used to more accurately extract large and small objects.

[0113] Step 3: Input ViT (Vision Transformer) network (see steps S321-S324)

[0114] Step 4: Finally, cat (Concatenate) outputs the final result.

[0115] Step S32 is further refined into the following processing process.

[0116] Step S321: Obtain a one-dimensional vector representation by encoding through a block encoder and output it. Specifically, the picture is first divided into P*P (such as 16*16) blocks, and then each block is flattened into a one-dimensional vector.

[0117] Step S322: Obtain a category vector through a category encoder to integrate image information for classification

[0118] Specifically: a learnable image patch embedding vector is used as a classification label and added to the front of the image patch sequence.

[0119] Step S323: obtaining a position vector through position coding to express the position information of the image block in the original image.

[0120] Specifically, a learnable position embedding vector is assigned to each image block to ensure that the position information of each block is preserved. The position embedding vector is added to the image block embedding vector element to complete the encoding.

[0121] Step S324: Perform feature extraction and information aggregation on the input sequence (i.e., the encoding outputted by step S323) through the Transformer encoder.

[0122] Specifically, we first perform layer normalization to reduce the fluctuation of the value range compared to batch normalization and perform normalization within the sample. We then extract features through a multi-head attention mechanism. Finally, we use a residual structure for the final output. We repeat the above steps multiple times to extract features. We then use a fully connected layer and a nonlinear activation function for the final output.

[0123] Step S33: Encode the prompt (such as point, box, text description) provided by the user into a feature representation through a prompt encoder, and output the prompt feature.

[0124] The specific implementation includes:

[0125] Point hint: Use the geometric encoding method to encode the position of the point as a feature. Here, the input is the center point of each dynamic area in step S1.

[0126] Textual Hints: Encode textual descriptions as features using a pre-trained text encoder.

[0127] Step S34: feature fusion, that is, fusing the image features and the prompt features together through feature splicing or attention mechanism, and outputting the fused features to prepare for subsequent mask decoding.

[0128] Step S34 further includes the following processing procedures.

[0129] Step S341: Through the linear projection layer, the image text features are projected into the same embedding space for feature alignment.

[0130] Step S342: Use the contrastive learning loss function (InfoNCE) to make the matching text image features close in the embedding space and the mismatched ones far away.

[0131] In step S342, based on RINCE and ReLICv2, RCILoss (Robust Causal Invariant Loss) is innovated. RINCE emphasizes the robustness of negative sample selection, and ReLIC2 improves the robustness and generalization of representation.

[0132] The advantages of RCILoss are:

[0133] Improved robustness: The negative sample selection strategy of RINCE is combined to enhance the robustness of the model to data imbalance and noise.

[0134] Local structure preservation: Drawing on the ideas of ReLICv2, the model is ensured to capture local structures and is suitable for processing scenarios with complex local information:

[0135]

[0136] RCILoss=αL1+βL2

[0137] The sum of a and b is 1, and the default values ​​in the early stage of training are 0.8 and 0.2, respectively. The values ​​of a and b are adjusted dynamically according to the training effect. , q represents the control parameter, which adjusts the behavior of the loss function;

[0138] S represents the similarity score of the feature vector,

[0139] δ+ represents the similarity score of the positive sample,

[0140] δi- represents the similarity score of the i-th negative sample, that is, the similarity between the target feature and the negative sample;

[0141] K represents the number of negative samples;

[0142] x i is the input feature of the current sample,

[0143] X i + is a positive sample of the current sample,

[0144] P(x i ) is the positive sample set,

[0145] α is a weight hyperparameter, which is used to balance the impact of the positive sample joint distribution loss on the total loss;

[0146] P(x i ;x i+ ) is the joint probability distribution used to quantify x i and x i + relationship;

[0147] β weight hyperparameter, used to control the contribution of KL divergence to the total loss;

[0148] P(x i ) Sample xi distribution,

[0149] P(x + ) positive sample distribution,

[0150] DKL is the KL divergence, which is used to measure p(x i ) and p(x + ) between the two.

[0151] Step S35: Through the mask decoder, based on the Transformer architecture, a segmentation mask is generated and output according to the fusion features. In the segmentation mask, the value of each pixel represents the probability that the pixel belongs to the target object.

[0152] Step S36: performing optimization operations (such as thresholding, morphological operations, etc.) on the segmentation mask to generate a final refined segmentation result, that is, a refined segmentation mask.

[0153] Step S37: combining the segmentation mask with the original image to extract the image region of the target object, including: using the segmentation mask to extract the image region of the target object on the original image, for example, accurately extracting the target region by binarization and morphological operations of the mask.

[0154] Step S38: Input the image area of ​​the target object into an image-text model to generate a feature vector of the image.

[0155] Step S39: Match the feature vector of the image with the feature vector of the predefined text tag, use a similarity metric (such as cosine similarity) to find the most similar text tag and output it.

[0156] Through the processing of step S3, static and dynamic objects can be segmented in an unsupervised manner.

[0157] Step S4: According to the targets of the matched instance IDs, group them for mapping, and finally calculate the average rotation matrix, that is, the motion state at this time.

[0158] Step S4 further includes the following processing steps.

[0159] Step S41: Obtain the covariance matrix according to the matching targets respectively.

[0160] Step S41 further includes:

[0161] Step S411: obtaining the centroid through the obtained matching feature points, and removing the centroid;

[0162] Step S412: Calculate the de-centroided covariance matrix, which represents the change trend of the matching point pairs of the previous and next frames, and can be used to infer the rotation relationship between the previous and next frames.

[0163] Step S42: Decompose the rotation matrix using SVD (Singular Value Decomposition).

[0164] Step S42 further includes:

[0165] Step S421: Use the covariance matrix obtained in step S41 to perform singular value decomposition. The formula is as follows: U and V are orthogonal matrices, ε is a diagonal matrix containing singular values ​​of the covariance matrix, and H represents the covariance matrix.

[0166] H=UεV T

[0167] Step S422: After SVD decomposition, the rotation matrix R is calculated. The formula is as follows to represent the rotation matrix of the previous and next frames: R = VU T .

[0168] Step S43: Calculate the mean of the rotation matrices of all targets.

[0169] Step S43 further includes:

[0170] Step S431: For each target, convert the rotation matrix into a quaternion for better calculation;

[0171] Step S432: performing weighted averaging on all quaternions to obtain an average quaternion;

[0172] Step S433: Convert the quaternion into a rotation matrix to obtain an average rotation matrix.

[0173] Step S5: Fusing the average rotation matrix with n IMU data before and after the frame to obtain the final state matrix, where the value of n is greater than or equal to 2.

[0174] The average rotation matrix obtained in step S4 is used together with the integral of at least 2 imu data before and after the frame to calculate the observed motion matrix. After weighting once, the motion state matrix of the object is updated by the weighted motion state value to obtain the predicted motion matrix. The predicted value at time t+1 in the predicted motion state matrix and the true value at time t+1 are weighted again as the output value. The formula is as follows:

[0175] Status ti =α·imu ti +β·img ti +γ·predict ti

[0176] Among them, α, β, and γ are weight hyperparameters, and the sum of the three is 1. imu represents the imu calculation value at time ti, img represents the image calculation value at time ti, predict represents the predicted value of the Kalman filter at time ti, and Status ti Represents the final motion state matrix.

[0177] Step S6: Save the state matrix and draw the global route.

[0178] That is, save the current motion state and draw the global motion state in combination with the historical motion trajectory.

[0179] Figure 3 The principle of an embodiment of the path planning system based on multi-modal joint integration of the present invention is shown. Figure 3 ,The system of this embodiment includes the following modules: a dynamic region preliminary screening module, a motion state matrix initialization module, a region refinement segmentation and instance recognition module, an average rotation matrix calculation module, a state matrix acquisition module, and a global route drawing module.

[0180] The dynamic area preliminary screening module is configured for image preprocessing and dynamic area preliminary screening, including image preprocessing, image feature point extraction, and preliminary screening of dynamic areas by matching the image at time t-1 with the image at time t. Figure 1 The step S1 shown is the same and will not be repeated here.

[0181] The motion state matrix initialization module is configured as loosely coupled online calibration and performs motion state matrix initialization. Figure 1 The step S2 shown is the same and will not be repeated here.

[0182] The regional refined segmentation and instance recognition module is configured to send the subsequent vehicle camera images as input to the SAM network after the motion state matrix is ​​initialized, obtain the regional refined segmentation output, and then input it to the CLIP network to obtain the instance ID corresponding to the refined region, save the output result, compare the targets at time t-1 and time t+1, if there are targets with the same static target instance ID, enter the average rotation matrix calculation module, if there are no targets with the same static target instance ID, use the instantaneous inertial measurement unit data as the motion data of the frame to enter the global route drawing module. The specific processing of this module is similar to Figure 1 The step S3 shown is the same and will not be repeated here.

[0183] The average rotation matrix calculation module is configured to map the targets according to the matched static target instance IDs, and finally calculate the average rotation matrix. Figure 1 The step S4 shown is the same and will not be repeated here.

[0184] The state matrix acquisition module is configured to fuse the average rotation matrix with the n inertial measurement unit data before and after the frame to obtain the final state matrix, where n is a natural number greater than or equal to 2. The specific processing of this module is similar to Figure 1 The step S5 shown is the same and will not be repeated here.

[0185] The global route drawing module is configured to save the state matrix and draw the global route. Figure 1 The step S6 shown is the same and will not be described again.

[0186] In addition, the present invention also discloses a computer system for path planning, the system includes a memory, a processor and program instructions stored in the memory for the processor to execute, wherein the processor executes the program instructions to implement the following Figure 1 The steps of an embodiment of a path planning method based on multi-modal joint integration are shown.

[0187] In addition, the present invention also discloses a computer-readable storage medium for path planning, which stores program instructions executable by a processor to execute the following steps: Figure 1 The steps of an embodiment of a path planning method based on multi-modal joint integration are shown.

[0188] In addition, the present invention also discloses a computer program product, including a computer program, which, when executed by a processor, implements the following Figure 1 The steps of an embodiment of a path planning method based on multi-modal joint integration are shown.

[0189] Although the above methods are illustrated and described as a series of actions for simplicity of explanation, it should be understood and appreciated that these methods are not limited by the order of the actions, because according to one or more embodiments, some actions may occur in a different order and / or concurrently with other actions from those illustrated and described herein or not illustrated and described herein but understandable to those skilled in the art.

[0190] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The technician may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present invention.

[0191] The various illustrative logic blocks, modules, and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in cooperation with a DSP core, or any other such configuration.

[0192] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read and write information from / to the storage medium. In an alternative, a storage medium may be integrated into a processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and the storage medium may reside in a user terminal as discrete components.

[0193] In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented as a computer program product in software, each function may be stored on or transmitted by a computer-readable medium as one or more instructions or codes. Computer-readable media include both computer storage media and communication media, including any medium that facilitates the transfer of a computer program from one place to another. Storage media may be any available medium that can be accessed by a computer. As an example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer. Any connection is also properly referred to as a computer-readable medium. For example, if the software is transmitted from a website, a server, or other remote source using a coaxial cable, a fiber optic cable, a twisted pair, a digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, wherein disk often reproduces data magnetically, while disc reproduces data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0194] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein, but should be granted the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A path planning method based on multi-modal joint integration, characterized in that: Methods include: Step S1: image preprocessing and preliminary screening of dynamic areas, including preprocessing the image, extracting image feature points, matching the image at time t-1 with the image at time t, and preliminary screening of dynamic areas; Step S2: based on loosely coupled online calibration, initialization of motion state matrix is ​​performed; Step S3: fine-grained regional segmentation and instance recognition, including after the motion state matrix is ​​initialized, the subsequent on-board camera image is input into the SAM network as input to obtain the fine-grained regional segmentation output, which is then input into the CLIP network to obtain the instance ID corresponding to the refined region, save the output result, compare the targets at time t-1 and time t+1, if there is a target with the same static target instance ID, proceed to step S4, if there is no target with the same static target instance ID, use the instantaneous inertial measurement unit data as the motion data of the frame to proceed to step S6; Step S4: Mapping is performed in groups according to the targets of the matched static target instance IDs, and finally the average rotation matrix is ​​calculated; Step S5: fusing the average rotation matrix with n inertial measurement unit data before and after the frame to obtain a final state matrix, where n is a natural number greater than or equal to 2; Step S6: Save the state matrix and draw the global route.

2. The path planning method based on multi-modal joint integration according to claim 1, characterized in that: Step S1 further comprises: Step S11: pre-processing the captured image, including statistically analyzing the frequency distribution of image pixel values, and making the pixel distribution uniform through equalization to improve contrast and visibility; Step S12: Use Shi-Tomasi to detect image feature points; Step S13: Match the feature points of the image at time t-1 with the image at time t, and then estimate the movement of the image feature points in consecutive frames by analyzing the movement of pixels in the image sequence, so as to track the key points and finally obtain a dynamic area heat map relative to the car.

3. The path planning method based on multi-modal joint integration according to claim 1, characterized in that: Step S2 further comprises: Step S21: combining the sensor information of the inertial measurement unit, performing online calibration based on loose coupling, and aligning information including speed and scale; Step S22: Initialize the motion state matrix of the object.

4. The path planning method based on multi-modal joint integration according to claim 1, characterized in that: Step S3 further comprises: Step S31: input the image to be segmented; Step S32: using an image encoder to extract a high-dimensional feature representation of the input image and output an image feature map; Step S33: encoding the prompt provided by the user into a feature representation through a prompt encoder, and outputting a prompt feature, wherein the prompt provided by the user includes a point, a box, and a text description; Step S34: feature fusion, that is, fusing the image features and the prompt features together through feature splicing or attention mechanism, and outputting the fused features to prepare for subsequent mask decoding; Step S35: Generate and output a segmentation mask based on the fusion features through a mask decoder based on the Transformer architecture. In the segmentation mask, the value of each pixel represents the probability that the pixel belongs to the target object. Step S36: Optimizing the segmentation mask to generate a final refined segmentation result, that is, a refined segmentation mask, wherein the optimization operation includes thresholding and morphological operations; Step S37: combining the segmentation mask with the original image to extract the image area of ​​the target object; Step S38: inputting the image region of the target object into an image-text model to generate a feature vector of the image; Step S39: Match the feature vector of the image with the feature vector of the predefined text tag, use a similarity metric to find the most similar text tag and output it.

5. The path planning method based on multi-modal joint integration according to claim 4, characterized in that: In step S32, the image encoder uses a pre-trained visual model to implement feature extraction, wherein the visual model is a multi-scale visual model based on CrossVIT, which extracts and fuses information through image patches of different sizes to improve the detection capability of targets of different scales.

6. The path planning method based on multi-modal joint integration according to claim 5, characterized in that: Step S32 further includes: Step S321: Obtain a one-dimensional vector representation by encoding through a block encoder and output; Step S322: obtaining a category vector through a category encoder so as to integrate image information for classification; Step S323: obtaining a position vector by position coding to express the position information of the image block in the original image; Step S324: Perform feature extraction and information aggregation on the input sequence through the Transformer encoder.

7. The path planning method based on multi-modal joint integration according to claim 4, characterized in that: Step S34 further comprises: Step S341: projecting the image text features into the same embedding space through a linear projection layer for feature alignment; Step S342: By contrasting the learning loss function, matching text image features are brought closer in the embedding space, while mismatched ones are moved further away.

8. The path planning method based on multi-modal joint integration according to claim 1, characterized in that: Step S4 further comprises: Step S41: Obtaining covariance matrices according to the matching targets respectively; Step S42: Obtain a rotation matrix using singular value decomposition; Step S43: Calculate the mean of the rotation matrices of all targets.

9. The path planning method based on multi-modal joint integration according to claim 8, characterized in that: Step S41 further includes: Step S411: obtaining the centroid through the obtained matching feature points, and removing the centroid; Step S412: Calculate the de-centroided covariance matrix, which represents the change trend of the matching point pairs of the previous and next frames and is used to infer the rotation relationship between the previous and next frames.

10. The path planning method based on multi-modal joint integration according to claim 8, characterized in that: Step S43 further includes: Step S431: for each target, convert the rotation matrix into a quaternion; Step S432: performing weighted averaging on all quaternions to obtain an average quaternion; Step S433: Convert the quaternion into a rotation matrix to obtain an average rotation matrix.

11. The path planning method based on multi-modal joint integration according to claim 1, characterized in that: In step S5, the average rotation matrix obtained in step S4 is jointly calculated with the integral of at least two inertial measurement unit data before and after the frame to calculate the observation motion matrix. After weighting once, the motion state matrix of the object is updated by the weighted motion state value to obtain the predicted motion matrix. The predicted value at time t+1 and the true value at time t+1 in the predicted motion state matrix are weighted again as the output value.

12. A path planning system based on multi-modal joint integration, characterized in that: The system includes: The dynamic area preliminary screening module is configured as image preprocessing and dynamic area preliminary screening, including image preprocessing, image feature point extraction, and preliminary screening of dynamic areas by matching the image at time t-1 with the image at time t. A motion state matrix initialization module is configured as an online calibration based on loose coupling and performs motion state matrix initialization; The region refinement segmentation and instance recognition module is configured to, after the motion state matrix is ​​initialized, send the subsequent on-board camera image as input to the SAM network to obtain the region refinement segmentation output, and then input it to the CLIP network to obtain the instance ID corresponding to the refined region, save the output result, compare the targets at time t-1 and time t+1, if there is a target with the same static target instance ID, enter the average rotation matrix calculation module, if there is no target with the same static target instance ID, use the instantaneous inertial measurement unit data as the motion data of the frame to enter the global route drawing module; The average rotation matrix calculation module is configured to group and map the targets according to the matched static target instance IDs, and finally calculate and find the average rotation matrix; A state matrix acquisition module is configured to fuse the average rotation matrix with n inertial measurement unit data before and after the frame to obtain a final state matrix, where n is a natural number greater than or equal to 2; A global route drawing module is configured to save the state matrix and draw the global route.

13. A computer system for path planning, characterized in that: The method comprises a memory, a processor and program instructions stored in the memory and executable by the processor, wherein the processor executes the program instructions to implement the steps of the path planning method based on multi-modal joint integration as claimed in any one of claims 1 to 11.

14. A computer-readable storage medium for path planning, characterized in that: It stores program instructions executable by a processor to implement the steps of the path planning method based on multi-modal joint integration as described in any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the path planning method based on multi-modal joint integration as claimed in any one of claims 1 to 11 are implemented.

Images