Path planning method and system based on multi-modal mechanism

Abstract

The invention discloses a path planning method and system based on a multi-modal mechanism, and relates to the technical field of artificial intelligence and robot navigation, and the method comprises the steps: extracting visual features, obtaining a visual feature map, recognizing obstacles on the visual feature map by applying a bounding box detection algorithm, and extracting semantic features from road state description; the visual features and the semantic features are aligned through a progressive fusion strategy to generate comprehensive features, feature weights in the progressive fusion strategy are adjusted in real time through a dynamic mask generation mechanism, and if modal conflicts are detected, the modal conflicts are solved through feature importance evaluation and a conflict resolution mechanism; and dynamically adjusting a feasible region by using a deep reinforcement learning model and utilizing the comprehensive characteristics after the modal conflict is solved, and optimizing path selection in a path planning process. According to the method, the adaptability and the real-time performance of path planning can be remarkably improved, and the stability and the reliability of path planning are improved.

Description

Technical Field

[0001] The present invention relates to the technical fields of artificial intelligence and robot navigation, and particularly to a path planning method and system based on a multi-modal mechanism. Background Art

[0002] In the field of robot navigation, path planning is a key technology for achieving autonomous movement and task execution. Traditional path planning methods usually rely on single-modal data input, but the stability and adaptability of single-modal information in dynamic environments are significantly insufficient, easily resulting in unreliable and non-real-time path planning results.

[0003] In recent years, multi-modal data fusion technology has gradually been applied to path planning. By integrating information from different data modalities, it can significantly improve the accuracy of environmental perception and the robustness of path planning. However, existing multi-modal path planning methods still have key problems: on the one hand, most methods adopt simple feature splicing or fixed-weight fusion strategies, without fully considering the importance and dynamic changes of different modal features, resulting in high-dimensional feature redundancy, increasing computational overhead and reducing decision-making efficiency; on the other hand, these methods lack an efficient response mechanism to dynamic environmental changes and the ability to dynamically adjust the priorities of different modal features, making it difficult to achieve efficient path planning in complex dynamic environments. For example, in a warehousing and logistics center, robots often need to cope with dynamically moving obstacles and sudden environmental changes. Traditional single-modal methods usually need to repeatedly collect data and reconstruct the environmental model in these scenarios, resulting in response delays or even task interruptions.

[0004] To address the above problems, an optimized path planning method should achieve real-time integration and flexible adjustment of multi-modal features through a dynamic weight adjustment and feature importance evaluation mechanism, thereby significantly improving the efficiency and reliability of path planning and meeting the navigation requirements of complex dynamic environments. Summary of the Invention

[0005] In view of the problems existing in the existing path planning methods based on a multi-modal mechanism, the present invention is proposed. Therefore, the problem to be solved by the present invention is how to provide a path planning method based on a multi-modal mechanism.

[0006] To solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides a path planning method based on a multi-modal mechanism, which includes obtaining real-time visual data from a camera or depth sensor carried by a logistics robot, extracting visual features through a deep convolutional neural network to obtain a visual feature map, applying a bounding box detection algorithm on the visual feature map to identify the positions and shapes of obstacles, and extracting semantic features from the road state description through natural language processing and semantic analysis;

[0008] A progressive fusion strategy is used to align visual and semantic features to generate comprehensive features. A dynamic mask generation mechanism is used to adjust the feature weights in the progressive fusion strategy in real time. If a modal conflict is detected, the modal conflict is resolved through feature importance evaluation and conflict resolution mechanisms.

[0009] A deep reinforcement learning model is used to dynamically adjust the feasible area using the comprehensive features after resolving modal conflicts, thereby optimizing path selection during the path planning process.

[0010] As a preferred solution of the multimodal path planning method of the present invention, the bounding box detection algorithm includes the following:

[0011] Apply the bounding box detection algorithm on the visual feature map to identify the position and shape of the obstacle, and calculate the bounding box B of each obstacle through the bounding box detection algorithm. i =(x min ,y min ,x max ,y max ), where x min and y min Indicates the coordinates of the upper left corner of the bounding box, x max and y max Represents the coordinates of the lower right corner of the bounding box;

[0012] The obstacle detection function f(V) in the bounding box detection algorithm is expressed as:

[0013]

[0014] Among them, D(B i ) represents the obstacle detection score for the i-th bounding box; V is the visual feature matrix, N is the number of obstacles detected, W and b are the weights and biases of the model, σ is the activation function, V i Represents the visual feature subvector of the i-th detection candidate region.

[0015] As a preferred solution of the multimodal mechanism-based path planning method of the present invention, the extraction of semantic features from the road state description through natural language processing and semantic analysis includes the following:

[0016] Semantic description is represented as text T = {t1, t2, ..., t i}, where t i is the i-th word or phrase describing the road status in the text;

[0017] Each word t is represented as a semantic vector by the word embedding method. iMapped to a high-dimensional vector space, the semantic analysis function F(S) is used to extract the road state feature S i , expressed as:

[0018]

[0019] Among them, S i represents the i-th path state feature after being processed by the Attention mechanism, Q, K, and V1 are query, key, and value matrices respectively; softmax is used as a normalization in the Attention mechanism to ensure that the sum of the attention weights between different positions is 1; K T represents the transposed matrix of K, d k Indicates the length of K.

[0020] As a preferred solution of the multimodal path planning method of the present invention, the method of aligning visual features and semantic features to generate comprehensive features through a progressive fusion strategy includes the following:

[0021] The visual feature V f and semantic features S f Projected into a unified feature space, through the mapping function f map To achieve the preliminary fusion of multimodal features, the expression of the mapping function is:

[0022] V'=f map (V f ;W V )∈R m*k

[0023] S'=f map (S f ;W S )∈R m*k

[0024] Among them, V' is the visual feature after mapping, S' is the semantic feature after mapping, and W V and W S are the projection matrices of visual and semantic features respectively, m and k are the feature dimensions after alignment; R represents a set of real numbers;

[0025] Obtain the initial fused feature F through the splicing operation early , the expression is:

[0026] F early =concat(V',S')

[0027] The multimodal features are weighted and combined through the weighted fusion strategy to generate the final comprehensive feature F fusion , the expression is:

[0028] F fusion = α·V' + β·S'

[0029] α + β = 1

[0030] Among them, α represents the importance of visual features, and β represents the importance of semantic features.

[0031] As a preferred solution of the path planning method based on the multi-modal mechanism described in the present invention, wherein: the use of the dynamic mask generation mechanism to adjust the feature weights in the progressive fusion strategy in real time includes the following:

[0032] By generating a dynamic mask matrix M to control the weights of each modal feature to generate the final integrated feature F fusion Perform weight adjustment;

[0033] After applying the dynamic mask matrix, the fused feature is weight-adjusted to obtain a new weighted feature F weighted , and the expression is:

[0034]

[0035] In the formula, α n is the importance weight coefficient of the nth feature, and f n is the nth multi-modal feature value after preliminary fusion;

[0036] Evaluate the feature importance of the new weighted feature obtained after weight adjustment.

[0037] As a preferred solution of the path planning method based on the multi-modal mechanism described in the present invention, wherein: the evaluation of the feature importance of the new weighted feature obtained after weight adjustment includes the following:

[0038] Calculate the relative importance of each feature through the feature importance evaluation function g(f i ), and dynamically evaluate the contribution of the feature in path planning, and adjust the feature weights in real time. The expression of the feature importance evaluation function g(f i ) is:

[0039]

[0040] Among them, Vr(f i ) is the variance of the feature f i , is the covariance of the feature f i with other features, and Dist(f i , E) represents the distance between the feature f i and the environmental state E, and λ1, λ2, and λ3 are weight coefficients;

[0041] Calculate the weight α of each feature i , represents the relative contribution in path planning decision, and the formula is:

[0042]

[0043] Where n is the number of features;

[0044] If the feature f i and feature f j The cosine similarity Sim(f i ,f j ) is greater than the conflict threshold δ, then it is determined that f i and f j There is a potential conflict. For the detected conflict feature pair (f i ,f j ), according to the feature importance weight α i and α j Determine priorities;

[0045] Keep high-priority features and make weighted corrections to low-priority features. Assume that α i >α j , the adjusted feature importance weight α′ j Expressed as:

[0046] α′ j =α j ·(1-Sim(f i ,f j ))

[0047] Among them, Sim(f i ,f j ) represents the feature f i and feature f j The cosine similarity of

[0048] After completing the priority adjustment and weighted correction, the smooth harmonic function h(f i ,f j ) Balance the conflicting features and use the balanced fusion feature f i,j Expressed as:

[0049]

[0050] The balanced fusion features are input into the deep reinforcement learning model for path planning.

[0051] As a preferred solution of the multimodal path planning method described in the present invention, the method of dynamically adjusting the feasible area using the comprehensive features after resolving the modal conflict using the deep reinforcement learning model includes the following:

[0052] Taking the balanced fusion features as the input, the path planning is optimized through the policy network in the deep reinforcement learning model ;

[0053] In the state space s t , it accepts the input of multimodal features, converts it into the state input of the policy network, determines the best action in the current state, and the policy network estimates the optimal policy function π θ (a|s) to select the next path decision, where are the parameters of the policy network;

[0054] At each time step, the policy network generates a path planning decision a t according to the current state s t , and at the same time makes dynamic adjustments according to the current environmental feedback;

[0055]

[0056] The policy function selects the optimal path direction by calculating the probability distribution of each possible action, and the goal of the policy network is to maximize the expected value E of the future cumulative reward R t , and the expression is:

[0057] maxE[R t |s t ,a t ,π θ

[0058] In the formula, π θ is the policy function;

[0059] By continuously monitoring and evaluating the effect of the current path, and converting the path feedback into an immediate reward γ t or a penalty value;

[0060] Taking the result of each path action as a reward signal, passing it to the value function in reinforcement learning, and updating the policy network parameters to optimize the path selection, and the value function V(s t ) represents the expected cumulative reward at state s t , and the expression is:

[0061]

[0062] where, R t is the accumulation of future rewards, r t+k is the immediate reward, γ is the discount factor, and k represents the time step from the current moment t onwards;

[0063] Adopting the policy gradient method, the policy network parameters are updated through the gradient ascent formula:​

[0064]

[0065] Among them, α1 is the learning rate, is the gradient with respect to the cumulative reward.

[0066] In a second aspect, the present invention provides a path planning system based on a multi-modal mechanism, which includes:

[0067] A multi-modal feature extraction module, configured to obtain visual bounding box information of obstacles and semantic descriptions of road states, and obtain multi-modal features;

[0068] A feature fusion module, configured to achieve alignment and weighted fusion of multi-modal features through a progressive fusion strategy;

[0069] A dynamic mask generation module, configured to generate a dynamic mask matrix to control the weights of each modal feature;

[0070] A feature importance evaluation module, configured to calculate the relative importance of each modal feature and perform dynamic evaluation;

[0071] A path planning module, configured to perform path planning and dynamic adjustment within a feasible region according to the fused multi-modal features.

[0072] In a third aspect, the present invention provides a computer device, including a memory and a processor, where the memory stores a computer program, and: when the processor executes the computer program, the steps of a path planning method based on a multi-modal mechanism are implemented.

[0073] In a fourth aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, and: when the computer program is executed by a processor, the steps of a path planning method based on a multi-modal mechanism are implemented.

[0074] The beneficial effects of the present invention are as follows: By real-time evaluating the importance of modal features in the environment and dynamically adjusting the feature weights, it is ensured to achieve efficient feature fusion and path planning in a complex environment. Compared with the traditional fixed weight strategy, the dynamic mask generation mechanism significantly improves the adaptability and real-time performance of path planning.

[0075] Aiming at the potential conflict problems in multi-modal feature fusion, through feature importance evaluation and priority adjustment, precise correction and balance of conflicting features are achieved, significantly improving the stability and reliability of path planning. Description of the Drawings

[0076] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on these drawings.

[0077] Figure 1 It is a flowchart of a path planning method based on a multi-modal mechanism. Detailed implementation manners

[0078] To make the above objects, features, and advantages of the present invention more understandable, the following will describe the detailed implementation manners of the present invention in conjunction with the accompanying drawings of the specification. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0079] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0080] Secondly, the so-called "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that can be included in at least one implementation manner of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it an embodiment that is separate or selectively exclusive of other embodiments.

[0081] Referring to Figure 1 , which is the first embodiment of the present invention. This embodiment provides a path planning method based on a multi-modal mechanism, including:

[0082] S1: Obtain real-time visual data from the camera or depth sensor carried by the logistics robot, extract visual features through a deep convolutional neural network to obtain a visual feature map, apply a bounding box detection algorithm on the visual feature map to identify the position and shape of obstacles, and extract semantic features from the road state description through natural language processing and semantic analysis;

[0083] Specifically, S1.1: Multi-modal feature extraction. Obtain real-time visual data from the camera or depth sensor carried by the logistics robot, and ensure the quality of visual features through image preprocessing. The visual features are represented as a matrix V ∈ R h*w*c, where V represents the matrix of visual feature representations, R represents the set of real numbers, h and w are the height and width of the image, and c is the number of channels. The visual features include the position, shape, and spatial relationship of the obstacles. The visual features are further extracted by a deep convolutional neural network to obtain a high-level feature map containing spatial information.

[0084] S1.2: Obstacle bounding box detection. Apply a bounding box detection algorithm on the visual feature map to identify the position and shape of the obstacles. Through the bounding box detection algorithm, calculate the bounding box B of each obstacle i =(x min , y min , x max , y max ), where x min and y min represent the coordinates of the upper left corner of the bounding box, and x max and y max represent the coordinates of the lower right corner of the bounding box. The obstacle detection function can be expressed as:

[0085]

[0086] where V is the visual feature matrix, N is the number of detected obstacles, W and b are the weights and biases of the model, and σ is the activation function.

[0087] S1.3: Semantic feature extraction. The semantic features are extracted from the description of the road state through natural language processing and semantic analysis. The semantic description is represented as text T = {t1, t2,..., t i}, where t i is the i-th word or phrase in the text that describes the road state; generate semantic vectors through the word embedding method, and map each word t i to a high-dimensional vector space. Represent the text as a semantic vector S = E·T through the embedding matrix E ∈ R n*d , where E is the embedding matrix, representing a real-valued matrix with dimensions n*d, used to map words or phrases in the text to a high-dimensional vector space; d is the embedding dimension. Then, use the semantic analysis function F(S) to focus on the semantic features related to path planning to form a weighted semantic vector, enabling the robot to understand the environmental attributes.

[0088] where S i represents the i-th path state feature after being processed by the Attention mechanism, and Q, K, and V1 are the query, key, and value matrices respectively; ATTENTION is a mechanism that can dynamically "attend" to different weights of different parts in the sequence when processing sequence data, and Softmax is used as normalization in ATTENTION to ensure that the sum of the attention weights between different positions is 1; KT denotes the transpose matrix of K, d k denotes the length of K.

[0089] S2: Align the visual features and semantic features through a progressive fusion strategy to generate comprehensive features, use a dynamic mask generation mechanism to adjust the feature weights in the progressive fusion strategy in real time, and if a modality conflict is detected, resolve the modality conflict through a feature importance evaluation and conflict resolution mechanism;

[0090] Specifically, the multi-modal feature fusion adopts a progressive fusion strategy of early and late feature extraction to gradually enhance the expression ability of multi-modal features.

[0091] S2.1: In the early fusion stage, through the preliminary joint extraction of the visual feature V f and the semantic feature S f achieve the alignment of basic features.

[0092] Project the visual feature V f and the semantic feature S f into a unified feature space F0, and this space realizes the preliminary fusion of multi-modal features through the mapping function f map (V f , S f ).

[0093] V' = f map (V f ; W V ) ∈ R m*k

[0094] S' = f map (S f ; W S ) ∈ R m*k

[0095] where V' is the mapped visual feature, S' is the mapped semantic feature, W V and W S are the projection matrices of visual and semantic features respectively, m and k are the aligned feature dimensions; R represents the set of real numbers;

[0096] S2.2: Preliminary joint feature splicing, splice the transformed visual feature V' and semantic feature S' to obtain a preliminary fusion feature. The feature F early after preliminary fusion is obtained through the splicing operation:

[0097] F early = concat(V', S')

[0098] This fusion feature has the comprehensiveness of basic features and is suitable for early feature representation.

[0099] S2.3: In the late fusion stage, through the weighted fusion strategy, the multi-modal features are further weighted and combined to generate the final comprehensive features.

[0100] In this process, weights α and β are assigned according to the importance of features in different modalities, representing the importance of visual features and semantic features respectively, and satisfying α + β = 1 for weighted combination. The values of α and β are adaptively adjusted according to the feature importance to optimize the expressive ability of the comprehensive features.

[0101] Comprehensive feature F fusion As the input for path planning, it ensures the effective integration of multi-modal information and improves the robustness and adaptability of path planning in different environments;

[0102] F fusion = α·V′ + β·S′

[0103] where α represents the importance of visual features and β represents the importance of semantic features.

[0104] S2.4: Adopt a dynamic mask generation mechanism to assign dynamic weights to different modal features during the path planning process, enabling the system to flexibly adjust the contributions of multi-modal features in real-time environmental changes. This mechanism generates a dynamic mask matrix M t to control the weight α i (t) of each modal feature, so that the model adaptively assigns feature weights to improve the robustness of path planning.

[0105] After applying the dynamic mask matrix M, the multi-modal fusion feature F fusion is weight-adjusted to obtain a new weighted feature F weighted :

[0106]

[0107] In the formula, α n is the importance weight coefficient of the nth feature, and f n is the nth multi-modal feature value after preliminary fusion, which is a certain dimension feature after the fusion of visual features and semantic features;

[0108] In this way, the contribution degree of each modal feature f i is amplified or reduced by the dynamically adjusted weight α i , enabling the path planning model to flexibly cope with dynamic changes in the environment. Feature importance evaluation is performed on the new weighted feature obtained after weight adjustment.

[0109] S2.5: Conduct feature importance evaluation. During the feature fusion process, to address potential conflicts between different modal features, through the feature importance evaluation function g(f i)Calculate the relative importance of each modal feature and dynamically evaluate its contribution in path planning. This evaluation function aims to adjust the weight of each modal feature in real time according to the current environmental state, enabling the path planning model to more effectively integrate multi-modal information. The feature importance evaluation function g(f i ) is expressed as:

[0110]

[0111] where, Var(f i ) is the variance of feature f i , is the covariance of feature f i with other features, used to measure the correlation between f i and other features, Dist(f i ,E) represents the distance between feature f i and the environmental state E, and λ1, λ2, and λ3 are weight coefficients;

[0112] Calculate the importance weight α i , representing the relative contribution in path planning decisions. The formula is:

[0113]

[0114] In the formula, n is the number of features;

[0115] S2.6: Conflict detection and resolution. If the cosine similarity Sim(f i ,f j ) is greater than the conflict threshold δ, it is determined that f i and f j have potential conflicts. For the detected conflict feature pair (f i ,f j ), first determine the priority according to the magnitudes of their importance α i and α j . Prioritize the retention of high-priority features and perform weighted correction on low-priority features. Assuming α i >α j , after adjustment, it is expressed as:

[0116] α′ j =α j ·(1 - Sim(f i ,f j ))

[0117] After completing the priority adjustment and weighted correction, further adopt the smoothing harmonic function h(f i ,f j)Balance the conflicting features to ensure the effective integration of multimodal information during path planning. Let the balanced fused feature f i,j be expressed as:

[0118]

[0119] Input the balanced fused feature into the deep reinforcement learning model for path planning.

[0120] S3: Use the deep reinforcement learning model to dynamically adjust the feasible region using the comprehensive feature after resolving the modal conflict, and optimize the path selection during the path planning process.

[0121] Specifically, the deep reinforcement learning model includes a multimodal feature extraction module based on a convolutional neural network (CNN) and a multi-head self-attention mechanism, which is used to extract and fuse key information from different modalities, so as to generate a path planning strategy to adapt to complex environmental requirements.

[0122] The CNN-based multimodal feature extraction module is responsible for extracting spatial and semantic features from visual and semantic data. The CNN first converts the input multimodal data F i into a feature map F CNN (F i ), to obtain a low-dimensional feature representation of each modality. After the feature extraction is completed, the multi-head self-attention mechanism further fuses the key information in the multimodal features to highlight important features. The attention mechanism calculates the attention weights of each feature by generating a set of query, key, and value matrices Q, K, V.

[0123] S3.1: During the path planning process, use the deep reinforcement learning model to perform path planning and dynamic adjustment on the feasible region. Using the balanced fused feature as the input, optimize the path planning through the policy network in deep reinforcement learning to achieve the optimization of path planning.

[0124] Specifically, the model receives the input of multimodal features in the state space s t and converts it into the state input of the policy network to determine the best action a t in the current state. The policy network selects the next path decision by estimating the optimal policy function , where are the parameters of the policy network. At each time step, the policy network generates a path planning decision a t based on the current state s t , and at the same time makes dynamic adjustments according to the current environmental feedback, so as to achieve real-time optimization of the path.

[0125]

[0126] This policy function selects the optimal path direction by calculating the probability distribution of each possible action a. The goal of the policy network is to maximize the expected value E of the future cumulative reward R t :

[0127]

[0128] In the formula, is the policy function, θ is the parameter of the policy network, and determines the probability distribution of taking each action a t according to the current state s t ;

[0129] During the execution of path planning, the model continuously monitors and evaluates the effect of the current path through a feedback mechanism, and converts the path feedback into an immediate reward γ t or a penalty value for dynamically optimizing path selection. The feedback mechanism uses the result of each path action as a reward signal and passes it to the value function V(s t ) in reinforcement learning, and updates the policy network parameters to optimize path selection. The value function V(s t ) represents the expected cumulative reward at state s t and is used to evaluate the quality of the current path:

[0130]

[0131] where R t is the accumulation of future rewards, r t+k is the immediate reward, γ is the discount factor, and its value range is usually between 0 and 1. Its role is to measure the importance of future rewards relative to current rewards. The closer γ is to 0, the more the agent focuses on current rewards; the closer γ is to 1, the more the agent values future long-term rewards; k is a non-negative integer representing the time step from the current moment t onwards, used to sum the rewards r t+k at different times to obtain the future cumulative reward, and γ k represents the discount degree of the reward r t+k at time t + k;

[0132] Using the policy gradient method, the policy network parameters are updated through the following gradient ascent formula:

[0133]

[0134] where α1 is the learning rate, is the gradient with respect to the cumulative reward, used to adjust the policy network parameters to enhance efficient path selection.

[0135] The collaborative optimization of the value function and the policy network can maximize the long-term path reward and ensure optimal path selection in complex environments.

[0136] Furthermore, this embodiment also provides a path planning system based on a multimodal mechanism, including:

[0137] A multimodal feature extraction module, configured to obtain the visual bounding box information of obstacles and the semantic description of the road state, and obtain multimodal features;

[0138] A feature fusion module, configured to achieve alignment and weighted fusion of multimodal features through a progressive fusion strategy;

[0139] A dynamic mask generation module, configured to generate a dynamic mask matrix to control the weight of each modal feature;

[0140] A feature importance evaluation module, configured to calculate the relative importance of each modal feature and perform dynamic evaluation;

[0141] A path planning module, configured to perform path planning and dynamic adjustment within the feasible region according to the fused multimodal features.

[0142] This embodiment also provides a computer device applicable to a path planning method based on a multimodal mechanism, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement all or part of the steps of the method described in the embodiments of the present invention as proposed above.

[0143] This embodiment also provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, it executes the method in any optional implementation manner of the above embodiments. Among them, the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, abbreviated as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, abbreviated as EEPROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, abbreviated as EPROM), programmable read-only memory (Programmable Red-Only Memory, abbreviated as PROM), read-only memory (Read-Only Memory, abbreviated as ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0144] The storage medium proposed in this embodiment and the data storage method proposed in the above embodiment belong to the same inventive concept. For the technical details not described in detail in this embodiment, reference may be made to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0145] In summary, this method dynamically adjusts the feature weights by evaluating the importance of modal features in the environment in real time to ensure efficient feature fusion and path planning in complex environments. Compared with the traditional fixed-weight strategy, the dynamic mask generation mechanism significantly improves the adaptability and real-time performance of path planning. Aiming at the potential conflict problem in multi-modal feature fusion, through feature importance evaluation and priority adjustment, precise correction and balance of conflicting features are achieved, significantly improving the stability and reliability of path planning.

[0146] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered within the scope of the claims of the present invention.

Claims

1. A path planning method based on a multi-modal mechanism, characterized in that: including Obtain real-time visual data from a camera or depth sensor mounted on a logistics robot, extract visual features through a deep convolutional neural network to obtain a visual feature map, apply a bounding box detection algorithm on the visual feature map to identify the position and shape of obstacles, and extract semantic features from the road state description through natural language processing and semantic analysis; Align visual features and semantic features through a progressive fusion strategy to generate comprehensive features, use a dynamic mask generation mechanism to adjust the feature weights in the progressive fusion strategy in real time, and if a modality conflict is detected, resolve the modality conflict through a feature importance evaluation and conflict resolution mechanism; Use a deep reinforcement learning model to dynamically adjust the feasible region using the comprehensive features after resolving the modality conflict, and optimize the path selection during the path planning process.

2. The path planning method based on a multimodal mechanism according to claim 1, wherein: The bounding box detection algorithm includes the following: Apply the bounding box detection algorithm on the visual feature map to identify the position and shape of obstacles. Through the bounding box detection algorithm, calculate the bounding box B of each obstacle i =(x min , y min , x max , y max ), where x min and y min represent the coordinates of the upper left corner of the bounding box, and x max and y max represent the coordinates of the lower right corner of the bounding box; The obstacle detection function f(V) in the bounding box detection algorithm is expressed as: Among them, D(B i ) represents the obstacle detection score for the i-th bounding box; V is the visual feature matrix, N is the number of detected obstacles, W and b are the weights and biases of the model, σ is the activation function, and V i represents the visual feature sub-vector of the i-th detection candidate region.

3. The path planning method based on a multi-modal mechanism according to claim 2, characterized in that: The extraction of semantic features from the road state description through natural language processing and semantic analysis includes the following: The semantic description is represented as text T = {t1, t2,..., t i}, where t i is the i-th word or phrase in the text that describes the road condition; Represented as a semantic vector by the word embedding method, each word t i is mapped to a high-dimensional vector space, and the state feature S of the road is extracted by using the semantic analysis function F(S) i , which is expressed as: Among them, S i represents the i-th path state feature processed by the Attention mechanism, and Q, K, and V1 are the query, key, and value matrices respectively; softmax is used as normalization in the Attention mechanism to ensure that the sum of the attention weights between different positions is 1; K T represents the transpose matrix of K, and d k represents the length of K.

4. The path planning method based on a multi-modal mechanism according to claim 3, wherein: The alignment of visual features and semantic features through a progressive fusion strategy to generate comprehensive features includes the following: Project the visual feature V f and the semantic feature S f into a unified feature space, and realize the preliminary fusion of multi-modal features through the mapping function f map The expression of the mapping function is: V' = f map (V f ; W V ) ∈ R m*k S' = f map (S f ; W S ) ∈ R m*k Among them, V' is the mapped visual feature, S' is the mapped semantic feature, and W V and W S are the projection matrices of visual and semantic features respectively, m and k are the aligned feature dimensions; R represents the set of real numbers; Obtain the preliminarily fused feature F through the splicing operation early , and the expression is: F early = concat(V', S') The multi-modal features are weighted and combined through a weighted fusion strategy to generate the final comprehensive feature F fusion , and the expression is: F fusion = α·V'+β·S' α+β＝1 Among them, α represents the importance of visual features, and β represents the importance of semantic features.

5. The path planning method based on a multi-modal mechanism according to claim 4, wherein: The use of a dynamic mask generation mechanism to adjust the feature weights in the progressive fusion strategy in real time includes the following: Generate a dynamic mask matrix M to control the weights of each modal feature and generate the final integrated feature F fusion Perform weight adjustment; After applying the dynamic mask matrix, the fused features are adjusted by weights to obtain the new weighted feature F weighted , and the expression is: where α n is the importance weight coefficient of the nth feature, and f n is the nth multi-modal eigenvalue after preliminary fusion; Perform feature importance evaluation on the new weighted features obtained after weight adjustment.

6. The path planning method based on a multi-modal mechanism according to claim 5, wherein: The feature importance evaluation of the new weighted features obtained after weight adjustment includes the following: Evaluate the relative importance of each feature through the feature importance evaluation function g(f i ), dynamically evaluate the contribution of the feature in path planning, and adjust the feature weight in real time. The expression of the feature importance evaluation function g(f i ) is as follows: where, Var(f i ) is the variance of feature f i , is the covariance of feature f i with other features, Dist(f i , E) represents the distance between feature f i and the environmental state E, and λ1, λ2, and λ3 are weight coefficients; Calculate the importance weight α of each feature i , representing the relative contribution in path planning decision-making, with the formula: In the formula, n is the number of features; If feature f i and feature f j have a cosine similarity Sim(f i , f j ) greater than the conflict threshold δ, then it is determined that f i and f j have a potential conflict. For the detected conflict feature pair (f i , f j ), the priority is determined according to the feature importance weights α i and α j . Retain high-priority features and perform weighted correction on low-priority features. When α i > α j , the adjusted feature importance weight α' j is expressed as: α' j = α j ·(1 - Sim(f i , f j )) Among them, Sim(f i , f j ) represents the cosine similarity between feature f i and feature f j ; After completing the priority adjustment and weighted correction, the smoothing harmonic function h(f i , f j ) is used to balance the conflicting features, and the balanced fused feature f i,j is expressed as: Input the balanced fusion features into a deep reinforcement learning model for path planning.

7. The path planning method based on a multi-modal mechanism according to claim 6, characterized in that: The use of a deep reinforcement learning model to dynamically adjust the feasible region using the comprehensive features after resolving the modality conflict includes the following: Taking the balanced fusion features as the input, the optimization of path planning is realized through the policy network in the deep reinforcement learning model to achieve the optimization of path planning; Receives the input of multi-modal features in the state space s t and converts it into the state input of the policy network to determine the best action in the current state. The policy network estimates the optimal policy function to select the path decision for the next step, where are the parameters of the policy network; At each time step, the policy network generates a path planning decision a based on the current state space s t and dynamically adjusts according to the current environmental feedback; t ​ The policy function selects the optimal path direction by calculating the probability distribution of each possible action. The goal of the policy network is to maximize the expected value E of the future cumulative reward R, and the expression is: t as follows: wherein, is the policy function; Continuously monitor and evaluate the effect of the current path, and convert the path feedback into an immediate reward γt or a penalty value; The result of each path action is used as a reward signal and passed to the value function in reinforcement learning to update the policy network parameters Optimize path selection. The value function V(s t ) represents the expected cumulative reward in the state space s t at a given time, and the expression is as follows: where R t is the accumulation of future rewards, r t+k is the immediate reward, γ is the discount factor, and k represents the time step starting from the current time t and going forward; Adopt a policy gradient method to update the policy network parameters through the gradient ascent formula: where α1 is the learning rate, is the gradient with respect to the cumulative reward.

8. A path planning system based on a multi-modal mechanism, based on the path planning method based on a multi-modal mechanism according to any one of claims 1 to 7, characterized in that: including A multimodal feature extraction module for obtaining visual bounding box information of obstacles and semantic descriptions of road states to obtain multimodal features; A feature fusion module for aligning and weighted fusing multimodal features through a progressive fusion strategy; A dynamic mask generation module for generating a dynamic mask matrix to control the weights of each modality feature; A feature importance evaluation module for calculating the relative importance of each modality feature and performing dynamic evaluation; A path planning module for performing path planning and dynamic adjustment within the feasible region according to the fused multimodal features.

9. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that: When the processor executes the computer program, it implements the steps of any one of claims 1 to 7, a path planning method based on a multimodal mechanism.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of any one of claims 1 to 7, a path planning method based on a multimodal mechanism.

Images