End-to-end automatic navigation method, system, device, medium and program product

Abstract

This application relates to the field of artificial intelligence technology, providing an end-to-end automatic navigation method, system, device, medium, and program product. The method includes: inputting a multi-moment image set observed by a camera integrated into a mobile platform and target position data into a control command prediction model, and outputting control commands for the mobile platform to navigate to the target position. The end-to-end automatic navigation method provided in this application relies solely on images and target position data observed by the mobile platform as input, and then combines this with noise for multimodal information fusion, enhancing perception of complex scenes, eliminating intermediate decision-making processes, directly generating control commands from perception, reducing system latency, and improving the accuracy and efficiency of autonomous navigation on the mobile platform. Simultaneously, the end-to-end architecture significantly simplifies system complexity, requiring only a camera and computing unit to achieve automatic navigation, greatly reducing hardware costs and integration difficulty, and the lightweight design is suitable for deployment on mobile platforms with limited computing resources.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an end-to-end automatic navigation method, system, device, medium and program product. Background Technology

[0002] In many scenarios of mobile platform applications, automatic navigation is an extremely critical technology. It is a technology that autonomously plans and executes a movement path by perceiving the external environment, understanding the relationship between its own position and the target position, and finally reaching the designated target.

[0003] Existing automatic navigation technologies require the collaborative work of multiple sensors to collect multimodal environmental information, synchronize and align this information, generate fused features, and then plan a route based on these features, generating specific control commands accordingly. Therefore, existing automatic navigation systems are structurally complex, requiring multiple sensors to work together, increasing hardware costs and integration difficulty, and lacking lightweight design. Summary of the Invention

[0004] This application provides an end-to-end automatic navigation method, system, device, medium, and program product to address the problem of the lack of lightweight design in the architecture of existing automatic navigation systems.

[0005] In a first aspect, this application provides an end-to-end automatic navigation method, comprising:

[0006] Acquire multi-time image sets and target location data observed by the camera integrated into the mobile platform;

[0007] The multi-time image set and the target location data are input into the control command prediction model to obtain the control command output by the control command prediction model to navigate the mobile platform to the target location;

[0008] The control command is generated based on multimodal fusion features, which are obtained by fusing noise, a first image feature set, and target location features. The first image feature set is extracted from the multi-time image set, and the target location features are extracted from the target location data.

[0009] In one embodiment, the control command prediction model includes a command noise-adding module, a feature extraction module, a feature fusion module, and a command prediction module.

[0010] The instruction noise-generating module is used to generate noise for control instructions;

[0011] The feature extraction module is used to extract a first image feature set from the multi-time image set and to extract target location features from the target location data;

[0012] The feature fusion module is used to fuse the noise, the first image feature set, and the target location features to generate multimodal fusion features;

[0013] The instruction prediction module is used to generate control instructions for the mobile platform to navigate to the target location based on the multimodal fusion features.

[0014] In one embodiment, the instruction noise-adding module employs a variational autoencoder architecture, and the noise of the control instruction is generated in the following manner:

[0015] If the random latent variable mechanism is enabled, latent variable embeddings are sampled from the training set distribution and the latent variable embeddings are determined as noise for the control command;

[0016] If the random latent variable mechanism is not enabled, the mean of the training set distribution will be determined as noise in the control commands.

[0017] In one embodiment, the first image feature set includes image features corresponding to images at each time point; the feature fusion module includes a feature enhancement submodule and a multimodal fusion submodule; the feature enhancement submodule is used to perform feature enhancement processing on the regions related to the target location in each of the image features using an attention mechanism to obtain each enhanced image feature; the multimodal fusion submodule is used to perform multimodal fusion of the noise, the second image feature set and the target location features to obtain multimodal fused features, wherein the second image feature set includes each of the enhanced image features.

[0018] In one embodiment, the enhanced image features are generated in the following manner:

[0019] The image features are flattened to obtain flattened image features;

[0020] An attention feature is generated based on the target location features and the flattened image features using an attention mechanism.

[0021] The attention features are fused with each of the image features to obtain the enhanced image features.

[0022] In one embodiment, the multimodal fusion feature is generated in the following manner:

[0023] Using the Transformer architecture, the noise, the second image feature set, and the target location features are fused in a multimodal manner to obtain the multimodal fused features.

[0024] In one embodiment, the control instruction includes multi-dimensional control parameters; the control instruction satisfies comprehensive physical constraints, which are obtained by fusing the physical constraints corresponding to each control parameter; the sequence dimension of the control instruction satisfies dimensional constraints, which are determined based on a preset batch size, a preset sequence length, and a number of dimensions.

[0025] In one embodiment, the control command prediction model is trained in the following manner:

[0026] Acquire multi-time image sets and target location data samples observed by the mobile platform;

[0027] Model training is performed based on the multi-time image set samples and the target location data samples;

[0028] The total loss value of the model prediction is determined based on the predicted loss value and relative entropy between the predictive control command and the actual control command.

[0029] Based on the total loss value, the model parameters are iteratively updated until the model converges to obtain the control command prediction model.

[0030] Secondly, this application also provides an end-to-end automatic navigation system, comprising:

[0031] The acquisition module is used to acquire multi-time-lapse image sets and target location data observed by the camera integrated into the mobile platform;

[0032] The control command generation module is used to input the multi-time image set and the target location data into the control command prediction model to obtain the control command output by the control command prediction model to navigate the mobile platform to the target location;

[0033] The control command is generated based on multimodal fusion features, which are obtained by fusing noise, a first image feature set, and target location features. The first image feature set is extracted from the multi-time image set, and the target location features are extracted from the target location data.

[0034] Thirdly, this application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the end-to-end automatic navigation methods described above.

[0035] Fourthly, this application also provides a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the end-to-end automatic navigation methods described above.

[0036] Fifthly, this application also provides a computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, and which, when executed by the processor, implements the steps of any of the end-to-end automatic navigation methods described above.

[0037] The end-to-end automatic navigation method, system, device, medium, and program products provided in this application rely solely on image and target location data observed by the mobile platform as input. They extract a first set of image features and target location features, and then combine these with noise to perform multimodal information fusion, enhancing perception of complex scenes. This eliminates intermediate decision-making processes and directly generates control commands for navigation to the target location from the perception data, reducing system latency and improving the accuracy and efficiency of autonomous navigation on the mobile platform. Simultaneously, the end-to-end architecture significantly simplifies system complexity, requiring only a camera and computing unit for automatic navigation. Hardware costs and integration difficulties are greatly reduced, and the lightweight design is suitable for deployment on mobile platforms with limited computing resources. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is one of the flowcharts of the end-to-end automatic navigation method provided in this application.

[0040] Figure 2 This is the second flowchart of the end-to-end automatic navigation method provided in this application.

[0041] Figure 3 This is a schematic diagram of the end-to-end automatic navigation system provided in this application.

[0042] Figure 4 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0044] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein.

[0045] The following is combined Figures 1-4 This application describes the end-to-end automatic navigation method, system, device, medium, and program products provided.

[0046] The end-to-end automatic navigation method provided in this application embodiment can be implemented based on an end-to-end automatic navigation system. Therefore, this application embodiment uses an end-to-end automatic navigation system as the execution subject to describe the end-to-end automatic navigation method.

[0047] Combination Figure 1 and Figure 2 , Figure 1 This is one of the flowcharts illustrating the end-to-end automatic navigation method provided in this application. Figure 2 This is the second flowchart of the end-to-end automatic navigation method provided in this application.

[0048] like Figure 1 As shown, the end-to-end automatic navigation method includes the following steps:

[0049] Step 101: Obtain a multi-time image set and target location data observed by the camera integrated into the mobile platform;

[0050] Step 102: Input the multi-time image set and the target location data into the control command prediction model to obtain the control command output by the control command prediction model for navigating the mobile platform to the target location;

[0051] The control command is generated based on multimodal fusion features, which are obtained by fusing noise, a first image feature set, and target location features. The first image feature set is extracted from the multi-time image set, and the target location features are extracted from the target location data.

[0052] Specifically, in this embodiment, the mobile platform only needs to integrate a camera, eliminating the need for additional complex sensor combinations. This significantly reduces the hardware cost and integration complexity of the mobile platform. Furthermore, multi-sensor data requires addressing data synchronization issues; this embodiment directly processes the image stream, eliminating the need for complex data alignment.

[0053] As the sole observation device, the camera is responsible for acquiring multi-timeframe image sets, providing rich visual information for subsequent control command generation. In practical applications, a high-resolution, high-frame-rate industrial camera can be used to ensure clear and stable image data under various lighting conditions. Simultaneously, the camera's installation position and angle can be carefully designed and adjusted to ensure comprehensive and accurate capture of the environmental information surrounding the mobile platform.

[0054] First, using a camera integrated into the mobile platform, images of the surrounding environment are captured at multiple different times, resulting in a multi-moment image set, including images from historical moments and the current moment. Simultaneously, relevant data about the target location is acquired; this data can be pre-defined coordinates or parameters related to the target location obtained through other positioning methods.

[0055] Furthermore, the observed multi-time image set and target location data are input into the control command prediction model to obtain the control command output by the control command prediction model for navigating the mobile platform to the target location.

[0056] In the prediction process, the control command prediction model first extracts features from a multi-time-lapse image set. Specifically, it extracts features from the image at each time step to obtain the image features for that time step, thus forming the first image feature set. These image features reflect key information in the image, such as the shape, color, texture of objects, and the layout of the scene. Simultaneously, it extracts features from the target location data to obtain target location features. These target location features accurately represent relevant information about the target's location, such as the target's coordinates in space and its relative position to the mobile platform.

[0057] Furthermore, when calling the control command prediction model for prediction, a command noise-adding module is introduced to add noise to the subsequent predicted control commands. Typically, this module generates controllable random noise according to the noise distribution patterns obtained during training and incorporates it into the control command prediction process. The purpose of this is to enhance the model's robustness to command prediction under different environments, ensuring that the final control commands can still guide the mobile platform to the target location accurately and stably even in the presence of certain interference factors. For example, in real-world scenarios, there may be uncertainties such as signal interference and sensor errors. Adding noise through the command noise-adding module for simulation training allows the control command prediction model to better adapt to these complex situations, improving the accuracy and stability of navigation.

[0058] Furthermore, the control command prediction model fuses noise, the first image feature set, and target location features to obtain multimodal fused features. During the fusion process, the control command prediction model comprehensively considers the interference factors caused by noise, the environmental visual information reflected by the first image feature set, and the target location-related information determined by the target location features. Through specific algorithms and model structures, it delves into the inherent connections and patterns among these information.

[0059] Furthermore, based on the fused feature information, the control command prediction model uses a pre-trained prediction mechanism to generate control commands for the mobile platform to navigate to the target location. These control commands include key parameters such as the mobile platform's angular velocity, linear velocity, and acceleration, which can guide the mobile platform to move accurately and efficiently toward the target location.

[0060] Control commands are transmitted to the execution system of the mobile platform to enable automatic navigation to the target location.

[0061] The end-to-end automatic navigation method provided in this application relies solely on image and target location data observed by the mobile platform as input. It extracts a first set of image features and target location features, then combines these with noise to perform multimodal information fusion, enhancing perception of complex scenes. This eliminates intermediate decision-making processes and directly generates control commands for navigation to the target location from the perception data, reducing system latency and improving the accuracy and efficiency of autonomous navigation on the mobile platform. Simultaneously, the end-to-end architecture significantly simplifies system complexity, requiring only a camera and computing unit for automatic navigation. This greatly reduces hardware costs and integration difficulty, and the lightweight design is suitable for deployment on mobile platforms with limited computing resources.

[0062] In one embodiment, the control command prediction model includes a command noise-adding module, a feature extraction module, a feature fusion module, and a command prediction module:

[0063] The instruction noise-generating module is used to generate noise for control instructions;

[0064] The feature extraction module is used to extract a first image feature set from the multi-time image set and to extract target location features from the target location data;

[0065] The feature fusion module is used to fuse the noise, the first image feature set, and the target location features to generate multimodal fusion features;

[0066] The instruction prediction module is used to generate control instructions for the mobile platform to navigate to the target location based on the multimodal fusion features.

[0067] Specifically, the control command prediction model includes a command noise-adding module, a feature extraction module, a feature fusion module, and a command prediction module. The command noise-adding module expands the sampling space by adding controllable random noise to the control commands, improving the system's robustness and generalization ability. The feature extraction module adopts a multimodal perception architecture, simultaneously processing visual image information and target location information to construct a comprehensive representation of the environment. The feature fusion module achieves efficient fusion of multimodal information. The command prediction module converts the multimodal fused features into precise control commands.

[0068] Understandably, after inputting the multi-time image set and target location data into the control command prediction model, the command noise module generates noise for the control commands and outputs it.

[0069] Furthermore, the multi-time image set and target location data are transmitted to the feature extraction module. The feature extraction module performs feature extraction on the multi-time image set to obtain the first image feature set. At the same time, it performs feature extraction on the target location data to obtain the target location features. Then, the first image feature set and the target location features are output.

[0070] Furthermore, the noise, the first image feature set, and the target location features are transmitted to the feature fusion module, which fuses the noise, the first image feature set, and the target location features to generate and output multimodal fused features.

[0071] Furthermore, the multimodal fusion features are transmitted to the command prediction module, which makes predictions based on the multimodal fusion features and outputs control commands for the mobile platform to navigate to the target location.

[0072] This application embodiment applies an integrated architecture of a noise addition module, a feature extraction module, a feature fusion module, and a command prediction module to the control command prediction model, achieving deep synergy between noise robustness and multimodal information. It can accurately capture scene changes and target position relationships in dynamic environments, effectively improving the accuracy, anti-interference ability, and environmental adaptability of mobile platform navigation control commands, and ensuring the stability and reliability of the navigation process.

[0073] In one embodiment, the instruction noise-adding module employs a variational autoencoder architecture, and the noise of the control instruction is generated in the following manner:

[0074] If the random latent variable mechanism is enabled, latent variable embeddings are sampled from the training set distribution and the latent variable embeddings are determined as noise for the control command;

[0075] If the random latent variable mechanism is not enabled, the mean of the training set distribution will be determined as noise in the control commands.

[0076] Specifically, the instruction noise addition module adopts a variational autoencoder (VAE) architecture. Through probabilistic modeling and noise injection of the VAE architecture, the system can learn robust feature representations of the input information. Even when there are disturbances or uncertainties in the control instructions, it can maintain stable performance and ultimately improve the generalization ability.

[0077] During training, the control command samples and target location data samples of the mobile platform are used as the training set. The training set is encoded and converted into probability distribution parameters of the latent variable space.

[0078] Using control command samples and target position data samples as input, the encoder network processes this information to obtain a high-dimensional feature vector, namely:

[0079]

[0080] in, This represents a sample of control commands; This represents a sample of data indicating the target location. Indicates the encoder network. This represents a high-dimensional feature vector.

[0081] Then, a projection network is used to transform the high-dimensional feature vectors, outputting the mean and log-variance of the latent variable distribution, i.e.:

[0082]

[0083] in, This represents the mean; Represents the logarithmic variance; Represents a high-dimensional feature vector; This represents the projection network. This step maps the input information to normal distribution parameters in the latent variable space, laying the foundation for the subsequent introduction of randomness.

[0084] Based on the mean obtained above Sum of logarithmic variance By introducing random noise to generate diverse latent variable samples, the sampling space can be expanded.

[0085] First, the logarithm variance... Convert to standard deviation The formula is:

[0086] ;

[0087] Simultaneously generate random noise that follows a standard normal distribution. ,in It is an identity matrix.

[0088] Combined with mean Standard deviation and random noise The final latent variables are generated through reparameterization techniques, namely:

[0089] ;

[0090] in, This represents a latent variable.

[0091] The core of this step is to introduce random noise. , making latent variables Instead of being limited to a deterministic distribution, it is randomly generated from the expanded sampling space, thereby simulating command changes under different noise interferences and improving the system's robustness to input disturbances.

[0092] During the inference phase, controllable randomness is introduced to generate diverse yet stable control strategies. When implementing randomness control during the inference phase, the first step is to determine whether to enable the random latent variable mechanism, i.e., to check whether the parameter use_random_latent is True.

[0093] If the random latent variable mechanism is enabled, a standard normal distribution will be generated. random variables The latent variable embeddings are obtained by scaling them by a factor of 0.1 (which can be adjusted according to the actual situation). This latent variable is then embedded as noise in the control command.

[0094] If the random latent variable mechanism is not enabled, the latent variable embedding is directly taken from the standard normal distribution. The mean of the latent variable is embedded as noise in the control command.

[0095] It can be expressed by the following formula:

[0096] .

[0097] In this way, the system can introduce controllable randomness when needed, thereby generating diverse control strategies, while ensuring that randomness is within a preset range to maintain overall stability.

[0098] This application's embodiments achieve flexibility and controllability in noise generation through a dual-path design that enables / disables the random latent variable mechanism. When enabled, latent variable embeddings are sampled from the training set distribution as noise, introducing randomness to simulate diverse environmental disturbances and enhancing the system's adaptability to complex scenarios. When disabled, the mean of the training set distribution is used as noise, ensuring output stability to meet the requirements of deterministic tasks. This mechanism captures the distribution characteristics of training data through the probabilistic modeling capability of the variational autoencoder and dynamically adjusts the noise generation strategy according to the actual scenario, achieving a balance between randomness and stability, ultimately improving the robustness and generalization ability of control commands in different environments.

[0099] In one embodiment, the first image feature set includes image features corresponding to images at each time point; the image features corresponding to images at each time point are generated in the following manner:

[0100] If the image is a visible light image, then visual features are extracted from the visible light image to obtain the image features corresponding to the image;

[0101] If the image includes the visible light image and the depth image, then visual information is extracted from the visible light image to obtain visual features, and depth information is extracted from the depth image to obtain depth features; the visual features and the depth features are then fused to obtain the image features corresponding to the image.

[0102] Specifically, typically, the camera captures visible light images. Optionally, to enhance the perception of environmental geometry, depth images can also be captured simultaneously, and the depth image information can be fused.

[0103] If only visible light images are acquired, mature visual feature extraction algorithms, such as Convolutional Neural Network (CNN) algorithms, can be used directly to extract representative visual features from the visible light images. These visual features can reflect key information such as color, texture, and shape in the image, thus constituting the image features corresponding to that moment.

[0104] The visual feature extraction process is as follows:

[0105]

[0106] in, Indicates the first Visual features corresponding to a visible light image at a given moment; Indicates the first Visible light images at each moment; This represents a visual information extraction network; Indicates the 0th to time.

[0107] If both visible light images and depth images are acquired simultaneously, the same visual feature extraction algorithm described above can be used to obtain visual features for the visible light images; for the depth images, a specialized depth information extraction algorithm, such as a stereo matching-based algorithm, can be used to extract the depth information of objects in the image and obtain depth features.

[0108] The deep feature extraction process is as follows:

[0109]

[0110] in, Indicates the first The depth features corresponding to the depth image at each time point; Indicates the first Depth images at each moment; This represents a deep information extraction network; Indicates the 0th to time.

[0111] Furthermore, by using specific feature fusion methods, such as weighted fusion or stitching operations, visual features and depth features are fused together, so that the fused image features contain both rich visual information and accurate depth information, thereby obtaining more comprehensive and accurate image features corresponding to the image at that moment.

[0112] The feature fusion process is as follows:

[0113]

[0114] in, and They refer to the first Visual and depth features at each moment; Indicates feature concatenation operation; This represents the projection function, used to map the concatenated features to a new space; Indicates the first Image features obtained by fusing visual and depth features at each time point.

[0115] This application embodiment employs differentiated processing strategies for different types of images to ensure the generation of comprehensive image features under different data input scenarios, thereby achieving accuracy and scene adaptability in feature extraction. At the same time, it effectively enhances the ability of subsequent tasks to perceive complex environments and enriches feature representation.

[0116] In one embodiment, the target location feature is generated in the following manner:

[0117] Feature extraction is performed on the target location data to obtain the target location features.

[0118] Specifically, taking the target location coordinates as an example, the target location data is converted into a high-dimensional feature representation so that it can be fused with visual features.

[0119] The target location feature extraction process is as follows:

[0120]

[0121] in, Indicates the target's location coordinates; and Represents a linear transformation function; It is an activation function; Indicates the target location features.

[0122] The embodiments of this application significantly improve the accuracy of spatial location perception by converting the target location coordinates into a high-dimensional feature representation and effectively fusing it with visual features.

[0123] In one embodiment, the first image feature set includes image features corresponding to images at each time point; the feature fusion module includes a feature enhancement submodule and a multimodal fusion submodule; the feature enhancement submodule is used to perform feature enhancement processing on the regions related to the target location in each of the image features using an attention mechanism to obtain each enhanced image feature; the multimodal fusion submodule is used to perform multimodal fusion of the noise, the second image feature set and the target location features to obtain multimodal fused features, wherein the second image feature set includes each of the enhanced image features.

[0124] Specifically, the feature fusion module includes a feature enhancement submodule and a multimodal fusion submodule. The feature enhancement submodule introduces a target-guided attention mechanism to enhance the understanding of target orientation; the multimodal fusion submodule is based on a multimodal fusion algorithm to achieve efficient fusion of multimodal information.

[0125] Specifically, the feature enhancement submodule calculates the relevance weights between different regions in each image feature and the target location, and uses an attention mechanism to enhance the features of highly correlated regions, thereby highlighting visual information closely related to the target location. The multimodal fusion submodule employs a multimodal fusion strategy to splice and fuse the enhanced image features, noise information, and target location features along the feature dimension to generate multimodal fused features.

[0126] The embodiments of this application not only improve the discriminativeness of features through the attention mechanism, but also achieve deep collaboration of complementary information through multimodal fusion, thereby enhancing the perception ability and robustness of the control command prediction model in complex scenarios, and ensuring that the mobile platform generates accurate and stable navigation control strategies in dynamic environments.

[0127] In one embodiment, the enhanced image features are generated in the following manner:

[0128] The image features are flattened to obtain flattened image features;

[0129] An attention feature is generated based on the target location features and the flattened image features using an attention mechanism.

[0130] The attention features are fused with each of the image features to obtain the enhanced image features.

[0131] Specifically, by introducing a target-guided attention mechanism, a multi-head attention mechanism can be used to direct visual features to focus on regions related to the target. The relevant formula for the multi-head attention mechanism is shown below:

[0132]

[0133] in, Represents the query vector; Represents the key vector; Represents a value vector; express and The dimension; Indicates the scaling factor; This represents a soft maximization function used to calculate the similarity between the query vector and the key vector, and to convert this similarity into a probability distribution; This indicates the output of attention.

[0134] Specifically, each image feature is flattened to obtain flattened image features, whose dimensions are unified into a format suitable for subsequent calculations.

[0135] Furthermore, attention features are generated by utilizing attention mechanisms, such as multi-head attention mechanisms, based on target location features and flattened image features.

[0136] Query vector Obtained through a linear transformation of target location features, it carries unique information about the target's location in the navigation task and is used to find related regions within image features. Key vector Sum value vector The key vector is extracted from the flattened image features, and the value vector is used for similarity matching with the query vector, while the value vector contains the specific content information of the image features.

[0137] In multi-head attention mechanisms, the query vector is used... Key vector Sum value vector The inputs are fed into multiple independent attention heads, each learning different attention weights. These attention heads compute in parallel, concatenating their outputs to form the final attention output, which is then converted into a high-dimensional feature representation for fusion with image features. This multi-head design allows the model to simultaneously focus on multiple important regions in the image, thereby improving navigation accuracy and robustness.

[0138] The output attention features are represented by the following formula using a multi-head attention mechanism:

[0139]

[0140]

[0141] in, Indicates the target location features; Represents the flattened image features; This represents the multi-head attention mechanism; Indicates attention output; and Represents a linear transformation function; It is an activation function; This indicates attentional characteristics.

[0142] Furthermore, the dimensions of the generated attention features are expanded to obtain expanded attention features, which have the same dimensions as the original image features, so that the attention features and image features can be fused to obtain enhanced image features.

[0143] Optionally, a weighted summation or other fusion strategy can be adopted, taking the weighted summation method as an example:

[0144]

[0145] in, Indicates the first Image features at any given time; This represents the weighting coefficient, which can be set according to the actual situation. This represents the attention features after expanding the dimensions; Indicates the first Enhanced image features at different times.

[0146] This application utilizes an attention mechanism to focus image features on key regions related to the target location, generating location-oriented attention features. These features are then fused with the original image features. This process preserves image detail while highlighting the representation intensity of target-related regions. It effectively suppresses interference from complex backgrounds, enhances the discriminative information related to the target location in image features, improves the utilization rate of core visual information in subsequent multimodal fusion and control command prediction, and ultimately optimizes the environmental perception accuracy and target orientation of the mobile platform during navigation.

[0147] In one embodiment, the multimodal fusion feature is generated in the following manner:

[0148] Using the Transformer architecture, the noise, the second image feature set, and the target location features are fused in a multimodal manner to obtain the multimodal fused features.

[0149] Specifically, using the Transformer architecture, noise, a second image feature set, and target location features are fused in a multimodal manner to obtain multimodal fused features.

[0150] The Transformer-based encoder-decoder architecture deeply fuses enhanced image features, noise, and target location features. In the encoder, multi-level feature extraction and abstraction are performed on the input multimodal features, capturing the intrinsic relationships between different modal features through a self-attention mechanism. In the decoder, multimodal information is further integrated and optimized based on the feature representation output by the encoder, ultimately generating multimodal fused features. This multimodal fusion approach effectively utilizes the complementarity of different modal data, improving the understanding and representation of environmental information, and providing richer and more accurate information support for subsequent automatic navigation decisions.

[0151] The noise, the second image feature set, and the target location features are deeply fused using an encoder-decoder Transformer architecture:

[0152]

[0153] in, This represents the second image feature set, which contains each enhanced image feature; Indicates noise; Indicates the target location features; in addition, This indicates the query embedding, which is used as input for the decoder to generate denoising control instructions. The position code representing the moment of image acquisition can be introduced into the multimodal information fusion process; This represents the multimodal fusion feature.

[0154] This application embodiment performs cross-modal fusion of enhanced image feature sets, noise, and target location features, fully combining multi-dimensional information such as visual semantics, environmental randomness, and spatial location to generate comprehensive multi-modal fusion features. This achieves deep synergy of complementary information, which not only improves the perception of complex environments but also enhances the accuracy and robustness of navigation decisions.

[0155] In one embodiment, the control command is generated in the following manner:

[0156] Based on the multimodal fusion features, control instructions for the mobile platform to navigate to the target location are generated; the control instructions include multi-dimensional control parameters; the control instructions satisfy comprehensive physical constraints, which are obtained by fusing the physical constraints corresponding to each control parameter; the sequence dimensions of the control instructions satisfy dimensional constraints, which are determined based on a preset batch size, a preset sequence length, and a number of dimensions.

[0157] Specifically, a specially designed neural network layer is used to map the high-dimensional feature of multimodal fusion into specific control commands, as shown in the following formula:

[0158]

[0159]

[0160] in, Representing multimodal fusion features through linear transformation and activation function Activate to obtain hidden features Hidden features Through linear transformation Receive control commands During the generation of control commands, this neural network layer can automatically adjust the weight parameters of the linear transformation through continuous learning and optimization. and bias parameters To adapt to different environments and navigation tasks.

[0161] The control commands include multi-dimensional control parameters, including but not limited to linear velocity and angular velocity components. During the generation of control commands, they must satisfy comprehensive physical constraints, which are obtained by fusing the physical constraints corresponding to the control parameters across various dimensions.

[0162] Taking linear velocity and angular velocity as control parameters as examples, the output range is limited by an activation function to ensure the physical feasibility of the control commands. That is:

[0163]

[0164]

[0165]

[0166] in, This represents the Sigmoid function. Represents the hyperbolic tangent function; and These represent the first and second components of the original output, namely the linear velocity and angular velocity of the original output, respectively. and These represent the linear velocity and angular velocity, respectively, whose output range is limited by the activation function. The first formula serves as the physical constraint for linear velocity, the second formula as the physical constraint for angular velocity, and the third formula as the combined physical constraint obtained by fusing the physical constraints for linear velocity and angular velocity.

[0167] By compressing the original instructions into a finite range through activation functions, abrupt changes in instructions caused by noise are avoided, resulting in continuous and smooth changes in multi-dimensional control parameters such as linear velocity and angular velocity. Simultaneously, this ensures that the final instructions do not exceed the hardware performance limits of the mobile platform, filtering out instructions that exceed physical feasibility due to noise. By comprehensively considering physical constraints, the smoothness and physical feasibility of control instructions can be ensured, achieving effective instruction denoising.

[0168] Furthermore, during the generation of control instructions, the sequence dimension of the control instructions must also satisfy dimensional constraints, which are determined based on a preset batch size, a preset sequence length, and the number of dimensions. Through these dimensional constraints, the system can maintain the time step dimension, supporting the generation of continuous control instruction sequences in a single inference operation.

[0169] Dimensional constraints are expressed by the following formula:

[0170]

[0171] in, Indicates control commands; Indicates the preset batch size; Indicates the preset sequence length; Indicates the number of dimensions; and The process has been determined during model training.

[0172] The embodiments of this application employ a dual constraint mechanism in the process of predicting control commands. By comprehensively considering physical constraints, the physical feasibility and smoothness of the commands are ensured. At the same time, dimensional constraints are used to ensure the continuous generation capability of the command sequence. This not only guarantees the denoising effect of the control commands but also takes into account the engineering efficiency of the algorithm implementation. Ultimately, a precise and stable control strategy is generated, which improves the navigation reliability and execution accuracy of the mobile platform in complex environments.

[0173] Understandably, the instruction noise addition module and instruction prediction module form a noise addition and denoising framework. Based on the variational autoencoder design, it can effectively cope with environmental uncertainties, realize the modeling of uncertainties, and improve the system robustness more effectively than parameter space noise.

[0174] In one embodiment, the control command prediction model is trained in the following manner:

[0175] Acquire multi-time image sets and target location data samples observed by the mobile platform;

[0176] Model training is performed based on the multi-time image set samples and the target location data samples;

[0177] The total loss value of the model prediction is determined based on the predicted loss value and relative entropy between the predictive control command and the actual control command.

[0178] Based on the total loss value, the model parameters are iteratively updated until the model converges to obtain the control command prediction model.

[0179] Specifically, the system acquires multi-time-lapse image sets and target location data samples observed by the mobile platform, and trains the model based on these sample data.

[0180] The model training objectives include minimizing the difference between the predicted control command and the actual control command, and minimizing the KL divergence (i.e., relative entropy).

[0181] The loss function is expressed by the following formula:

[0182]

[0183] in, This represents the total loss value; This represents the predicted loss value between the predicted control command and the actual control command. The tradeoff coefficient representing the KL divergence; Indicates KL divergence; This represents the approximate posterior distribution. Representing latent variables, This represents a condition variable, which is the probability distribution of a latent variable estimated given the condition variable. This represents the prior distribution, i.e., the probability distribution of the pre-defined latent variables. The total model loss consists of the prediction loss and the KL divergence regularization term, used to balance the predictive power of the model with the reasonableness of the probability distribution.

[0184] The total loss value predicted by the model is calculated in the above manner. Based on the total loss value, the model parameters are iteratively updated until the model converges to obtain the control command prediction model.

[0185] The model training uses supervised learning instead of reinforcement learning, and can learn directly from expert demonstration data. It does not require an experience replay pool or complex learning rate adjustment strategies, and also avoids complex reward function design, making the training process more efficient and concise.

[0186] The embodiments of this application adopt an end-to-end training method, which eliminates the need for manual design of intermediate representations or rules, reduces information loss, and enables adaptive learning of the mapping from raw perception information to control commands, thereby achieving efficient and robust autonomous navigation capabilities.

[0187] Figure 3 This is a schematic diagram of the end-to-end automatic navigation system provided in this application.

[0188] like Figure 3 As shown, the end-to-end automatic navigation system includes:

[0189] The acquisition module 310 is used to acquire a set of multi-time images and target location data observed by the camera integrated into the mobile platform;

[0190] The control command generation module 320 is used to input the multi-time image set and the target location data into the control command prediction model to obtain the control command output by the control command prediction model for navigating the mobile platform to the target location.

[0191] The control command is generated based on multimodal fusion features, which are obtained by fusing noise, a first image feature set, and target location features. The first image feature set is extracted from the multi-time image set, and the target location features are extracted from the target location data.

[0192] The end-to-end automatic navigation system provided in this application relies solely on image and target location data observed by the mobile platform as input. It extracts a first set of image features and target location features, then combines these with noise to perform multimodal information fusion, enhancing perception of complex scenes. This eliminates intermediate decision-making processes and directly generates control commands for navigation to the target location from the perception data, reducing system latency and improving the accuracy and efficiency of autonomous navigation on the mobile platform. Simultaneously, the end-to-end architecture significantly simplifies system complexity, requiring only a camera and computing unit for automatic navigation. This greatly reduces hardware costs and integration difficulty, and the lightweight design is suitable for deployment on mobile platforms with limited computing resources.

[0193] In one embodiment, the control command prediction model includes a command noise-adding module, a feature extraction module, a feature fusion module, and a command prediction module.

[0194] The instruction noise-generating module is used to generate noise for control instructions;

[0195] The feature extraction module is used to extract a first image feature set from the multi-time image set and to extract target location features from the target location data;

[0196] The feature fusion module is used to fuse the noise, the first image feature set, and the target location features to generate multimodal fusion features;

[0197] The instruction prediction module is used to generate control instructions for the mobile platform to navigate to the target location based on the multimodal fusion features.

[0198] In one embodiment, the instruction noise-adding module employs a variational autoencoder architecture, and the noise of the control instruction is generated in the following manner:

[0199] If the random latent variable mechanism is enabled, latent variable embeddings are sampled from the training set distribution and the latent variable embeddings are determined as noise for the control command;

[0200] If the random latent variable mechanism is not enabled, the mean of the training set distribution will be determined as noise in the control commands.

[0201] In one embodiment, the first image feature set includes image features corresponding to images at each time point; the feature fusion module includes a feature enhancement submodule and a multimodal fusion submodule; the feature enhancement submodule is used to perform feature enhancement processing on the regions related to the target location in each of the image features using an attention mechanism to obtain each enhanced image feature; the multimodal fusion submodule is used to perform multimodal fusion of the noise, the second image feature set and the target location features to obtain multimodal fused features, wherein the second image feature set includes each of the enhanced image features.

[0202] In one embodiment, the enhanced image features are generated in the following manner:

[0203] The image features are flattened to obtain flattened image features;

[0204] An attention feature is generated based on the target location features and the flattened image features using an attention mechanism.

[0205] The attention features are fused with each of the image features to obtain the enhanced image features.

[0206] In one embodiment, the multimodal fusion feature is generated in the following manner:

[0207] Using the Transformer architecture, the noise, the second image feature set, and the target location features are fused in a multimodal manner to obtain the multimodal fused features.

[0208] In one embodiment, the control instruction includes multi-dimensional control parameters; the control instruction satisfies comprehensive physical constraints, which are obtained by fusing the physical constraints corresponding to each control parameter; the sequence dimension of the control instruction satisfies dimensional constraints, which are determined based on a preset batch size, a preset sequence length, and a number of dimensions.

[0209] In one embodiment, the control command prediction model is trained in the following manner:

[0210] Acquire multi-time image sets and target location data samples observed by the mobile platform;

[0211] Model training is performed based on the multi-time image set samples and the target location data samples;

[0212] The total loss value of the model prediction is determined based on the predicted loss value and relative entropy between the predictive control command and the actual control command.

[0213] Based on the total loss value, the model parameters are iteratively updated until the model converges to obtain the control command prediction model.

[0214] It should be noted that the end-to-end automatic navigation system provided in this application can execute the end-to-end automatic navigation method described in any of the above embodiments during actual operation, which will not be elaborated in this embodiment.

[0215] Figure 4 This is a schematic diagram of the structure of the electronic device provided in this application, such as... Figure 4As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, communications interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute an end-to-end automatic navigation method. This method includes: acquiring a multi-time-lapse image set and target location data observed by a camera integrated into the mobile platform; inputting the multi-time-lapse image set and the target location data into a control command prediction model to obtain a control command output by the control command prediction model for navigating the mobile platform to the target location; wherein the control command is generated based on multi-modal fusion features, which are obtained by fusing noise, a first image feature set, and target location features. The first image feature set is extracted from the multi-time-lapse image set, and the target location features are extracted from the target location data.

[0216] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0217] On the other hand, this application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can execute the end-to-end automatic navigation method provided in the above embodiments. The method includes: acquiring a set of multi-time images observed by a camera integrated on a mobile platform and target location data; inputting the set of multi-time images and the target location data into a control command prediction model to obtain a control command output by the control command prediction model for navigating the mobile platform to a target location; wherein the control command is generated based on multimodal fusion features, the multimodal fusion features are obtained by fusing noise, a first image feature set, and target location features, the first image feature set is extracted based on the set of multi-time images, and the target location features are extracted based on the target location data.

[0218] In another aspect, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program is implemented to perform the end-to-end automatic navigation method provided in the above embodiments. The method includes: acquiring a set of multi-time-lapse images observed by a camera integrated on a mobile platform and target location data; inputting the set of multi-time-lapse images and the target location data into a control command prediction model to obtain a control command output by the control command prediction model for navigating the mobile platform to a target location; wherein the control command is generated based on multi-modal fusion features, the multi-modal fusion features are obtained by fusing noise, a first image feature set, and target location features, the first image feature set is extracted based on the set of multi-time-lapse images, and the target location features are extracted based on the target location data.

[0219] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0220] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0221] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An end-to-end automatic navigation method, characterized in that, The end-to-end automatic navigation method includes: Acquire multi-time image sets and target location data observed by the camera integrated into the mobile platform; The multi-time image set and the target location data are input into the control command prediction model to obtain the control command output by the control command prediction model to navigate the mobile platform to the target location; The control command is generated based on multimodal fusion features, which are obtained by fusing noise, a first image feature set, and target location features. The first image feature set is extracted from the multi-time-time image set, and the target location features are extracted from the target location data. The control command prediction model includes a command noise-adding module, a feature extraction module, a feature fusion module, and a command prediction module. The instruction noise-generating module is used to generate noise for control instructions; The feature extraction module is used to extract a first image feature set from the multi-time image set and to extract target location features from the target location data; The feature fusion module is used to fuse the noise, the first image feature set, and the target location features to generate multimodal fusion features; The instruction prediction module is used to generate control instructions for the mobile platform to navigate to the target location based on the multimodal fusion features.

2. The end-to-end automatic navigation method according to claim 1, characterized in that, The instruction noise-adding module adopts a variational autoencoder architecture, and the noise of the control instructions is generated in the following way: If the random latent variable mechanism is enabled, latent variable embeddings are sampled from the training set distribution and the latent variable embeddings are determined as noise for the control command; If the random latent variable mechanism is not enabled, the mean of the training set distribution will be determined as noise in the control commands.

3. The end-to-end automatic navigation method according to claim 1, characterized in that, The first image feature set includes image features corresponding to images at each time point; the feature fusion module includes a feature enhancement submodule and a multimodal fusion submodule; The feature enhancement submodule is used to perform feature enhancement processing on the regions related to the target location in each of the image features using an attention mechanism, so as to obtain each enhanced image feature; The multimodal fusion submodule is used to perform multimodal fusion of the noise, the second image feature set, and the target location features to obtain multimodal fused features, wherein the second image feature set includes each of the enhanced image features.

4. The end-to-end automatic navigation method according to claim 3, characterized in that, Each enhanced image feature is generated in the following way: The image features are flattened to obtain flattened image features; An attention feature is generated based on the target location features and the flattened image features using an attention mechanism. The attention features are fused with each of the image features to obtain the enhanced image features.

5. The end-to-end automatic navigation method according to claim 3, characterized in that, The multimodal fusion features are generated in the following way: Using the Transformer architecture, the noise, the second image feature set, and the target location features are fused in a multimodal manner to obtain the multimodal fused features.

6. The end-to-end automatic navigation method according to claim 1, characterized in that, The control instructions include multi-dimensional control parameters; the control instructions satisfy comprehensive physical constraints, which are obtained by fusing the physical constraints corresponding to each control parameter; the sequence dimensions of the control instructions satisfy dimensional constraints, which are determined based on a preset batch size, a preset sequence length, and a number of dimensions.

7. The end-to-end automatic navigation method according to any one of claims 1 to 6, characterized in that, The control command prediction model is trained in the following way: Acquire multi-time image sets and target location data samples observed by the mobile platform; Model training is performed based on the multi-time image set samples and the target location data samples; The total loss value of the model prediction is determined based on the predicted loss value and relative entropy between the predictive control command and the actual control command. Based on the total loss value, the model parameters are iteratively updated until the model converges to obtain the control command prediction model.

8. An end-to-end automatic navigation system, characterized in that, The end-to-end automatic navigation system includes: The acquisition module is used to acquire multi-time-lapse image sets and target location data observed by the camera integrated into the mobile platform; The control command generation module is used to input the multi-time image set and the target location data into the control command prediction model to obtain the control command output by the control command prediction model to navigate the mobile platform to the target location; The control command is generated based on multimodal fusion features, which are obtained by fusing noise, a first image feature set, and target location features. The first image feature set is extracted from the multi-time-time image set, and the target location features are extracted from the target location data. The control command prediction model includes a command noise-adding module, a feature extraction module, a feature fusion module, and a command prediction module. The instruction noise-generating module is used to generate noise for control instructions; The feature extraction module is used to extract a first image feature set from the multi-time image set and to extract target location features from the target location data; The feature fusion module is used to fuse the noise, the first image feature set, and the target location features to generate multimodal fusion features; The instruction prediction module is used to generate control instructions for the mobile platform to navigate to the target location based on the multimodal fusion features.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the end-to-end automatic navigation method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, wherein a computer program is stored on the non-transitory computer-readable storage medium, characterized in that, When the computer program is executed by a processor, it implements the steps of the end-to-end automatic navigation method as described in any one of claims 1 to 7.

11. A computer program product, the computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the end-to-end automatic navigation method as described in any one of claims 1 to 7.

Images