Navigation method and device, electronic equipment and storage medium

Abstract

The invention discloses a navigation method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a to-be-executed voice control instruction and a panoramic environment image of a current driving position; inputting the voice control instruction to be executed and the panoramic environment image of the current driving position into the target navigation model; performing global feature extraction on the panoramic environment image of the current driving position by using the model to obtain a global feature map of the current driving position, and performing local feature extraction on the global feature map of the current driving position by using the model to obtain a local feature map of the current driving position; performing feature extraction on a to-be-executed voice control instruction by using the model to obtain a semantic feature of the to-be-executed voice control instruction; and predicting the next driving position of the current driving position by using the model based on the local feature map of the current driving position and the semantic feature of the voice control instruction to be executed. According to the technical scheme, the determination precision of the next driving position can be improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of navigation technology, and in particular to a navigation method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of computer vision and natural language processing technologies, Visual Language Navigation (VLN) has gradually become a research hotspot as an emerging human-computer interaction method. VLN technology aims to enable robots or intelligent agents to navigate autonomously in complex indoor environments under the guidance of natural language instructions. This technology combines the capabilities of Computer Vision (CV) and Natural Language Understanding (NLU), requiring the system to not only understand human verbal instructions, but also be able to identify objects and scenes in the environment, and then plan a reasonable action path.

[0003] At present, visual language navigation methods mainly rely on extracting a single level of feature information from images to make navigation decisions. However, relying solely on a single level of feature data cannot fully capture the rich visual clues in the image, resulting in insufficient accuracy of the motion trajectory determined during the navigation process, making it difficult to meet the needs of high-precision navigation.

[0004] Therefore, it is urgent to propose a new method to solve the above problems. Summary of the invention

[0005] The present invention provides a navigation method, device, electronic equipment and storage medium, which can improve the accuracy of determining the next driving position, thereby improving the accuracy and reliability of navigation.

[0006] In a first aspect, an embodiment of the present invention provides a navigation method, including:

[0007] Obtaining the voice control command to be executed and obtaining the panoramic environment image of the current driving position;

[0008] Inputting the voice control command to be executed and the panoramic environment image of the current driving position into a target navigation model, wherein the target navigation model includes a first voice processing module, a first image processing module and a first trajectory prediction module;

[0009] Using the first image processing module to perform global feature extraction on the panoramic environment image of the current driving position to obtain a global feature map of the current driving position, and using the first image processing module to perform local feature extraction on the global feature map of the current driving position to obtain a local feature map of the current driving position;

[0010] Using the first voice processing module to perform feature extraction on the voice control instruction to be executed, so as to obtain semantic features of the voice control instruction to be executed;

[0011] The first trajectory prediction module is used to predict the next driving position of the current driving position based on the local feature map of the current driving position and the semantic features of the voice control instruction to be executed.

[0012] The technical solution of the present invention obtains the voice control instruction to be executed and obtains the panoramic environment image of the current driving position; inputs the voice control instruction to be executed and the panoramic environment image of the current driving position into the target navigation model, which includes a first voice processing module, a first image processing module and a first trajectory prediction module; uses the first image processing module to perform global feature extraction on the panoramic environment image of the current driving position to obtain the global feature map of the current driving position, and uses the first image processing module to perform local feature extraction on the global feature map of the current driving position to obtain the local feature map of the current driving position; uses the first voice processing module to perform feature extraction on the voice control instruction to be executed to obtain the semantic features of the voice control instruction to be executed; and uses the first trajectory prediction module to predict the next driving position of the current driving position based on the local feature map of the current driving position and the semantic features of the voice control instruction to be executed. The above technical solution first obtains the voice control instruction to be executed and the panoramic environment image of the current driving position, which provides a data basis for determining the local feature map of the current driving position and the semantic features of the voice control instruction to be executed. Then, the voice control command to be executed and the panoramic environment image of the current driving position are input into the target navigation model, which provides a data basis for determining the next driving position of the current driving position. Then, the panoramic environment image of the current driving position is extracted by the first image processing module to obtain the global feature map of the current driving position, and the local feature map of the current driving position is extracted by the first image processing module to obtain the local feature map of the current driving position, which can fully capture the rich visual clues in the image, thereby improving the accuracy of the predicted next driving position of the current driving position. Then, the first voice processing module is used to extract the features of the voice control command to be executed to obtain the semantic features of the voice control command to be executed, which provides a data basis for predicting the next driving position of the current driving position. Finally, the first trajectory prediction module is used to predict the next driving position of the current driving position based on the local feature map of the current driving position and the semantic features of the voice control command to be executed, which can make the accuracy of the determined next driving position higher, thereby improving the accuracy and reliability of navigation. Compared with the prior art that mainly relies on extracting feature information of a single level from an image to make navigation decisions, the present invention extracts global features from the panoramic environment image of the current driving position to obtain a global feature map of the current driving position, and then extracts local features from the global feature map of the current driving position to obtain a local feature map of the current driving position. This can fully capture the rich visual clues in the image, thereby improving the accuracy of determining the next driving position, and further improving the accuracy and reliability of navigation. This solves the problem that the prior art that only relies on feature data of a single level cannot fully capture the rich visual clues in the image, resulting in insufficient accuracy of the motion trajectory determined during the navigation process, making it difficult to meet the needs of high-precision navigation.

[0013] In a second aspect, an embodiment of the present invention further provides a navigation device, the device comprising:

[0014] An acquisition module, used to acquire the voice control command to be executed and acquire a panoramic environment image of the current driving position;

[0015] An input module, used for inputting the voice control command to be executed and the panoramic environment image of the current driving position into a target navigation model, wherein the target navigation model includes a first voice processing module, a first image processing module and a first trajectory prediction module;

[0016] a first determining module, configured to extract global features from the panoramic environment image of the current driving position using the first image processing module to obtain a global feature map of the current driving position, and to extract local features from the global feature map of the current driving position using the first image processing module to obtain a local feature map of the current driving position;

[0017] a second determining module, configured to extract features of the voice control instruction to be executed by using the first voice processing module to obtain semantic features of the voice control instruction to be executed;

[0018] The third determination module is used to use the first trajectory prediction module to predict the next driving position of the current driving position based on the local feature map of the current driving position and the semantic features of the voice control instruction to be executed.

[0019] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0020] at least one processor; and a memory communicatively coupled to the at least one processor;

[0021] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can implement any navigation method described in the first aspect.

[0022] In a fourth aspect, an embodiment of the present invention further provides a storage medium including computer executable instructions,

[0023] The computer executable instructions implement any navigation method described in the first aspect when executed by a computer processor.

[0024] It should be noted that the above computer instructions may be stored in whole or in part on a computer-readable storage medium, wherein the computer-readable storage medium may be packaged together with the processor of the navigation device, or may be packaged separately from the processor of the navigation device, which is not limited in this application.

[0025] The description of the second, third and fourth aspects in this application can refer to the detailed description of the first aspect; and the beneficial effects of the description of the second, third and fourth aspects can refer to the beneficial effect analysis of the first aspect, which will not be repeated here.

[0026] In this application, the name of the above navigation device does not limit the device or functional module itself. In actual implementation, these devices or functional modules may appear with other names. As long as the functions of each device or functional module are similar to those of this application, they fall within the scope of the claims of this application and their equivalent technologies.

[0027] These and other aspects of the present application will become more apparent from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 A flowchart of a navigation method provided by an embodiment of the present invention;

[0030] Figure 2a A flowchart of another navigation method provided by an embodiment of the present invention;

[0031] Figure 2b An example diagram of obtaining a local feature map of a current driving position provided by an embodiment of the present invention;

[0032] Figure 2c An example diagram of determining the next driving position of a current driving position provided by an embodiment of the present invention;

[0033] Figure 3 A schematic diagram of the structure of a navigation device provided by an embodiment of the present invention;

[0034] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0036] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0037] The terms "first" and "second" and the like in the specification and drawings of this application are used to distinguish different objects, or to distinguish different processing of the same object, rather than to describe a specific order of objects.

[0038] In addition, the terms "including" and "having" and any variations thereof mentioned in the description of the present application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0039] It should be mentioned before discussing exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe various operations (or steps) as sequential processes, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of various operations can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to methods, functions, procedures, subroutines, subprograms, etc. In addition, the embodiments in the present invention and the features in the embodiments can be combined with each other without conflict.

[0040] It should be noted that, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0041] In the description of the present application, unless otherwise specified, “plurality” means two or more.

[0042] Figure 1 This is a flowchart of a navigation method provided by an embodiment of the present invention. This embodiment is applicable to situations where a mobile device needs to navigate automatically. For example, the mobile device may be a robot or a vehicle. The method may be executed by a navigation device, which may be implemented in software and / or hardware. In a specific embodiment, the navigation device may be integrated in an electronic device, which may be a server, etc. Figure 1 The navigation method of this embodiment specifically includes the following steps:

[0043] Step 110: Obtain the voice control command to be executed, and obtain the panoramic environment image of the current driving position.

[0044] Specifically, voice control instructions refer to commands issued by human natural language to control or instruct mobile devices to perform specific operations, for example, voice control instructions can be "go forward" or "turn right at the first intersection". For example, a mobile device can be a vehicle, a drone, or a robot. The current driving position refers to the geographical location of the mobile device at the current moment. A panoramic environment image refers to an image that can display a 360-degree panoramic view around the current driving position.

[0045] In a specific implementation, the user's voice can be acquired through a voice acquisition device (such as a microphone) to obtain the voice control instructions to be executed, and then the acquired voice control instructions to be executed are preprocessed to remove noise and redundant data, improve data quality, and facilitate subsequent processing. In addition, multiple cameras installed around the mobile device can be used to capture images from various angles, and then each image can be distorted to prevent image distortion. These images are then stitched together using image stitching software to form a complete 360-degree panoramic image, thereby obtaining a panoramic environment image of the current driving position.

[0046] In this embodiment, by acquiring the panoramic environment image of the voice control command to be executed and the current driving position, a data basis is provided for subsequently determining the local feature map of the current driving position and the semantic features of the voice control command to be executed.

[0047] Step 120: input the voice control command to be executed and the panoramic environment image of the current driving position into the target navigation model.

[0048] Specifically, the target navigation model is a model obtained after training based on historical voice control instructions, panoramic environment images of each historical driving position and actual driving trajectory. The target navigation model includes a first voice processing module, a first image processing module and a first trajectory prediction module.

[0049] In the specific implementation, after obtaining the voice control command to be executed and the panoramic environment image of the current driving position, the voice control command to be executed and the panoramic environment image of the current driving position can be input into the target navigation model. Since the target navigation model is a model obtained after training based on historical voice control commands, panoramic environment images of various historical driving positions and actual driving trajectories, the target navigation model can determine the next driving position of the current driving position based on the voice control command to be executed and the panoramic environment image of the current driving position.

[0050] In this embodiment, the voice control command to be executed and the panoramic environment image of the current driving position are input into the target navigation model, which provides a data basis for determining the next driving position of the current driving position.

[0051] Step 130: Use the first image processing module to perform global feature extraction on the panoramic environment image of the current driving position to obtain a global feature map of the current driving position, and use the first image processing module to perform local feature extraction on the global feature map of the current driving position to obtain a local feature map of the current driving position.

[0052] Specifically, the first image processing module refers to a module that performs global and local feature extraction on the panoramic environment image. The global feature map refers to a feature map obtained after global feature extraction on the panoramic environment image. Global feature extraction refers to extracting features that can represent the characteristics of the entire image (such as color distribution, texture, shape, etc.) from the entire image. The local feature map refers to an image obtained after local feature extraction on the global feature map. Local feature extraction refers to extracting features with specific attributes (such as key points, corners, edges and other detail features) from the local area of ​​the image.

[0053] In a specific implementation, a suitable global feature extraction neural network can be selected according to actual conditions or requirements, for example, the global feature extraction neural network can be VGG16, ResNet101 or ResNet152, etc. Then, the selected global feature extraction neural network is applied to perform global feature extraction on the panoramic environment image of the current driving position to obtain a global feature map of the current driving position. For example, if the selected global feature extraction neural network is VGG16, the panoramic environment image of the current driving position can be first input into the convolution layer, and multiple convolution kernels (the convolution kernel size is 3×3) are used to perform convolution processing on the panoramic environment image of the current driving position to extract edge, texture and other features in the image, thereby obtaining a primary global feature map, and then the primary global feature map is input into the maximum pooling layer for pooling processing, that is, the maximum value is selected in each pooling window (the pooling window size is 2×2, and the step size is 2) to obtain an intermediate global feature map, and finally the intermediate global feature map is input into the fully connected layer, and all elements in the intermediate global feature map are arranged in sequence into a long vector to obtain the image vector corresponding to the global feature map of the current driving position, thereby obtaining the global feature map of the current driving position.

[0054] Then, according to the actual situation or needs, a suitable local feature extraction neural network is selected, for example, the local feature extraction neural network can be Fast RCNN or Hard Net, etc. Then, the selected local feature extraction neural network is applied to extract local features from the global feature map of the current driving position to obtain the local feature map of the current driving position. For example, if the selected local feature extraction neural network is FastRCNN, the region proposal network (RPN) can be used to determine each candidate region (the image sub-region containing the object) from the global feature map of the current driving position, and then each candidate region is input into the ROIPooling layer, and each candidate region is spatially sampled to obtain a primary local feature map, and then the primary local feature map is input into the fully connected layer, and feature compression and classification are performed to obtain an intermediate local feature map, and finally the intermediate local feature map is input into two parallel output layers (one for identifying the object category in the intermediate local feature map, and the other for adjusting the bounding box of the intermediate local feature map to more accurately locate the object) to obtain the local feature map of the current driving position.

[0055] In this embodiment, global feature extraction is first performed on the panoramic environment image of the current driving position to obtain a global feature map of the current driving position, and then local feature extraction is performed on the global feature map of the current driving position to obtain a local feature map of the current driving position. This can fully capture the rich visual clues in the image, thereby improving the accuracy of predicting the next driving position of the current driving position.

[0056] Step 140: Use the first voice processing module to perform feature extraction on the voice control instruction to be executed to obtain semantic features of the voice control instruction to be executed.

[0057] Specifically, the first voice processing module refers to a module that extracts features from the voice control instruction to be executed. The semantic feature refers to a feature extracted from the voice control instruction that can express language meaning and context information.

[0058] In a specific implementation, a suitable text conversion method can be selected according to the actual situation or requirements, for example, the text conversion method can be a hidden Markov model or a Transformer model, and then the selected text conversion method is applied to process the preprocessed voice control instruction to be executed to obtain a text string, and then a suitable semantic information extraction model can be selected according to the actual situation or requirements, for example, the semantic information extraction model can be a BERT model. Then the selected semantic information extraction model is applied to analyze the text string to obtain the semantic features of the voice control instruction to be executed. For example, if the selected speech feature vector analysis model is a BERT model, the text string is first converted to the input format expected by the model, for example, special start and end tags (such as [CLS] and [SEP]) are added to the text string, and then the processed text string is input into the model, and then the BERT model's word segmenter is used to segment the text, that is, the text is divided into words or subwords, and then a vector representation is generated for each word, which captures the contextual semantic information of the word, and then the vector corresponding to the [CLS] tag is taken as the representation of the entire sentence, thereby obtaining the semantic features of the voice control instruction to be executed.

[0059] In this embodiment, by obtaining the semantic features of the voice control instruction to be executed, a data basis is provided for predicting the next driving position of the current driving position.

[0060] Step 150: Use the first trajectory prediction module to predict the next driving position of the current driving position based on the local feature map of the current driving position and the semantic features of the voice control instruction to be executed.

[0061] Specifically, the first trajectory prediction module refers to a module that predicts the next driving position of the current driving position by using the local feature map of the current driving position and the semantic features of the voice control command to be executed. The next driving position refers to the location where the mobile device is expected to arrive or should go after executing the voice control command.

[0062] In a specific implementation, after obtaining the local feature map of the current driving position and the semantic features of the voice control command to be executed, the local feature map of the current driving position and the semantic features of the voice control command to be executed can be feature fused based on concatenation, weighted summation, attention mechanism, etc. to obtain a context vector, and then a suitable prediction neural network can be selected according to the actual situation or needs, for example, the prediction network can be LSTM. Then the selected prediction neural network is applied to process the context vector to obtain the next driving position of the current driving position. For example, if the prediction network is LSTM, the context vector is input into the input gate, forget gate and output gate respectively. The context vector and the hidden vector of the previous moment are processed by the input gate to obtain the input gate vector, and the context vector and the hidden vector of the previous moment are fully connected by the activation function to obtain the candidate vector. The context vector and the hidden vector of the previous moment are processed by the forget gate to obtain the forget gate vector. Then the product of the candidate vector and the input gate vector, the product of the forget gate vector and the cell vector of the previous moment are calculated, and the sum of the two products is calculated to obtain the cell vector of the current moment. Finally, the output gate is used to process the cell vector at the current moment to obtain the next driving position of the current driving position. Among them, the cell vector and the candidate vector are initialized to zero vector.

[0063] In this embodiment, the first trajectory prediction module is used to predict the next driving position of the current driving position based on the local feature map of the current driving position and the semantic features of the voice control command to be executed, so that the accuracy of the determined next driving position can be higher, thereby improving the accuracy and reliability of navigation.

[0064] In the embodiment of the present invention, the panoramic environment image of the voice control instruction to be executed and the current driving position is first obtained, which provides a data basis for determining the local feature map of the current driving position and the semantic features of the voice control instruction to be executed. Then, the voice control instruction to be executed and the panoramic environment image of the current driving position are input into the target navigation model, which provides a data basis for determining the next driving position of the current driving position. Then, the panoramic environment image of the current driving position is extracted with the first image processing module to obtain the global feature map of the current driving position, and the local feature map of the current driving position is extracted with the first image processing module to obtain the local feature map of the current driving position, which can fully capture the rich visual clues in the image, thereby improving the accuracy of the predicted next driving position of the current driving position. Then, the first voice processing module is used to extract the features of the voice control instruction to be executed to obtain the semantic features of the voice control instruction to be executed, which provides a data basis for predicting the next driving position of the current driving position. Finally, the first trajectory prediction module is used to predict the next driving position of the current driving position based on the local feature map of the current driving position and the semantic features of the voice control command to be executed, so that the accuracy of the next driving position can be higher, thereby improving the accuracy and reliability of navigation. Compared with the prior art that mainly relies on extracting a single layer of feature information from the image to make navigation decisions, the present invention extracts global features from the panoramic environment image of the current driving position to obtain a global feature map of the current driving position, and then extracts local features from the global feature map of the current driving position to obtain a local feature map of the current driving position. It can fully capture the rich visual clues in the image, thereby improving the accuracy of determining the next driving position, and further improving the accuracy and reliability of navigation. It solves the problem that the prior art only relies on a single layer of feature data and cannot fully capture the rich visual clues in the image, resulting in insufficient accuracy of the motion trajectory determined during the navigation process, and it is difficult to meet the needs of high-precision navigation.

[0065] Figure 2a This is a flowchart of another navigation method provided by an embodiment of the present invention. This embodiment is specific based on the above embodiment. In this embodiment, the method may also include:

[0066] Step 210: Acquire the voice control command to be executed, and acquire the panoramic environment image of the current driving position.

[0067] Furthermore, a panoramic environment image of the current driving position is obtained, including: dividing the panoramic view into sections to obtain a plurality of section views; photographing the current environment at different angles in each section view to obtain a panoramic view at different angles in each section view.

[0068] Specifically, a panoramic perspective refers to a perspective that can cover the field of view of the surrounding 360-degree environment. A partitioned perspective refers to dividing a panoramic perspective into several smaller, more manageable and processable regional perspectives. The current scene refers to the physical space around the current location of the mobile device and the objects, landscapes, etc. therein. A panoramic view refers to an image or a collection of images that can display the surrounding 360-degree environment. The panoramic environment image includes panoramic views at different angles of different partitioned perspectives.

[0069] In a specific implementation, multiple image acquisition devices (such as cameras) can be installed on a mobile device first, and it is ensured that these image acquisition devices can cover a 360-degree viewing angle. Then, the 360-degree viewing angle is divided into multiple partitions according to actual conditions or needs, and each partition is responsible for one or more image acquisition devices. For example, the 360-degree panoramic viewing angle can be divided into 12 viewing angles at intervals of 30 degrees, that is, divided into 12 partition viewing angles. Then, the current environment is photographed at different angles in each partition viewing angle to obtain a panoramic view at different angles of each partition viewing angle, that is, a panoramic environment image. For example, the current environment at elevation angles of -30 degrees, 0 degrees, and 30 degrees can be photographed in each partition viewing angle to obtain a panoramic environment image.

[0070] In this embodiment, by dividing the panoramic view into partitions and shooting at different angles in each partition view, full coverage of the environmental image can be ensured, while more detailed information can be captured, which helps to reduce blind spots and improve the integrity and accuracy of the image.

[0071] Step 211: input the voice control command to be executed and the panoramic environment image of the current driving position into the target navigation model.

[0072] Furthermore, the training process of the target navigation model includes: obtaining historical voice control instructions and obtaining a panoramic environment image of each historical driving position; inputting the historical voice control instructions and the panoramic environment image of each historical driving position into a preset navigation model, the preset navigation model including a second voice processing module, a second image processing module and a second trajectory prediction module; using the second image processing module to perform global feature extraction on the panoramic environment image of each historical position, and correspondingly obtain a global feature map of each historical driving position, and using the second image processing module to perform local feature extraction on the global feature map of each historical driving position, and obtain a local feature map of each historical driving position; using the second voice processing module to perform feature extraction on the historical voice control instructions, and obtain semantic features of the historical voice control instructions; using the second trajectory prediction module to predict the next driving position of each historical driving position based on the local feature map of each historical driving position and the semantic features of the historical voice control instructions, and obtain a predicted driving trajectory; determining the function value of the loss function according to the predicted driving trajectory and the actual driving trajectory; optimizing the model parameters of the preset navigation model based on the function value to obtain the target navigation model.

[0073] Specifically, the historical voice control instructions refer to the control commands issued by the user to the mobile device in the past by voice. The panoramic environment image of the historical driving position refers to the 360-degree image of the surrounding environment of the mobile device at different time points. The preset navigation model refers to a predefined machine learning model architecture for learning how to predict the driving trajectory of the mobile device based on the environmental image and voice instructions, which is composed of a second voice processing module, a second image processing module and a second trajectory prediction module. The second image processing module is a module in the preset navigation model, which is responsible for global and local feature extraction of the panoramic environment image. The second voice processing module is a module in the preset navigation model, which is responsible for converting the historical voice control instructions into a computer-understandable format and extracting key information (such as destination, path requirements, etc.) as semantic features. The second trajectory prediction module is a module in the preset navigation model, which is responsible for predicting the next driving position of the mobile device based on the local feature map of the input panoramic environment image and the semantic features of the voice control instructions to obtain a predicted driving trajectory. The actual driving trajectory refers to the route actually driven by the mobile device, which is used to evaluate the accuracy of the predicted driving trajectory. The loss function can be a cross entropy loss function, a mean absolute error loss function or an improved cross entropy loss function, which is not limited in the embodiments of the present invention.

[0074] In the specific implementation, the historical voice control instructions and the panoramic environment images of each historical driving position are first obtained. Then these data are cleaned and preprocessed to ensure that they meet the input requirements of the preset navigation model. After that, the historical voice control instructions and the panoramic environment images of each historical driving position are input into the preset navigation model to train the preset navigation model. Specifically, you can first select a suitable global feature extraction neural network (such as VGG16, ResNet101 or ResNet152, etc.) according to the actual situation or needs. Then, the selected global feature extraction neural network is used to perform global feature extraction on the panoramic environment images of each historical driving position to obtain the global feature map of each historical driving position. Then, according to the actual situation or needs, a suitable local feature extraction neural network (such as Fast RCNN or Hard Net, etc.) is selected. Then, the selected local feature extraction neural network is used to perform local feature extraction on the global feature map of each historical driving position to obtain the local feature map of each historical driving position. Then, according to the actual situation or needs, a suitable text conversion method (such as a hidden Markov model or a Transformer model) can be selected, and then the selected text conversion method can be used to process the historical voice control instructions to obtain a text string. Then, according to the actual situation or needs, a suitable semantic information extraction model (such as a BERT model) can be selected. Then, the selected semantic information extraction model is used to analyze the text string to obtain the semantic features of the historical voice control instructions. After that, the semantic features of the historical voice control instructions and the local feature maps of each historical driving position can be fused based on concatenation, weighted summation, attention mechanism, etc. to obtain a context vector, and then a suitable prediction neural network (such as LSTM) can be selected according to the actual situation or needs. Then, the selected prediction neural network is used to process the context vector to obtain the next driving position of each historical driving position, and then the predicted driving trajectory is obtained. Then, the actual driving trajectory is obtained, and the distance between the predicted driving trajectory and the actual driving trajectory is calculated to obtain the function value of the loss function; then, it is determined whether the function value of the loss function is less than the preset function threshold. If it is less than, the current model is determined to be the target navigation model. If it is not less than, the parameters such as the weight and bias inside the model are adjusted, and the current model is continuously trained until the function value of the loss function is less than the preset function threshold. The preset navigation model can be CornerNet, FCOS, RetinaNet, YolV7, etc., which is not limited in the embodiment of the present invention.

[0075] Exemplarily, the relevant configuration of model training can be: the Place365 dataset is selected as the training set, the Dropout rate is 0.7, the mask size of the attention mechanism is 3×3, the learning rate update strategy is cosine annealing and learning rate warm-up, the BatchSize is 8, and the number of iterations is 100,000.

[0076] In this embodiment, the target navigation model is trained through historical voice control instructions, panoramic environment images of each historical driving position and actual driving trajectory, so that the target navigation model can be used to determine the next driving position of the current driving position based on obtaining the voice control instructions to be executed and the panoramic environment image of the current driving position.

[0077] Step 212: Use a convolution unit to perform convolution processing on the panoramic environment image of the current driving position to obtain a primary global feature map.

[0078] Specifically, the first image processing module includes a global image processing module. The global image processing module is used to extract global features of the panoramic environment image of the current driving position. The global image processing module includes a convolution unit, a residual unit and a global average pooling unit. The convolution unit is composed of a convolution layer. The residual unit is composed of a residual block. The global average pooling unit is composed of a global average pooling layer.

[0079] In a specific implementation, after obtaining the panoramic environment image of the current driving position, the panoramic environment image can be preprocessed based on resizing, normalization, etc. to ensure that the input data meets the input requirements of the convolution unit. The preprocessed data is then input into the convolution unit, after which each convolution kernel slides on the input image, performing a convolution operation with a local area on the image each time it slides, and obtaining a value on the output feature map. Repeat the above sliding and convolution operations until the entire input panoramic environment image is traversed, and multiple output feature maps are obtained, which together constitute a multi-channel feature map, i.e., a primary global feature map.

[0080] In this embodiment, by obtaining a primary global feature map, a data basis is provided for obtaining an intermediate global feature map later.

[0081] Step 213: Use the residual unit to perform nonlinear transformation and down-sampling on the primary global feature map to obtain an intermediate global feature map.

[0082] In the specific implementation, after obtaining the primary global feature map, in order to keep the size of the primary global feature map unchanged, the primary global feature map is processed by a convolution layer containing a convolution kernel with a step size of 1, and then the processed primary global feature map is nonlinearly transformed using a nonlinear activation function (such as ReLU). Then, a convolution layer with a step size greater than 1 is used to downsample the feature map after the nonlinear transformation. Then, the primary global feature map (if the size is different from the size of the downsampled feature map, a 1×1 convolution layer needs to be used to adjust its size to the same size as the downsampled feature map) is added to the downsampled feature map to obtain an intermediate global feature map.

[0083] In this embodiment, the residual unit is used to perform nonlinear transformation processing and down-sampling on the primary global feature map, which not only obtains the intermediate global feature map, but also alleviates the gradient vanishing problem and improves the feature expression capability.

[0084] Step 214: Use a global average pooling unit to perform global average pooling processing on the intermediate global feature map to obtain a global feature map of the current driving position.

[0085] In the specific implementation, after obtaining the intermediate global feature map, a global average pooling operation is performed for each channel of the intermediate global feature map (for each channel, the average value of all pixel values ​​is calculated), thereby obtaining the global feature vector of the current driving position. For example: the size of the intermediate global feature map is H×W×C (H is the height, W is the width, and C is the number of channels). If a global average pooling operation is performed on it, the average value of the H×W pixels on the C channels will be calculated respectively. After global average pooling, the H×W pixels on each channel are compressed into a numerical value. Therefore, the entire intermediate global feature map is converted into a 1×1×C feature map. This 1×1×C feature map is the global feature vector of the current driving position, in which each dimension represents the global average value of a specific feature channel.

[0086] In this embodiment, the global average pooling unit is used to perform global average pooling processing on the intermediate global feature map, which can not only obtain the global feature map of the current driving position, but also reduce the model parameters and calculation amount, and improve the generalization ability of the model.

[0087] Step 215: Determine the azimuth angle and elevation angle of each panoramic view.

[0088] Specifically, the azimuth angle refers to the angle between a fixed point (such as the position of the observer, the position of the mobile device, or the geographic North Pole) and the target direction line measured along the horizontal plane, usually ranging from 0 degrees to 360 degrees. The elevation angle refers to the angle between the line of sight of the image acquisition device and its projection on the horizontal plane, ranging from -90 degrees (downward) to 90 degrees (upward).

[0089] In a specific implementation, a mapping relationship between image pixel coordinates and real-world angle coordinates can be established based on the camera's projection model (such as perspective projection, spherical projection, or cylindrical projection). For each pixel in each panoramic view, its position in the image coordinate system is determined. Then, the image coordinates are converted into azimuth angles and elevation angles using the camera calibration parameters (such as focal length, optical center position, distortion coefficient, etc.) and the projection model, thereby obtaining the azimuth angles and elevation angles of each panoramic view.

[0090] In this embodiment, by determining the azimuth angle and elevation angle of each panoramic view, a data basis is provided for obtaining the position coding of the panoramic environment image of the current driving position later.

[0091] Step 216: Encode the azimuth angle and elevation angle of each panoramic view to obtain a position code of the panoramic environment image of the current driving position.

[0092] Specifically, position encoding refers to converting the position information of each panoramic view into a coding form that can be understood and processed by the target navigation model, and is used to assign a unique identifier that reflects its position information to a specific area in the panoramic view (such as a pixel block, feature point, etc.).

[0093] In the specific implementation, you can first use the formula radians = angle × π / 180 to convert the azimuth angle and elevation angle into radians. Then for each position in the panoramic view, use its azimuth and elevation to calculate the corresponding three-dimensional vector. Specifically, the calculation formula for the three-dimensional vector (x, y, z) can be: x = cos (azimuth) × cos (elevation), y = sin (azimuth) × cos (elevation), z = sin (elevation). After that, you can use quaternions to store the calculated three-dimensional vector as a position code.

[0094] In this embodiment, by obtaining the position encoding of the panoramic environment image of the current driving position, a data basis is provided for obtaining the key vector and value vector of the slot attention mechanism later.

[0095] Step 217: Use the first vector determination unit to process the position encoding of the global feature map of the current driving position and the panoramic environment image of the current driving position to obtain the key vector and value vector of the slot attention mechanism.

[0096] Specifically, the first image processing module includes a local image processing module. The local image processing module is mainly used to extract local features from the global feature map of the current driving position. The local image processing module includes a first vector determination unit, a first weight determination unit, a first weighted vector determination unit, a gated loop unit and a multilayer perceptron. The key vector of the slot attention mechanism is obtained by splicing the global image vector and the position vector, and the value vector of the slot attention mechanism is the global image vector, the global image vector is the image vector corresponding to the global feature map of the current driving position, and the position vector is the position vector corresponding to the position encoding of the panoramic environment image of the current driving position. The first vector determination unit is a unit responsible for processing the global feature map of the current driving position and the position encoding of the panoramic environment image of the current driving position to generate the key vector and value vector in the slot attention mechanism.

[0097] In the specific implementation, the key vector of the slot attention mechanism = the image vector corresponding to the global feature map of the current driving position + the position vector corresponding to the position encoding of the panoramic environment image of the current driving position, and the value vector of the slot attention mechanism = the image vector corresponding to the global feature map of the current driving position.

[0098] For example, if the image vector corresponding to the global feature map of the current driving position is A, and the position vector corresponding to the position encoding of the panoramic environment image of the current driving position is B, then the key vector of the slot attention mechanism = A + B, and the value vector of the slot attention mechanism = A.

[0099] In this embodiment, the first vector determination unit is used to process the position encoding of the global feature map of the current driving position and the panoramic environment image of the current driving position to obtain the key vector and value vector of the slot attention mechanism, which can effectively fuse the image information and position information, help generate a more accurate and rich feature representation, thereby improving the accuracy and efficiency of subsequent calculations.

[0100] Step 218: Use the first weight determination unit to normalize the query vector of the slot attention mechanism, the key vector of the slot attention mechanism, and the value vector of the slot attention mechanism to obtain the weight of the slot attention mechanism.

[0101] Specifically, the query vector of the slot attention mechanism is a preset slot vector. The first weight determination unit is responsible for processing the query vector of the slot attention mechanism, the key vector of the slot attention mechanism and the value vector of the slot attention mechanism to obtain the weight of the slot attention mechanism.

[0102] In the specific implementation, after obtaining the query vector of the slot attention mechanism, the key vector of the slot attention mechanism, and the value vector of the slot attention mechanism, the query vector of the slot attention mechanism, the key vector of the slot attention mechanism, and the value vector of the slot attention mechanism can be normalized to obtain the weight of the slot attention mechanism. The specific calculation formula is as follows:

[0103]

[0104] Among them, Mask is the mask (the mask size can be 3×3), q is the query vector of the slot attention mechanism, k is the key vector of the slot attention mechanism, attn is the weight of the slot attention mechanism, and Softmax is the activation function. T is the transpose operation of the matrix, D v is the dimension of the key vector of the slot attention mechanism. For example, if the dimension of the image vector corresponding to the global feature map of the current driving position is 2048, and the dimension of the position vector corresponding to the position encoding of the panoramic environment image of the current driving position is 128, then the dimension of the key vector of the slot attention mechanism can be determined to be 2176.

[0105] In this embodiment, the first weight determination unit is used to normalize the query vector of the slot attention mechanism, the key vector of the slot attention mechanism, and the value vector of the slot attention mechanism to obtain the weight of the slot attention mechanism, thereby improving the accuracy of weight determination.

[0106] Step 219: Use the first weighted vector determination unit to calculate the product of the weight of the slot attention mechanism and the value vector of the slot attention mechanism to obtain a weighted value vector.

[0107] Specifically, the first weighted vector determination unit is responsible for processing the weight of the slot attention mechanism and the value vector of the slot attention mechanism to obtain a weighted value vector. The weighted value vector refers to a vector obtained by multiplying each value vector by its corresponding attention weight.

[0108] In the specific implementation, the weighted value vector = the weight of the slot attention mechanism × the product of the value vector of the slot attention mechanism.

[0109] In this embodiment, by calculating the product of the weight of the slot attention mechanism and the value vector of the slot attention mechanism, a weighted value vector is obtained, which can more prominently express the key information related to the current task, and help the model focus more on these important information in subsequent processing, thereby improving the overall performance.

[0110] Step 220: Use a gated recurrent unit to update the weighted value vector to obtain an intermediate slot vector.

[0111] Specifically, a gated recurrent unit (GRU) is a variant of a recurrent neural network. In this embodiment, the weighted value vector is updated by the GRU to obtain an intermediate slot vector. The intermediate slot vector refers to a vector obtained after the weighted value vector is updated.

[0112] In a specific implementation, the weighted value vector is used as the input of the gated recurrent unit, and then the weighted value vector is updated through the gating mechanism (reset gate and update gate) of the gated recurrent unit to obtain the intermediate slot vector.

[0113] In this embodiment, the weighted value vector is updated by using a gated cyclic unit, so that the information in the weighted value vector can be further integrated and refined, thereby generating a more refined and accurate intermediate slot vector.

[0114] Step 221: Use a multilayer perceptron to perform nonlinear processing on the intermediate slot vector to obtain a suboptimal slot vector.

[0115] Specifically, the suboptimal slot vector refers to a vector obtained by performing nonlinear processing on the intermediate slot vector. A multi-layer perceptron (MLP) is a forward-structured artificial neural network that maps a set of input vectors to a set of output vectors, including an input layer, a hidden layer, and an output layer. In this embodiment, the suboptimal slot vector can be obtained by performing nonlinear processing on the intermediate slot vector through MLP.

[0116] In the specific implementation, the intermediate slot vector is used as the input of the MLP, and then the intermediate slot vector is processed by nonlinear transformation through the neurons of the hidden layer of the MLP. After that, the output layer of the MLP adopts a linear activation function to convert the feature vector output by the hidden layer into a suboptimal slot vector.

[0117] In this embodiment, by obtaining the suboptimal slot vector, a data basis is provided for subsequently obtaining a local feature map of the current driving position.

[0118] Step 222: Determine whether the execution times have reached the preset times.

[0119] Specifically, the number of executions refers to the number of times the first weight determination unit executes the step of obtaining the weight of the slot attention mechanism. The preset number of times refers to the number of times the first weight determination unit is expected to execute the step of obtaining the weight of the slot attention mechanism according to actual conditions or requirements.

[0120] In a specific implementation, a timer may be set to record the number of times the first weight determination unit obtains the weight of the slot attention mechanism. After each execution of the step of obtaining the weight of the slot attention mechanism by the first weight determination unit, the timer is incremented by one, and the initial value of the timer is zero. The value of the timer is then compared with the preset number of times. If the number of executions reaches the preset number of times, the feature map corresponding to the suboptimal slot vector is determined as the local feature map of the current driving position. If the number of executions does not reach the preset number of times, the suboptimal slot vector is determined as the query vector of the slot attention mechanism, and the process returns to step 218.

[0121] In this embodiment, by judging whether the number of executions reaches a preset number, it helps to determine whether to determine the feature map corresponding to the suboptimal slot vector as the local feature map of the current driving position.

[0122] Step 223: Determine the suboptimal slot vector as the query vector of the slot attention mechanism.

[0123] In a specific implementation, if the number of executions does not reach the preset number, the suboptimal slot vector is determined as the query vector of the slot attention mechanism, and the process returns to step 218.

[0124] Step 224: determine the feature map corresponding to the suboptimal slot vector as the local feature map of the current driving position.

[0125] In a specific implementation, if the number of executions reaches a preset number, the feature map corresponding to the suboptimal slot vector is determined as the local feature map of the current driving position.

[0126] For example, the process of obtaining the local feature map of the current driving position through the global feature map of the current driving position can be as follows: Figure 2b As shown in the figure, W q is the query vector of the slot attention mechanism, W k is the key vector of the slot attention mechanism, W v is the value vector of the slot attention mechanism. First, the query vector, key vector and value vector of the slot attention mechanism are normalized by Multi-Layer Softmax-Attention, i.e., multi-layer softmax attention mechanism, to obtain a weighted value vector. Then, the weighted value vector is updated by GRU to obtain an intermediate slot vector. The intermediate slot vector is then processed nonlinearly by multi-layer perceptron to obtain a suboptimal slot vector. Then, the suboptimal slot vector can be fed back to the slot attention mechanism through a jump connection, that is, the suboptimal slot vector is determined as the query vector of the slot attention mechanism, and the step of obtaining the weighted value vector is re-executed until the preset stop condition is met, and then the feature map corresponding to the suboptimal slot vector is determined as the local feature map of the current driving position.

[0127] Step 225: Use the first voice processing module to perform feature extraction on the voice control instruction to be executed to obtain semantic features of the voice control instruction to be executed.

[0128] Specifically, the semantic features of the voice control instruction to be executed include initial semantic features and intermediate semantic features. The initial semantic features refer to the semantic features to which the corresponding voice control instruction to be executed is added with special start and end marks when converted into a text string. The intermediate semantic features refer to the semantic features to which the corresponding voice control instruction to be executed is not added with special start and end marks when converted into a text string.

[0129] Step 226: Use the second vector determination unit to process the local feature map, initial semantic features and intermediate semantic features of the current driving position to obtain the key vector, query vector and value vector of the cross-attention mechanism.

[0130] Specifically, the first trajectory prediction module includes a second vector determination unit, a context vector determination unit, an optimization unit, and a third weight determination unit. The query vector of the cross-attention mechanism is obtained by concatenating the local feature map of the current driving position and the initial semantic features, the value vector of the cross-attention mechanism is the intermediate semantic features, and the key vector of the cross-attention mechanism is the intermediate semantic features. The second vector determination unit is a unit responsible for processing the local feature map of the current driving position, the initial semantic features, and the intermediate semantic features to generate the key vector, query vector, and value vector of the cross-attention mechanism.

[0131] In the specific implementation, the key vector of the cross-attention mechanism = the intermediate semantic feature. The query vector of the cross-attention mechanism = the image vector corresponding to the local feature map of the current driving position + the initial semantic feature. The value vector of the cross-attention mechanism = the intermediate semantic feature.

[0132] In this embodiment, the second vector determination unit is used to process the local feature map, initial semantic features and intermediate semantic features of the current driving position to obtain the key vector, query vector and value vector of the cross-attention mechanism, which can enhance the feature expression capability and improve the model performance.

[0133] Step 227: Use the context vector determination unit to process the key vector of the cross-attention mechanism, the query vector of the cross-attention mechanism, and the value vector of the cross-attention mechanism to obtain a context vector.

[0134] Specifically, the context vector determination unit is a unit responsible for processing the key vector of the cross attention mechanism, the query vector of the cross attention mechanism, and the value vector of the cross attention mechanism to generate a context vector. The context vector refers to a vector obtained according to the key vector of the cross attention mechanism, the query vector of the cross attention mechanism, and the value vector of the cross attention mechanism.

[0135] Furthermore, step 227 may specifically include: calculating the inner product of the key vector of the cross-attention mechanism and the query vector of the cross-attention mechanism to obtain the similarity of the cross-attention mechanism; normalizing the similarity of the cross-attention mechanism based on a normalized exponential function to obtain the attention weight of the cross-attention mechanism; calculating the product of the attention weight of the cross-attention mechanism and the intermediate value vector to obtain the context vector.

[0136] In the specific implementation, we can first calculate the inner product of the key vector of the cross attention mechanism and the query vector of the cross attention mechanism to obtain the similarity of the cross attention mechanism. The specific calculation formula is as follows:

[0137]

[0138] Among them, S represents the similarity of the cross-attention mechanism, K represents the key vector of the cross-attention mechanism, Q represents the query vector of the cross-attention mechanism, and d K Represents the dimension of the key vector for the criss-cross attention mechanism.

[0139] Then, the softmax function can be used to normalize the similarity of the cross-attention mechanism to obtain the attention weight of the cross-attention mechanism. Finally, the product of the attention weight of the cross-attention mechanism and the intermediate value vector is determined as the context vector.

[0140] In this embodiment, the above steps can combine visual features and text features to help the model better understand the scene semantics.

[0141] Step 228: Utilize the optimization unit to optimize the context vector, the local feature map, and the initial semantic feature to obtain an intermediate local feature map and a target semantic feature.

[0142] Specifically, there is at least one intermediate local feature map, and each intermediate local feature map corresponds to a target semantic feature. The optimization unit is responsible for processing the context vector, the local feature map, and the initial semantic feature to generate the intermediate local feature map and the target semantic feature. The intermediate local feature map refers to the local feature map after optimization processing. The target semantic feature refers to the semantic feature after optimization processing.

[0143] In the specific implementation, the context vector can be combined with the local feature map based on addition, multiplication or gating mechanism to obtain the intermediate local feature map, and the local feature map can be fused with the initial semantic features based on skip connection, element-by-element addition, splicing, etc., and then the fused features can be nonlinearly transformed using convolutional layers, activation functions, normalization layers, etc. to obtain the target semantic features.

[0144] In this embodiment, the optimization unit generates a richer and more comprehensive feature representation (intermediate local feature maps and target semantic features) by integrating information from different sources (context vectors, local feature maps, and initial semantic features), which helps to capture multi-level and multi-dimensional information in the image and improve the spatial resolution and semantic accuracy of the features, thereby improving the performance of the model and further improving the accuracy of determining the next driving position.

[0145] Step 229: Use the third weight determination unit to normalize the intermediate local feature map and the target semantic feature to obtain the weight of the self-attention mechanism corresponding to the intermediate local feature map.

[0146] Specifically, the third weight determination unit is a unit responsible for processing the intermediate local feature map and the target semantic feature to obtain the weight of the self-attention mechanism corresponding to the intermediate local feature map.

[0147] Furthermore, step 229 may specifically include: for any intermediate local feature map, calculating the inner product of the key vector of the self-attention mechanism and the query vector of the self-attention mechanism to obtain the similarity of the self-attention mechanism; normalizing the similarity of the self-attention mechanism based on the normalized exponential function to obtain the weight of the self-attention mechanism corresponding to the intermediate local feature map.

[0148] Specifically, the query vector of the self-attention mechanism, the value vector of the self-attention mechanism, and the key vector of the self-attention mechanism are all concatenated by the intermediate local feature map and the target semantic features.

[0149] In the specific implementation, for any intermediate local feature map, the inner product of the key vector of the self-attention mechanism and the query vector of the self-attention mechanism can be calculated first to obtain the similarity of the self-attention mechanism. The specific calculation formula is as follows:

[0150]

[0151] Among them, Si represents the similarity of the self-attention mechanism, K1 represents the key vector of the self-attention mechanism, Q1 represents the query vector of the self-attention mechanism, and the key vector of the self-attention mechanism = the vector corresponding to the target semantic feature + the image vector corresponding to the intermediate local feature map; the query vector of the self-attention mechanism = the vector corresponding to the target semantic feature + the image vector corresponding to the intermediate local feature map.

[0152] Then the similarity of the self-attention mechanism is normalized with the softmax function to obtain the weight of the self-attention mechanism corresponding to the intermediate local feature map. In addition, in order to avoid the saturation of the softmax function due to the excessive value of the inner product, the similarity of the self-attention mechanism is usually divided by sqrt (the dimension of the key vector of the self-attention mechanism) before normalizing the similarity of the self-attention mechanism with the softmax function.

[0153] In this embodiment, the above steps can capture the similarities between different features, thereby determining which features should be given more attention in subsequent processing and improving the performance of the model.

[0154] Step 230: Determine the next driving position of the current driving position according to the weight of the self-attention mechanism corresponding to the intermediate local feature map.

[0155] Furthermore, step 230 may specifically include: using a screening unit to select the largest attention weight of the self-attention mechanism, and determining the next driving position of the current driving position based on the intermediate local feature map with the largest attention weight of the self-attention mechanism.

[0156] Specifically, the first trajectory prediction module further includes a screening unit. The screening unit is a unit for selecting the largest attention weight of the self-attention mechanism.

[0157] In a specific implementation, after obtaining the weight of the self-attention mechanism corresponding to each intermediate local feature map, each intermediate local feature map can be sorted according to the weight of the self-attention mechanism corresponding to each intermediate local feature map, and then the intermediate local feature map with the largest weight of the self-attention mechanism is selected, and then the intermediate local feature map with the largest attention weight of the self-attention mechanism is analyzed (such as feature extraction, pattern recognition or classification, etc.) to obtain the analysis result. Based on the analysis result, and in combination with the current position and driving direction of the vehicle, a decision algorithm (such as rule-based, reinforcement learning or path planning algorithm, etc.) is used to determine the next driving position of the current driving position. After determining the next driving position, the control system of the mobile device will perform corresponding steering, acceleration or deceleration and other actions to guide the mobile device toward the target position.

[0158] In this embodiment, the above steps can improve the decision accuracy and select the optimal next driving position.

[0159] For example, Figure 2c As shown in the figure, after obtaining the panoramic environment image of the current driving position, the panoramic environment image of the current driving position can be extracted globally through the neural network ResNet-152 to obtain the global feature map of the current driving position, and then the local slot attention module is used to extract local features of the global feature map of the current driving position to obtain a suboptimal slot vector. Then the suboptimal slot vector is fed back to the slot attention mechanism through a jump connection, that is, the suboptimal slot vector is determined as the query vector of the slot attention mechanism, and the step of obtaining the suboptimal slot vector is re-executed until the preset stop condition is met, and then the feature map corresponding to the suboptimal slot vector is determined as the local feature map of the current driving position. At the same time, after obtaining the language instruction, the language instruction is first converted into a vector form by using semantic embedding, and then the language instruction in vector form is processed by using the Transformer encoder to obtain a vector with rich semantic information. Then, the vector with rich semantic information is processed by using state embedding to obtain the semantic features of the language instruction. Then, the local feature map of the current driving position and the semantic features of the language instruction are processed by a multi-layer Transformer to obtain the next driving position of the current driving position, that is, a t It should be noted that t = 0 means that the model is in the starting state. And t> 1 means that the model has been processed at least once, and the state a of the previous step can be t Embedded into the semantic space for this generation.

[0160] Therefore, the technical solution of the present invention first obtains the panoramic environment image of the voice control command to be executed and the current driving position, which provides a data basis for determining the local feature map of the current driving position and the semantic features of the voice control command to be executed. Then, the voice control command to be executed and the panoramic environment image of the current driving position are input into the target navigation model, which provides a data basis for determining the next driving position of the current driving position. Then, the panoramic environment image of the current driving position is convoluted by the convolution unit to obtain the primary global feature map, which provides a data basis for obtaining the intermediate global feature map. The primary global feature map is nonlinearly transformed and downsampled by the residual unit, which not only obtains the intermediate global feature map, but also alleviates the gradient disappearance problem and improves the feature expression ability. The intermediate global feature map is processed by the global average pooling unit for global average pooling, which not only obtains the global feature map of the current driving position, but also reduces the model parameters and calculation amount, and improves the generalization ability of the model. Then, the azimuth angle and elevation angle of each panoramic view are determined, which provides a data basis for obtaining the position encoding of the panoramic environment image of the current driving position. Then, the azimuth angle and elevation angle of each panoramic view are encoded to obtain the position encoding of the panoramic environment image of the current driving position, which provides a data basis for obtaining the key vector and value vector of the slot attention mechanism. The first vector determination unit is used to process the global feature map of the current driving position and the position encoding of the panoramic environment image of the current driving position to obtain the key vector and value vector of the slot attention mechanism, which can effectively fuse image information and position information, help generate more accurate and rich feature representations, and thus improve the accuracy and efficiency of subsequent calculations. The query vector, key vector and value vector of the slot attention mechanism are normalized by the first weight determination unit to obtain the weight of the slot attention mechanism, thereby improving the accuracy of weight determination. The first weighted vector determination unit is used to calculate the product of the weight of the slot attention mechanism and the value vector of the slot attention mechanism to obtain the weighted value vector, which can more prominently express the key information related to the current task, and help the model focus more on these important information in subsequent processing, thereby improving the overall performance. By updating the weighted value vector using a gated recurrent unit, the information in the weighted value vector can be further integrated and refined, thereby generating a more refined and accurate intermediate slot vector. The intermediate slot vector is processed nonlinearly using a multilayer perceptron to obtain a suboptimal slot vector, which provides a data basis for obtaining the local feature map of the current driving position. Then determine whether the number of executions reaches the preset number. If the number of executions reaches the preset number, the feature map corresponding to the suboptimal slot vector is determined as the local feature map of the current driving position.If the number of executions does not reach the preset number, the suboptimal slot vector is determined as the query vector of the slot attention mechanism, and the step of obtaining the weight of the slot attention mechanism by using the first weight determination unit is returned to execute, so as to fully capture the rich visual clues in the image, thereby improving the accuracy of the predicted next driving position of the current driving position. Then, the first voice processing module is used to extract the features of the voice control instructions to be executed, and the semantic features of the voice control instructions to be executed are obtained, which provides a data basis for predicting the next driving position of the current driving position. Then, the local feature map, initial semantic features and intermediate semantic features of the current driving position are processed by the second vector determination unit to obtain the key vector, query vector and value vector of the cross attention mechanism, which strengthens the feature expression ability and improves the model performance. The key vector of the cross attention mechanism, the query vector of the cross attention mechanism and the value vector of the cross attention mechanism are processed by the context vector determination unit to obtain the context vector, which helps the model better understand the scene semantics. The context vector, the local feature map and the initial semantic features are optimized by the optimization unit to obtain the intermediate local feature map and the target semantic features, which helps to capture the multi-level and multi-dimensional information in the image and improve the spatial resolution and semantic accuracy of the features. The intermediate local feature map and the target semantic feature are normalized by the third weight determination unit to obtain the weight of the self-attention mechanism corresponding to the intermediate local feature map, which helps to improve the performance of the model. According to the weight of the self-attention mechanism corresponding to the intermediate local feature map, the next driving position of the current driving position is determined, which improves the decision accuracy and makes the accuracy of the next driving position determined higher, thereby improving the accuracy and reliability of navigation. Compared with the prior art that mainly relies on extracting a single-level feature information from the image to make navigation decisions, the present invention extracts global features from the panoramic environment image of the current driving position to obtain a global feature map of the current driving position, and then extracts local features from the global feature map of the current driving position to obtain a local feature map of the current driving position, which can fully capture the rich visual clues in the image, thereby improving the accuracy of determining the next driving position, and further improving the accuracy and reliability of navigation. It solves the problem that the prior art only relies on a single-level feature data and cannot fully capture the rich visual clues in the image, resulting in insufficient accuracy of the motion trajectory determined during the navigation process, and it is difficult to meet the needs of high-precision navigation.

[0161] Figure 3 A structural schematic diagram of a navigation device provided in an embodiment of the present invention. The device and the navigation methods of the above embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the navigation device, reference can be made to the embodiments of the above navigation methods.

[0162] like Figure 3 As shown, the device comprises:

[0163] The acquisition module 310 is used to acquire the voice control command to be executed and acquire the panoramic environment image of the current driving position;

[0164] An input module 320, used for inputting the voice control command to be executed and the panoramic environment image of the current driving position into a target navigation model, wherein the target navigation model includes a first voice processing module, a first image processing module and a first trajectory prediction module;

[0165] A first determination module 330 is used to perform global feature extraction on the panoramic environment image of the current driving position using the first image processing module to obtain a global feature map of the current driving position, and perform local feature extraction on the global feature map of the current driving position using the first image processing module to obtain a local feature map of the current driving position;

[0166] A second determination module 340 is used to extract features of the voice control instruction to be executed by using the first voice processing module to obtain semantic features of the voice control instruction to be executed;

[0167] The third determination module 350 is used to predict the next driving position of the current driving position based on the local feature map of the current driving position and the semantic features of the voice control instruction to be executed by using the first trajectory prediction module.

[0168] Based on the above embodiment, the acquisition module 310 acquires the panoramic environment image of the current driving position, including:

[0169] Partition the panoramic view to obtain multiple partitioned viewpoints;

[0170] The current environment is photographed at different angles in each subarea viewing angle to obtain a panoramic view at different angles of each subarea viewing angle, wherein the panoramic environment image includes panoramic views at different angles of different subarea viewing angles.

[0171] On the basis of the above embodiment, the first image processing module includes a global image processing module, and the global image processing module includes a convolution unit, a residual unit and a global average pooling unit. The first determination module 330 uses the first image processing module to perform global feature extraction on the panoramic environment image of the current driving position to obtain a global feature map of the current driving position, including:

[0172] Using the convolution unit to perform convolution processing on the panoramic environment image of the current driving position to obtain a primary global feature map;

[0173] Using the residual unit to perform nonlinear transformation and down-sampling on the primary global feature map to obtain an intermediate global feature map;

[0174] The global average pooling unit is used to perform global average pooling processing on the intermediate global feature map to obtain the global feature map of the current driving position.

[0175] Based on the above embodiment, the device further includes:

[0176] A position module is used to determine the azimuth angle and elevation angle of each panoramic view before using the first image processing module to perform local feature extraction on the global feature map of the current driving position to obtain the local feature map of the current driving position; encode the azimuth angle and elevation angle of each panoramic view to obtain the position code of the panoramic environment image of the current driving position.

[0177] On the basis of the above embodiment, the first image processing module includes a local image processing module, and the local image processing module includes a first vector determination unit, a first weight determination unit, a first weighted vector determination unit, a gated recurrent unit GRU and a multi-layer perceptron MLP. The first determination module 330 uses the first image processing module to extract local features from the global feature map of the current driving position to obtain the local feature map of the current driving position, including:

[0178] The first vector determination unit is used to process the global feature map of the current driving position and the position encoding of the panoramic environment image of the current driving position to obtain a key vector and a value vector of a slot attention mechanism; wherein the key vector of the slot attention mechanism is obtained by concatenating a global image vector and a position vector, and the value vector of the slot attention mechanism is the global image vector; the global image vector is an image vector corresponding to the global feature map of the current driving position, and the position vector is a position vector corresponding to the position encoding of the panoramic environment image of the current driving position;

[0179] Using the first weight determination unit, normalizing the query vector of the slot attention mechanism, the key vector of the slot attention mechanism, and the value vector of the slot attention mechanism to obtain the weight of the slot attention mechanism; wherein the query vector of the slot attention mechanism is a preset slot vector;

[0180] Utilizing the first weighted vector determination unit to calculate the product of the weight of the slot attention mechanism and the value vector of the slot attention mechanism to obtain a weighted value vector;

[0181] Using the GRU to update the weighted value vector to obtain an intermediate slot vector;

[0182] The MLP is used to perform nonlinear processing on the intermediate slot vector to obtain a suboptimal slot vector, and the suboptimal slot vector is determined as the query vector of the slot attention mechanism, and the step of obtaining the weight of the slot attention mechanism by using the first weight determination unit is returned to be executed until the number of executions reaches a preset number of times, and the feature map corresponding to the suboptimal slot vector obtained for the last time is determined as the local feature map of the current driving position.

[0183] On the basis of the above embodiment, the semantic features of the voice control instruction to be executed include initial semantic features and intermediate semantic features, and the first trajectory prediction module includes: a second vector determination unit, a context vector determination unit, an optimization unit and a third weight determination unit, and the third determination module 340 is specifically used for:

[0184] The local feature map of the current driving position, the initial semantic feature and the intermediate semantic feature are processed by the second vector determination unit to obtain a key vector, a query vector and a value vector of a cross-attention mechanism; wherein the query vector of the cross-attention mechanism is obtained by concatenating the local feature map of the current driving position and the initial semantic feature, the value vector of the cross-attention mechanism is the intermediate semantic feature, and the key vector of the cross-attention mechanism is the intermediate semantic feature;

[0185] Processing the key vector of the cross-attention mechanism, the query vector of the cross-attention mechanism, and the value vector of the cross-attention mechanism by the context vector determination unit to obtain a context vector;

[0186] Utilizing the optimization unit to optimize the context vector, the local feature map and the initial semantic feature, to obtain an intermediate local feature map and a target semantic feature; wherein there is at least one intermediate local feature map, and each intermediate local feature map corresponds to a target semantic feature;

[0187] Using a third weight determination unit to normalize the intermediate local feature map and the target semantic feature to obtain a weight of a self-attention mechanism corresponding to the intermediate local feature map;

[0188] The next driving position of the current driving position is determined according to the weight of the self-attention mechanism corresponding to the intermediate local feature map.

[0189] On the basis of the above embodiment, the third determination module 340 uses the context vector determination unit to process the key vector of the cross attention mechanism, the query vector of the cross attention mechanism, and the value vector of the cross attention mechanism to obtain a context vector, including:

[0190] Calculating the inner product of the key vector of the cross attention mechanism and the query vector of the cross attention mechanism to obtain the similarity of the cross attention mechanism;

[0191] Normalizing the similarity of the cross attention mechanism based on a normalized exponential function to obtain an attention weight of the cross attention mechanism;

[0192] The product of the attention weight of the cross-attention mechanism and the intermediate value vector is calculated to obtain a context vector.

[0193] On the basis of the above embodiment, the third determination module 340 uses the third weight determination unit to normalize the intermediate local feature map and the target semantic feature to obtain the weight of the self-attention mechanism corresponding to the intermediate local feature map, including:

[0194] For any intermediate local feature map, calculate the inner product of the key vector of the self-attention mechanism and the query vector of the self-attention mechanism to obtain the similarity of the self-attention mechanism;

[0195] Normalizing the similarity of the self-attention mechanism based on a normalized exponential function to obtain a weight of the self-attention mechanism corresponding to the intermediate local feature map;

[0196] Among them, the query vector of the self-attention mechanism, the value vector of the self-attention mechanism and the key vector of the self-attention mechanism are all obtained by concatenating the intermediate local feature map and the target semantic feature.

[0197] On the basis of the above embodiment, the first trajectory prediction module further includes a screening unit, and the third determination module 340 determines the next driving position of the current driving position according to the weight of the self-attention mechanism corresponding to the intermediate local feature map, including:

[0198] The screening unit is used to select the largest attention weight of the self-attention mechanism, and the next driving position of the current driving position is determined according to the intermediate local feature map with the largest attention weight of the self-attention mechanism.

[0199] Based on the above embodiment, the training process of the target navigation model includes:

[0200] Obtain historical voice control instructions and obtain panoramic environment images of each historical driving position;

[0201] Inputting the historical voice control instructions and the panoramic environment images of each historical driving position into a preset navigation model, wherein the preset navigation model includes a second voice processing module, a second image processing module, and a second trajectory prediction module;

[0202] Using the second image processing module to perform global feature extraction on the panoramic environment image of each historical position, and correspondingly obtain a global feature map of each historical driving position, and using the second image processing module to perform local feature extraction on the global feature map of each historical driving position, and obtain a local feature map of each historical driving position;

[0203] Using the second voice processing module to perform feature extraction on the historical voice control instructions to obtain semantic features of the historical voice control instructions;

[0204] Using the second trajectory prediction module to predict the next driving position of each historical driving position based on the local feature map of each historical driving position and the semantic feature of the historical voice control instruction, so as to obtain a predicted driving trajectory;

[0205] Determining a function value of a loss function according to the predicted driving trajectory and the actual driving trajectory;

[0206] The model parameters of the preset navigation model are optimized based on the function value to obtain the target navigation model.

[0207] The navigation device provided in the embodiment of the present invention can execute the navigation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0208] It is worth noting that in the embodiment of the above-mentioned navigation device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0209] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 4 A block diagram of an exemplary electronic device 4 suitable for implementing embodiments of the present invention is shown. Figure 4 The electronic device 4 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0210] like Figure 4 As shown, the electronic device 4 is in the form of a general purpose computing electronic device. The components of the electronic device 4 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).

[0211] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0212] The electronic device 4 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 4, including volatile and non-volatile media, removable and non-removable media.

[0213] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 4 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 4 not shown, usually called a "hard drive"). Although Figure 4 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0214] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0215] The electronic device 4 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), one or more devices that enable a user to interact with the electronic device 4, and / or any device that enables the electronic device 4 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the electronic device 4 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 20. Figure 4 As shown, the network adapter 20 communicates with other modules of the electronic device 4 via the bus 18. It should be understood that although Figure 4 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 4, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0216] The processing unit 16 executes various functional applications and page displays by running the programs stored in the system memory 28, for example, implementing the navigation method provided by the embodiment of the present invention, which includes:

[0217] Obtaining the voice control command to be executed and obtaining the panoramic environment image of the current driving position;

[0218] Inputting the voice control command to be executed and the panoramic environment image of the current driving position into a target navigation model, wherein the target navigation model includes a first voice processing module, a first image processing module and a first trajectory prediction module;

[0219] Using the first image processing module to perform global feature extraction on the panoramic environment image of the current driving position to obtain a global feature map of the current driving position, and using the first image processing module to perform local feature extraction on the global feature map of the current driving position to obtain a local feature map of the current driving position;

[0220] Using the first voice processing module to perform feature extraction on the voice control instruction to be executed, so as to obtain semantic features of the voice control instruction to be executed;

[0221] The first trajectory prediction module is used to predict the next driving position of the current driving position based on the local feature map of the current driving position and the semantic features of the voice control instruction to be executed.

[0222] Of course, those skilled in the art will appreciate that the processor may also implement the technical solution of the navigation method provided by any embodiment of the present invention.

[0223] An embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, for example, a navigation method provided by an embodiment of the present invention is implemented. The method includes:

[0224] Obtaining the voice control command to be executed and obtaining the panoramic environment image of the current driving position;

[0225] Inputting the voice control command to be executed and the panoramic environment image of the current driving position into a target navigation model, wherein the target navigation model includes a first voice processing module, a first image processing module and a first trajectory prediction module;

[0226] Using the first image processing module to perform global feature extraction on the panoramic environment image of the current driving position to obtain a global feature map of the current driving position, and using the first image processing module to perform local feature extraction on the global feature map of the current driving position to obtain a local feature map of the current driving position;

[0227] Using the first voice processing module to perform feature extraction on the voice control instruction to be executed, so as to obtain semantic features of the voice control instruction to be executed;

[0228] The first trajectory prediction module is used to predict the next driving position of the current driving position based on the local feature map of the current driving position and the semantic features of the voice control instruction to be executed.

[0229] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0230] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0231] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0232] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0233] It should be understood by those skilled in the art that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0234] In addition, the acquisition, storage, use, and processing of data in the technical solution of the present invention are in compliance with the relevant provisions of national laws and regulations.

[0235] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A navigation method, characterized in that: The method comprises: Obtaining the voice control command to be executed and obtaining the panoramic environment image of the current driving position; Inputting the voice control command to be executed and the panoramic environment image of the current driving position into a target navigation model, wherein the target navigation model includes a first voice processing module, a first image processing module and a first trajectory prediction module; Using the first image processing module to perform global feature extraction on the panoramic environment image of the current driving position to obtain a global feature map of the current driving position, and using the first image processing module to perform local feature extraction on the global feature map of the current driving position to obtain a local feature map of the current driving position; Using the first voice processing module to perform feature extraction on the voice control instruction to be executed, so as to obtain semantic features of the voice control instruction to be executed; The first trajectory prediction module is used to predict the next driving position of the current driving position based on the local feature map of the current driving position and the semantic features of the voice control instruction to be executed.

2. The navigation method according to claim 1, characterized in that: The step of obtaining a panoramic environment image of the current driving position includes: Partition the panoramic view to obtain multiple partitioned viewpoints; The current environment is photographed at different angles in each subarea viewing angle to obtain a panoramic view at different angles of each subarea viewing angle, wherein the panoramic environment image includes panoramic views at different angles of different subarea viewing angles.

3. The navigation method according to claim 1, characterized in that: The first image processing module includes a global image processing module, and the global image processing module includes a convolution unit, a residual unit, and a global average pooling unit. The first image processing module is used to perform global feature extraction on the panoramic environment image of the current driving position to obtain a global feature map of the current driving position, including: Using the convolution unit to perform convolution processing on the panoramic environment image of the current driving position to obtain a primary global feature map; Using the residual unit to perform nonlinear transformation and down-sampling on the primary global feature map to obtain an intermediate global feature map; The global average pooling unit is used to perform global average pooling processing on the intermediate global feature map to obtain the global feature map of the current driving position.

4. The navigation method according to claim 2, characterized in that: Before using the first image processing module to extract local features from the global feature map of the current driving position to obtain the local feature map of the current driving position, the method further includes: Determining the azimuth angle and elevation angle of each panoramic view; The azimuth angle and elevation angle of each panoramic view are encoded to obtain a position code of the panoramic environment image of the current driving position.

5. The navigation method according to claim 4, characterized in that: The first image processing module includes a local image processing module, and the local image processing module includes a first vector determination unit, a first weight determination unit, a first weighted vector determination unit, a gated recurrent unit GRU, and a multi-layer perceptron MLP. The first image processing module is used to perform local feature extraction on the global feature map of the current driving position to obtain the local feature map of the current driving position, including: The first vector determination unit is used to process the global feature map of the current driving position and the position encoding of the panoramic environment image of the current driving position to obtain a key vector and a value vector of a slot attention mechanism; wherein the key vector of the slot attention mechanism is obtained by concatenating a global image vector and a position vector, and the value vector of the slot attention mechanism is the global image vector; the global image vector is an image vector corresponding to the global feature map of the current driving position, and the position vector is a position vector corresponding to the position encoding of the panoramic environment image of the current driving position; Using the first weight determination unit, normalizing the query vector of the slot attention mechanism, the key vector of the slot attention mechanism, and the value vector of the slot attention mechanism to obtain the weight of the slot attention mechanism; wherein the query vector of the slot attention mechanism is a preset slot vector; Utilizing the first weighted vector determination unit to calculate the product of the weight of the slot attention mechanism and the value vector of the slot attention mechanism to obtain a weighted value vector; Using the GRU to update the weighted value vector to obtain an intermediate slot vector; The MLP is used to perform nonlinear processing on the intermediate slot vector to obtain a suboptimal slot vector, and the suboptimal slot vector is determined as the query vector of the slot attention mechanism, and the step of obtaining the weight of the slot attention mechanism by using the first weight determination unit is returned to be executed until the number of executions reaches a preset number of times, and the feature map corresponding to the suboptimal slot vector obtained for the last time is determined as the local feature map of the current driving position.

6. The navigation method according to claim 1, characterized in that: The semantic features of the voice control instruction to be executed include initial semantic features and intermediate semantic features, the first trajectory prediction module includes: a second vector determination unit, a context vector determination unit, an optimization unit and a third weight determination unit, and the first trajectory prediction module is used to predict the next driving position of the current driving position based on the local feature map of the current driving position and the semantic features of the voice control instruction to be executed, including: The local feature map of the current driving position, the initial semantic feature and the intermediate semantic feature are processed by the second vector determination unit to obtain a key vector, a query vector and a value vector of a cross-attention mechanism; wherein the query vector of the cross-attention mechanism is obtained by concatenating the local feature map of the current driving position and the initial semantic feature, the value vector of the cross-attention mechanism is the intermediate semantic feature, and the key vector of the cross-attention mechanism is the intermediate semantic feature; Processing the key vector of the cross-attention mechanism, the query vector of the cross-attention mechanism, and the value vector of the cross-attention mechanism by the context vector determination unit to obtain a context vector; Utilizing the optimization unit to optimize the context vector, the local feature map and the initial semantic feature, to obtain an intermediate local feature map and a target semantic feature; wherein there is at least one intermediate local feature map, and each intermediate local feature map corresponds to a target semantic feature; Using a third weight determination unit to normalize the intermediate local feature map and the target semantic feature to obtain a weight of a self-attention mechanism corresponding to the intermediate local feature map; The next driving position of the current driving position is determined according to the weight of the self-attention mechanism corresponding to the intermediate local feature map.

7. The navigation method according to claim 6, characterized in that: The context vector determination unit is used to process the key vector of the cross attention mechanism, the query vector of the cross attention mechanism, and the value vector of the cross attention mechanism to obtain a context vector, including: Calculating the inner product of the key vector of the cross attention mechanism and the query vector of the cross attention mechanism to obtain the similarity of the cross attention mechanism; Normalizing the similarity of the cross attention mechanism based on a normalized exponential function to obtain an attention weight of the cross attention mechanism; The product of the attention weight of the cross-attention mechanism and the intermediate value vector is calculated to obtain a context vector.

8. The navigation method according to claim 6, characterized in that: The intermediate local feature map and the target semantic feature are normalized by using a third weight determination unit to obtain a weight of a self-attention mechanism corresponding to the intermediate local feature map, including: For any intermediate local feature map, calculate the inner product of the key vector of the self-attention mechanism and the query vector of the self-attention mechanism to obtain the similarity of the self-attention mechanism; Normalizing the similarity of the self-attention mechanism based on a normalized exponential function to obtain a weight of the self-attention mechanism corresponding to the intermediate local feature map; Among them, the query vector of the self-attention mechanism, the value vector of the self-attention mechanism and the key vector of the self-attention mechanism are all obtained by concatenating the intermediate local feature map and the target semantic feature.

9. The navigation method according to claim 6, characterized in that: The first trajectory prediction module further includes a screening unit, which determines the next driving position of the current driving position according to the weight of the self-attention mechanism corresponding to the intermediate local feature map, including: The screening unit is used to select the largest attention weight of the self-attention mechanism, and the next driving position of the current driving position is determined according to the intermediate local feature map with the largest attention weight of the self-attention mechanism.

10. The navigation method according to claim 1, characterized in that: The training process of the target navigation model includes: Obtain historical voice control instructions and obtain panoramic environment images of each historical driving position; Inputting the historical voice control instructions and the panoramic environment images of each historical driving position into a preset navigation model, wherein the preset navigation model includes a second voice processing module, a second image processing module, and a second trajectory prediction module; Using the second image processing module to perform global feature extraction on the panoramic environment image of each historical position, and correspondingly obtain a global feature map of each historical driving position, and using the second image processing module to perform local feature extraction on the global feature map of each historical driving position, and obtain a local feature map of each historical driving position; Using the second voice processing module to perform feature extraction on the historical voice control instructions to obtain semantic features of the historical voice control instructions; Using the second trajectory prediction module to predict the next driving position of each historical driving position based on the local feature map of each historical driving position and the semantic feature of the historical voice control instruction, so as to obtain a predicted driving trajectory; Determining a function value of a loss function according to the predicted driving trajectory and the actual driving trajectory; The model parameters of the preset navigation model are optimized based on the function value to obtain the target navigation model.

11. A navigation device, characterized in that: The device comprises: An acquisition module, used to acquire the voice control command to be executed and acquire a panoramic environment image of the current driving position; An input module, used for inputting the voice control command to be executed and the panoramic environment image of the current driving position into a target navigation model, wherein the target navigation model includes a first voice processing module, a first image processing module and a first trajectory prediction module; a first determining module, configured to extract global features from the panoramic environment image of the current driving position using the first image processing module to obtain a global feature map of the current driving position, and to extract local features from the global feature map of the current driving position using the first image processing module to obtain a local feature map of the current driving position; a second determining module, configured to extract features of the voice control instruction to be executed by using the first voice processing module to obtain semantic features of the voice control instruction to be executed; The third determination module is used to use the first trajectory prediction module to predict the next driving position of the current driving position based on the local feature map of the current driving position and the semantic features of the voice control instruction to be executed.

12. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the navigation method described in any one of claims 1-10.

13. A storage medium containing computer executable instructions, characterized in that: When the computer executable instructions are executed by a computer processor, the navigation method according to any one of claims 1 to 10 is implemented.

Images