Electric two-wheeler navigation method and system based on multi-modal semantic enhancement

Abstract

This application relates to a navigation method for electric two-wheelers based on multimodal semantic enhancement. The method includes: acquiring user input information; parsing the input information based on a pre-built multimodal large language model to obtain multimodal semantic information; fusing the multimodal semantic information with location data, vehicle battery data, and time data based on a semantically enhanced spatiotemporal trajectory model to generate a semantically enhanced spatiotemporal trajectory representation, which includes energy consumption constraints and safety preferences; generating a navigation path that conforms to the energy consumption constraints and safety preferences based on the spatiotemporal trajectory representation; and outputting the navigation path in a visual form. The semantically enhanced spatiotemporal trajectory model supports the inclusion of unstructured roads in the output path. This application solves the problem of navigation systems being unsuitable for electric two-wheelers. The semantically enhanced spatiotemporal trajectory model supports outputting paths containing unstructured roads, allowing electric two-wheelers to travel on routes that motor vehicles cannot access, such as alleys and pedestrian streets.

Description

Technical Field

[0001] This application relates to the field of route navigation technology, and in particular to a navigation method and system for electric two-wheeled vehicles based on multimodal semantic enhancement. Background Technology

[0002] Whether professional riders or ordinary users, their ability to cope with unknown routes and complex road conditions during actual riding is limited. Especially in densely populated urban environments, the riding routes of electric two-wheelers are highly dependent on familiarity with unstructured roads (such as alleys, lanes, internal passages of residential areas, and shortcuts that are not marked on maps but are actually passable). Novice riders or users who rely on traditional navigation are prone to detours, running out of power, and even facing safety risks due to unreasonable route planning.

[0003] Most current mainstream navigation systems are based on static map data and use A Classical path planning algorithms such as Dijkstra's algorithm were primarily designed for four-wheeled vehicles, imposing strong constraints on road rules and lane structures. These methods struggle to capture the unique agility and maneuverability of two-wheeled vehicles. Furthermore, existing navigation systems typically require users to explicitly input destination coordinates or names, lacking the ability to understand user intent and proactively guide them. Summary of the Invention

[0004] This application provides a navigation method, system, electronic device, and storage medium for electric two-wheelers based on multimodal semantic enhancement, to at least solve the problem that existing navigation systems in the related art are not compatible with electric two-wheelers.

[0005] In a first aspect, embodiments of this application provide a navigation method for electric two-wheeled vehicles based on multimodal semantic enhancement, the method comprising: The system acquires user input information and parses the input information based on a pre-built multimodal large language model to obtain multimodal semantic information. Based on the semantically enhanced spatiotemporal trajectory model, the multimodal semantic information is fused with location data, vehicle battery data and time data to generate a semantically enhanced spatiotemporal trajectory representation, which includes energy consumption constraints and safety preferences. Based on the spatiotemporal trajectory representation, a navigation path that conforms to the energy consumption constraints and the safety preferences is generated and output in a visual form. The semantically enhanced spatiotemporal trajectory model supports the inclusion of unstructured roads in the output path.

[0006] In some embodiments, the multimodal semantic information includes the target point for path planning; the process of parsing the input information based on a pre-built multimodal large language model to obtain the multimodal semantic information includes: Based on the multimodal large language model, the input information is parsed to obtain semantic parsing results; If the semantic parsing result contains a clear destination, the destination will be used as the target point for route planning. If the semantic parsing result does not contain a specific destination, the target point for route planning is determined based on the semantic parsing result, the geographic semantic knowledge base, and the power consumption data.

[0007] In some embodiments, determining the target point for route planning based on the semantic parsing results, the geographic semantic knowledge base, and the power consumption data includes: Obtain the distribution data of battery swapping facilities from the geographic semantic knowledge base; Based on the semantic parsing results, the geographic semantic knowledge base, the distribution of the battery swapping facilities, weather data, and the electricity data, candidate destinations are determined. The candidate destinations are ranked based on accessibility, task suitability, and range safety. The target point for the route planning is then determined from the candidate destinations based on the ranking results.

[0008] In some embodiments, fusing the multimodal semantic information with location data, the battery data of the two-wheeled vehicle, and time data to generate a semantically enhanced spatiotemporal trajectory representation includes: Based on the multimodal semantic information, the location data, the battery data of the two-wheeled vehicle, and the time data, a riding trajectory sequence with time-series battery information is generated; Encode the cycling trajectory sequence with time-series battery information based on the spatiotemporal trajectory encoder; The multimodal semantic information is encoded based on a semantic feature encoder; By using a cross-modal attention mechanism, the encoding of the multimodal semantic information is deeply fused with the encoding of the cycling trajectory sequence to generate the semantically enhanced spatiotemporal trajectory representation.

[0009] In some embodiments, generating a navigation path that conforms to the energy consumption constraints and the safety preferences based on the spatiotemporal trajectory representation includes: The spatiotemporal trajectory representation is decoded using a Transformer-based decoder. The decoded spatiotemporal trajectory representation is input into the safety prediction head, displacement prediction head, and power consumption prediction head respectively, and three corresponding output results are obtained. By combining the three output results, candidate trajectory points for the next position are determined, and a complete navigation path is formed in an autoregressive manner.

[0010] In some embodiments, the safety prediction head is used to perform a binary classification task to predict the safety factor of a road segment; The displacement prediction head is used to perform a multi-classification task to predict the displacement direction and distance at the next moment; The power consumption prediction head is used to perform regression tasks to estimate power loss in road sections.

[0011] In some embodiments, the method further includes: Based on the output of the power consumption prediction head, the estimated remaining power is calculated; If the estimated remaining power is less than a preset power threshold, the nearest energy replenishment point is retrieved based on the current location. Insert the energy replenishment point as a temporary destination into the current path planning sequence, triggering path replanning with the temporary destination as the target.

[0012] In some embodiments, the method further includes: Obtain key waypoints in the navigation path, match the key waypoints with a geographic knowledge base, and obtain traffic data related to the navigation path; Traffic condition alerts are generated based on the traffic condition data, and then output.

[0013] Secondly, embodiments of this application provide a navigation system for electric two-wheeled vehicles based on multimodal semantic enhancement, the system comprising: The semantic parsing module is used to acquire user input information and parse the input information based on a pre-built multimodal large language model to obtain multimodal semantic information. The spatiotemporal trajectory representation generation module is used to fuse the multimodal semantic information with location data, vehicle power data and time data based on the semantically enhanced spatiotemporal trajectory model to generate a semantically enhanced spatiotemporal trajectory representation, which includes energy consumption constraints and safety preferences. The path generation module is used to generate a navigation path that conforms to the energy consumption constraints and the safety preferences based on the spatiotemporal trajectory representation, and output the navigation path in a visual form. The semantically enhanced spatiotemporal trajectory model supports the inclusion of unstructured roads in the output path.

[0014] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the electric two-wheeled vehicle navigation method based on multimodal semantic enhancement as described in the first aspect above.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the electric two-wheeler navigation method based on multimodal semantic enhancement as described in the first aspect above.

[0016] Compared to related technologies, the electric two-wheeler navigation method based on multimodal semantic enhancement provided in this application parses user input information based on a pre-built multimodal large language model, and is compatible with multiple input forms such as voice, text, and images. The semantically enhanced spatiotemporal trajectory model supports outputting paths containing unstructured roads, allowing electric two-wheelers to travel on routes that motor vehicles cannot use, such as alleys and pedestrian streets. This solves the problem that existing navigation systems are not compatible with electric two-wheelers, enabling shorter-distance and more flexible navigation, avoiding congested main roads, and improving travel efficiency. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of a navigation method for electric two-wheeled vehicles based on multimodal semantic enhancement according to an embodiment of this application; Figure 2 This is a semantically enhanced spatiotemporal trajectory model architecture diagram according to an embodiment of this application; Figure 3 This is an interactive flowchart from multimodal input to multimodal output according to an embodiment of this application; Figure 4 This is a flowchart of path generation reasoning based on a multimodal semantic enhancement spatiotemporal trajectory model according to an embodiment of this application; Figure 5 This is a structural block diagram of an electric two-wheeled vehicle navigation system based on multimodal semantic enhancement according to an embodiment of this application; Figure 6 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0019] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0020] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0021] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0022] This embodiment provides a navigation method for electric two-wheeled vehicles based on multimodal semantic enhancement. Figure 1 This is a flowchart of a navigation method for electric two-wheeled vehicles based on multimodal semantic enhancement according to an embodiment of this application, such as... Figure 1 As shown, the process includes the following steps: Step S101: Obtain user input information, parse the input information based on the pre-built multimodal large language model, and obtain multimodal semantic information.

[0023] When using this navigation method in practice, users first open the relevant application (applicable to electric two-wheeled vehicle scenarios such as on-demand delivery, daily commuting, or short trips) and provide their current status and travel needs through multimodal interaction. User input includes, but is not limited to, text, voice, images (such as photos of intersections), short video clips, voice + image mixed commands, environmental recordings, and biometric / behavioral signals collected through smart helmets or handles (such as gaze direction and grip pressure). For spatiotemporal status information not explicitly provided by the user (such as current location, remaining battery power, and current time), the system automatically combines GPS positioning, battery management system (BMS) data, and system clock data for intelligent completion.

[0024] In addition to user input, multimodal contextual information from external knowledge bases or environmental perception systems can be integrated, including but not limited to: real-time traffic camera footage, weather radar images, road construction announcement images, battery swapping station status APIs, implicit semantic labels generated by historical trajectory clustering, and text and image feedback reported by community users. This information is used to dynamically update the semantic attributes of the road network and enhance the model's ability to distinguish unstructured roads (such as temporarily opened alleys and waterlogged sections).

[0025] This embodiment utilizes a multimodal large language model fine-tuned with enterprise delivery business logic and urban electric two-wheeler traffic environment data to jointly understand and extract key information from raw multimodal signals (whether speech-to-text, direct text, image, or mixed modality), accurately identifying structured navigation constraints such as location, battery level, time window, safety preferences, and task priority. The large language model base used in this embodiment is an open-source general-purpose multimodal language model, which has been customized and fine-tuned for specific semantics related to electric two-wheeler riding, enabling it to more accurately understand fine-grained semantics in riding scenarios.

[0026] In this embodiment, the multimodal large language model mainly includes three logical modules: a multimodal encoder, a general multimodal large language model base, and a projection / alignment module.

[0027] Multimodal encoders are responsible for converting raw non-textual modal data (such as images and speech) into machine-understandable vectorized features, including but not limited to visual encoders, speech / audio encoders, and text encoders.

[0028] Visual encoder: For street view photos and road condition images uploaded by users, a pre-trained visual encoder is used to extract global semantic features and features of key objects (such as roads, vehicles, and signs) from the images.

[0029] Voice / audio encoder: For user voice commands or environmental recordings, a pre-trained speech recognition model is used to convert them into text, or an audio feature extraction model is used to directly obtain the semantic features of environmental sounds (such as construction noise and horn sounds).

[0030] Text encoder: Directly processes natural language text input by the user, usually implemented by the word embedding layer of the base large language model itself.

[0031] Multimodal large language model foundations are open-source models that have been pre-trained on large-scale image and text data and possess powerful general semantic understanding and reasoning capabilities. As the "brain" of the model, they receive features from the multimodal encoder. Examples include, but are not limited to, LLaVA, Qwen-VL, or InternLM-XComposer2. These foundational models themselves already possess the ability to align visual features with linguistic features and perform reasoning in a unified semantic space.

[0032] The projection / alignment module is a trainable network layer (usually a linear layer or a multilayer perceptron MLP) whose function is to project heterogeneous features extracted by different encoders (such as image feature vectors and audio feature vectors) into a unified semantic space that the base large language model can understand, thereby achieving the alignment and fusion of cross-modal information.

[0033] The user-input image is extracted into a feature vector by a visual encoder, and then fed into the base language model along with the embedding vector of the question text through a projection module. Based on its pre-trained knowledge, the base model performs a joint understanding of the image and text information, ultimately outputting structured semantics.

[0034] To adapt the aforementioned general base model to the vertical field of electric two-wheeled vehicle navigation, this embodiment performs supervised fine-tuning. The goal is to enable the model to learn to accurately extract structured constraint parameters that are crucial for path planning from the multimodal interactions in the riding scenario.

[0035] Optionally, the fine-tuning method in this embodiment employs a parameter-efficient fine-tuning method, such as LoRA. LoRA injects low-rank adaptation matrices into key parts of the base model, such as the attention layer, and trains only these few newly added parameters, thereby efficiently enabling the model to learn domain knowledge while avoiding catastrophic forgetting and significantly saving computational resources. Alternatively, a full-scale fine-tuning approach that continues training all parameters of the base model can also be used, suitable for scenarios with ample computational resources and a pursuit of ultimate performance.

[0036] The fine-tuning dataset is specially constructed, containing high-quality "multimodal input-structured output" paired samples, mainly divided into domain-general knowledge question answering and navigation scenario intent parsing data. The domain-general knowledge question answering covers common sense, rules, and facility knowledge related to riding electric two-wheelers; the navigation scenario intent parsing data simulates real user requests, training the model to extract precise navigation constraints from fuzzy descriptions. The training objective of the fine-tuning phase is to make the structured text output by the model (usually formatted as JSON or key-value pairs) consistent with the standard answers in the training data. This is typically calculated using the cross-entropy loss function, optimizing model parameters to maximize the probability of generating correct structured content.

[0037] After the above fine-tuning, the model was integrated into the navigation process of this invention, and its implementation mechanism and functions are clearly defined as follows: Input: Receive raw multimodal signals from users or the system (voice commands, directly entered text, uploaded real-time traffic images, and status description text triggered by onboard sensors).

[0038] Processing: The model uses its learned domain knowledge to perform joint reasoning on the input. For example, when it receives the text "low battery" and an image containing a battery swapping station sign at the same time, the model can associate the two and understand that the user's core need is "to find a battery swapping station".

[0039] Output: Multimodal semantic information (label information), i.e., numericalized, to facilitate subsequent model recognition. After converting the input information into digital results, it will be organized into a suitable format to adapt to the trajectory generation part of the semantically enhanced spatiotemporal trajectory model.

[0040] In some embodiments, the multimodal semantic information includes the target point for path planning; step S101 includes: Step S1011: Based on the multimodal large language model, the input information is parsed to obtain the semantic parsing result.

[0041] Step S1012: If the semantic parsing result contains a clear destination, the destination is used as the target point for path planning.

[0042] Step S1013: If the semantic parsing result does not contain a clear destination, determine the target point for route planning based on the semantic parsing result, the geographic semantic knowledge base, and the power data.

[0043] It's important to note that the geographic semantic knowledge base is a structured database that associates entities in geospatial space (roads, intersections, areas, facilities) with rich, human-understandable semantic information (i.e., meaning and context). Based on the geographic semantic knowledge base, it can draw on the collective experience of all riders to avoid accident-prone areas and choose the most economical routes.

[0044] The intelligent destination recommendation capability is deeply integrated into the navigation process of electric two-wheelers based on multimodal semantic enhancement: by understanding the user's vague or constrained travel intentions expressed in natural language through a large language model, and combining real-time environmental status (such as remaining battery power, weather, and regional traffic restrictions), geographic semantic knowledge base and individual historical preferences, destination candidates that meet multiple conditions are dynamically generated.

[0045] In some embodiments, step S1013 specifically includes: Step S201: Obtain the distribution data of battery swapping facilities from the geographic semantic knowledge base.

[0046] Step S202: Based on the semantic parsing results, the geographic semantic knowledge base, the distribution of battery swapping facilities, weather data, and electricity data, determine the candidate destinations.

[0047] Step S203: Sort the candidate destinations based on accessibility, task suitability and endurance safety, and determine the target point for route planning from the candidate destinations according to the sorting results.

[0048] If the user does not provide a specific destination, the system first generates several candidate destinations that meet the constraints based on the semantic parsing results, combined with the geographic semantic knowledge base, the distribution of battery swapping facilities, real-time weather and the user's battery status. Then, it sorts these destinations according to their accessibility, task suitability and battery life safety, and selects the optimal recommended destination as the initial navigation destination.

[0049] Accessibility score is determined based on road network travel distance adaptability (calculating the actual accessible road network distance from the user's current location to the candidate location), road condition smoothness (calculating the route's congestion coefficient), and road suitability (determining whether the route primarily uses roads friendly to electric two-wheelers). Task fit score is determined based on POI type matching and POI quality priority (combining POI reputation ratings, operational status, and facility completeness). Range safety score is determined based on the remaining battery power compared to the candidate location.

[0050] The system calculates a weighted average of accessibility score, task suitability score, and battery life and safety score to obtain a comprehensive score for candidate destinations. These destinations are then ranked based on their comprehensive scores. Alternatively, the candidate destination with the highest comprehensive score can be directly used as the target point for route planning; or, the system can push the top 3 candidate destinations (with visual representation and rating reasons, such as "Recommended A Milk Tea Shop: Ample Battery Life + Closest Distance") to the user, allowing them to choose one as their route planning target.

[0051] Continue to refer to Figure 1 After obtaining the multimodal semantic information, step S102 is executed.

[0052] Step S102: Based on the semantically enhanced spatiotemporal trajectory model, multimodal semantic information is fused with location data, vehicle power data and time data to generate a semantically enhanced spatiotemporal trajectory representation, which includes energy consumption constraints and safety preferences.

[0053] After fully acquiring the user's status (including time, location, and battery level) and navigation intent (such as safety priority, timeliness priority requirements, or complex constraints), all multimodal inputs are uniformly encoded and sent to the core model for reasoning and trajectory generation.

[0054] In some embodiments, step S102, which fuses multimodal semantic information with location data, the battery data of the two-wheeled vehicle, and time data to generate a semantically enhanced spatiotemporal trajectory representation, includes: Step S1021: Generate a riding trajectory sequence with time-series battery information based on multimodal semantic information, location data, battery data of the two-wheeled vehicle, and time data.

[0055] Step S1022: Encode the cycling trajectory sequence with time-series battery information based on the spatiotemporal trajectory encoder.

[0056] Step S1023: Encode the multimodal semantic information based on the semantic feature encoder.

[0057] Step S1024: Through a cross-modal attention mechanism, the encoding of multimodal semantic information is deeply fused with the encoding of the cycling trajectory sequence to generate a semantically enhanced spatiotemporal trajectory representation.

[0058] The semantically enhanced spatiotemporal trajectory model is the core algorithm module of this embodiment. Figure 2 This is a semantically enhanced spatiotemporal trajectory model architecture diagram according to an embodiment of this application, such as... Figure 2 As shown, the spatiotemporal trajectory data with charge mainly includes continuous or discrete time series values ​​such as time series, spatial latitude and longitude series, and battery energy series. These data are converted into corresponding vector sequence representations through a trainable spatiotemporal label encoding and vectorization module. The multimodal semantic information includes text descriptions (such as "passing through a narrow alley at low speed"), visual features (semantic vectors extracted from user-uploaded images by a visual encoder), and sensor context (such as slope, acceleration and deceleration).

[0059] The semantically enhanced spatiotemporal trajectory model takes as input the user's starting point, battery level, and spatiotemporal trajectory data dimensionality information, along with multimodal semantic information (obtained in step S101). Through point-by-point generation, destination arrival determination, and navigation point concatenation, a complete navigation trajectory is generated point by point. In the point-by-point generation process, in addition to the core geographical location, the model can also include road safety and battery status information, thus it can also be used to assess road safety and battery consumption.

[0060] Specifically, the data processing flow for the semantically enhanced spatiotemporal trajectory model is as follows: In step S201, the input module receives two types of input: spatiotemporal trajectory data and multimodal semantic information. Spatiotemporal trajectory data: historical and current time-series data of the user, including latitude and longitude sequences, timestamp sequences, and remaining battery power sequences. This numerical sequence is converted into a high-dimensional vector sequence through a trainable embedding layer. Multimodal semantic information: mainly textual descriptions related to the trajectory or intent (obtained in step S101). This text is converted into a semantic vector sequence through a pre-trained word embedding model.

[0061] Step S202: By splicing data, the two types of vector sequences are spliced ​​or added together to form a preliminary fused sequence representation.

[0062] In step S203, this fused representation is then input into a general large language model with fixed parameters. It should be noted that the role of this large language model here is not to generate text, but rather as a semantic understanding and associator. Utilizing the world knowledge gained during its pre-training, it performs deep modeling of the implicit relationships between trajectory data and semantic data in the fused representation, outputting a semantically enhanced spatiotemporal trajectory representation. This is a crucial step in achieving "semantic enhancement."

[0063] Large language models, through their deep Transformer architecture, can implicitly model the relationships between concatenated vectors of trajectory and semantic data. The model maps trajectory and semantic data to the same representation space. Under a multi-head attention mechanism, the model dynamically calculates the cross-attention weights between trajectory points and semantic concepts, automatically discovering implicit associations between "specific movement patterns and environmental semantics," such as identifying the correspondence between "slow-moving trajectory segments" and "shopping behavior in commercial areas." Through iterative interaction of multiple Transformer blocks, information from the two modalities is deeply fused at the feature level, forming a unified representation that includes spatiotemporal patterns and semantic context.

[0064] Step S204: By using position encoding, sinusoidal position encoding is superimposed on each time step in the semantically enhanced representation above, so that the model can perceive the temporal relationship between trajectory points.

[0065] Step S205: The decoding module consists of N identical Transformer decoder layers stacked together. Each layer includes: a masked multi-head self-attention mechanism to ensure that the prediction at each time step depends only on historical and current information (consistent with autoregressive generation characteristics) and to uncover long-range spatiotemporal dependencies between trajectory points; layer normalization and residual connections to stabilize the training process and accelerate convergence; and a feedforward neural network to perform nonlinear transformations on the features at each time step, enhancing the model's expressive power.

[0066] Step S206, Output module (multi-task head): The high-dimensional features output by the decoder are adjusted in dimension by a linear projection layer and then input in parallel to three independent, simple task-specific heads: displacement task head, which outputs displacement prediction results; safety task head, which outputs safety assessment results; and energy consumption task head, which outputs energy consumption prediction results.

[0067] Before training, a massive amount of spatiotemporal trajectory data of electric two-wheeled vehicle riders with charge levels and corresponding multimodal semantic data are prepared. These are numerical and textual information, respectively, vectorized using a torch embedding module and a word-to-vect semantic vector. The two types of vectorized data are concatenated and input into a fixed large language model module, which fuses the concatenated vectors into a single vector. The fused vector is then labeled with positional encoding to indicate the sequence order. Simultaneously, a decoding module performs a series of attention mechanisms, normalization, and feedforward layers to analyze the data features. The features processed by the decoder are then further adjusted through a linear layer for dimensionality adjustment and output, ultimately producing a displacement task head as the navigation point generation result. Safety and energy consumption task heads can be added based on the training data labels.

[0068] In summary, the training objective of the semantically enhanced spatiotemporal trajectory model is to predict the most reasonable driving point for the user at the next location by combining historical trajectory information of charged components. This process is repeated iteratively, combining... Figure 4 The process guides you to your destination, providing a complete trajectory to that destination.

[0069] The text portion can be converted into a fixed-dimensional vector using a pre-trained word embedding model, while images and audio are processed using CLIP, Whisper, or a dedicated lightweight encoder, respectively. All modal vectors are normalized and then fused through cross-modal attention or a shared projection space to form a preliminary multimodal semantic context. To further enhance the understanding of the connection between semantics and trajectory, the fused overall vector is input again into a general multimodal large language module with fixed parameters (not involved in end-to-end training). Through its powerful context modeling capabilities, an implicit mapping relationship is explicitly established between road spatiotemporal information and rich multimodal semantic information, thereby achieving deep semantic enhancement of the original trajectory data.

[0070] The spatiotemporal trajectory encoder can capture the spatial features of unstructured roads (alleys, pedestrian streets, etc.) (such as short-distance advantages and traffic restrictions). Combined with cross-modal semantic fusion (such as a user wanting to "avoid congested main roads"), it can generate flexible routes that cannot be covered by motor vehicle navigation. The semantic feature encoder accurately extracts the user's core needs (including explicit needs such as "going to the mall" and implicit needs such as "taking the non-motorized vehicle lane"), and the cross-modal attention mechanism aligns these needs with the spatiotemporal and energy consumption features of the trajectory.

[0071] Step S103: Based on the spatiotemporal trajectory representation, generate a navigation path that meets energy consumption constraints and safety preferences, and output the navigation path in a visual form. The semantically enhanced spatiotemporal trajectory model supports the inclusion of unstructured roads in the output path.

[0072] In this embodiment, "unstructured roads" refer to paths that lack the structure of standard motor vehicle lanes but are legally, safely, or habitually accessible to electric two-wheelers. Examples include alleyways, lanes, internal passageways within residential areas, village roads, internal park paths, unpaved roads, and all shortcuts not marked on standard maps but actually passable. Supporting unstructured roads breaks through the strong dependence of traditional navigation algorithms on "structured road networks," enabling the modeling and understanding of a more complex and granular real-world environment.

[0073] The trajectory representation better aligns with the core needs of electric two-wheelers, breaking the limitations of a single data dimension. It deeply couples user semantic needs (such as "avoiding congestion" and "finding shortcuts") with location, battery level, and time data, rather than simply piecing them together. By embedding time-series battery levels, the trajectory sequence inherently carries point-by-point energy consumption status (such as rapid battery consumption when climbing a certain section of road, or stable battery consumption at a constant speed during a certain period). Subsequent path planning can directly determine battery suitability based on the representation, fundamentally avoiding the problem of breakdowns due to insufficient battery power.

[0074] After the trajectory is generated, based on the geographic knowledge base and semantic templates, the machine-readable trajectory data is transformed into a natural language description or contextualized prompts that are easy for users to understand. In this embodiment, navigation results can be conveyed to the user through text-to-speech (TTS), graphical interface, augmented reality (AR) overlay prompts, vibration feedback, light guidance, or multi-channel multimedia. Figure 3 This is an interactive flowchart from multimodal input to multimodal output according to an embodiment of this application. Considering that the obtained complete trajectory data cannot generate good interaction with the user, relevant geographic text information is subsequently queried from pre-organized geographic information based on the trajectory's latitude and longitude, while also combining historical trajectory text fragments to obtain relevant text information. The relevant text information retrieved from the trajectory query is input into a fine-tuned multimodal large language model, and prompt words are used to semantically describe the generated trajectory for the user. Ultimately, the user will receive a complete navigation path and corresponding text description, achieving a complete semantic-to-semantic closed loop.

[0075] In some embodiments, step S103, generating a navigation path that meets energy consumption constraints and safety preferences based on the spatiotemporal trajectory representation, includes: Step S1031: The spatiotemporal trajectory representation is decoded using a Transformer-based decoder.

[0076] Step S1032: The decoded spatiotemporal trajectory representation is input into the safety prediction head, displacement prediction head and power consumption prediction head respectively to obtain the corresponding three output results.

[0077] Step S1033: Combine the three output results to determine the candidate trajectory points for the next position, and form a complete navigation path in an autoregressive manner.

[0078] The spatiotemporal sequence representation enhanced with multimodal semantics is then fed into a Transformer-based decoder module to learn the temporal dependencies and spatial relationships between points in the trajectory sequence. Before entering the decoder, the system first overlays positional order information onto the entire sequence through a positional encoding module, enabling the model to accurately perceive the logical relationships between trajectory points. The sequence with added positional information is then processed sequentially through multiple decoding layers. Each decoding layer consists of two layer normalization modules, two sets of residual connection structures, a multi-head self-attention mechanism, and a feedforward neural network.

[0079] In some embodiments, the safety prediction head is used to perform a binary classification task to predict the road segment safety factor; the displacement prediction head is used to perform a multi-classification task to predict the displacement direction and distance at the next moment; and the power consumption prediction head is used to perform a regression task to estimate the power loss of the road segment.

[0080] The high-dimensional representation, after multi-layer decoding, is finally organized through a linear projection layer and connected to three parallel task heads: the safety prediction head performs a binary classification task to predict the road safety factor, i.e., to determine whether the current road segment is safe; the displacement prediction head performs a multi-class classification task to predict the displacement direction and distance at the next moment, i.e., to divide the displacement space into several discrete categories; and the power consumption prediction head performs a regression task to estimate the power consumption of the current road segment, i.e., to output continuous power consumption values.

[0081] It should be noted that, in this embodiment, the spatiotemporal trajectory encoding or path generation module can use not only Transformer, but also Graph Neural Network (GNN), Recurrent Neural Network (RNN), Diffusion Model, State Space Model (SSM) or any combination thereof, as long as it can output feasible path or destination suggestions that meet multidimensional constraints (such as safety, endurance feasibility, traffic compliance, comfort or timeliness) based on enhanced trajectory features generated by the fusion of multimodal semantic information.

[0082] In some embodiments, the method further includes: Based on the output of the power consumption prediction head, the estimated remaining power is calculated; If the estimated remaining power is less than a preset power threshold, the nearest energy replenishment point is searched based on the current location. Inserting energy supply points as temporary destinations into the current path planning sequence triggers path replanning with the temporary destinations as the target.

[0083] If the system detects that the cumulative power loss has reduced the remaining power to a preset threshold (e.g., 30%) during the trajectory generation process, it will automatically search for the nearest battery swapping station that meets the semantic conditions around the current location and temporarily insert it into the destination queue, triggering a new round of route replanning until the generated trajectory meets the range safety requirements.

[0084] When all original or recommended destinations are successfully reached and the battery status is normal, the entire trajectory generation process ends. If there are still unfinished destinations, the generated trajectory sequence is concatenated with the newly predicted waypoints and returned to the inference input to continue the trajectory generation steps until all destinations are covered, thus completing the complete navigation inference process. Figure 4 This is a flowchart of path generation reasoning based on a multimodal semantic enhancement spatiotemporal trajectory model according to an embodiment of this application.

[0085] Through the above steps, user input information is parsed based on a pre-built multimodal large language model, which is compatible with various input forms such as voice, text, and images. The semantically enhanced spatiotemporal trajectory model supports outputting paths that include unstructured roads, allowing electric two-wheelers to travel on routes that motor vehicles cannot use, such as alleys and pedestrian streets. This solves the problem that existing navigation systems are not compatible with electric two-wheelers, enabling shorter-distance and more flexible navigation, avoiding congested main roads, and improving travel efficiency.

[0086] Whether you're a professional delivery rider or a daily commuter, you can get efficient, safe, and energy-saving personalized route suggestions in scenarios with a known destination, no specific destination, or restrictive descriptions. New riders or users entering unfamiliar areas can quickly obtain route planning at a level close to that of experienced riders, and receive intelligent guidance when lacking a clear goal, significantly improving task completion efficiency or daily travel experience. At the same time, the system integrates battery status assessment, semantic matching of battery swapping / charging points, and route collaborative optimization, effectively alleviating the common "range anxiety," and achieving a more intuitive and easy-to-use navigation experience through natural language interaction.

[0087] Optimizing routes reduces unnecessary detours, hasty lane changes, and passage through high-risk sections, which can systematically reduce the traffic accident rate and improve the safety level of urban slow-moving traffic. At the same time, more accurate destination matching and more efficient route planning mean lower energy consumption and carbon emissions, reducing resource waste caused by blind searching or inefficient scheduling, and contributing to the construction of a green and low-carbon transportation system.

[0088] In some embodiments, the method further includes: Obtain key waypoints in the navigation path, match the key waypoints with the geographic knowledge base, and obtain traffic data related to the navigation path; Traffic condition alerts are generated and output based on traffic data.

[0089] To enhance the understandability and usability of navigation results, before returning the trajectory to the user, key waypoints in the generated path are matched with a built-in geographic knowledge base, and structured knowledge fragments related to the road segment (such as slope information, historical accident hotspots, non-motorized vehicle lane width, battery swapping station locations, weather impact levels, etc.) are extracted to generate targeted road condition prompts. Finally, a complete, visualized trajectory path is returned to the user, along with corresponding natural language semantic descriptions or multimedia feedback (such as voice broadcasts and AR overlay prompts). This not only recommends the optimal driving route but also proactively prompts necessary safety information, energy consumption nodes, and environmental characteristics along the path, thereby ensuring that users can reach their destination quickly, safely, and efficiently.

[0090] The method described in this embodiment is not only applicable to on-demand delivery riders, but also to various electric two-wheeled vehicle travel scenarios such as daily commuting, leisure riding, shared electric bicycle dispatching, campus / industrial park logistics, and emergency inspections. Its navigation target can be flexibly configured to minimize time, energy consumption, safety, traffic lights, and provide optimal shade / rain shelter. Furthermore, it supports automatically generating recommended targets and planning the entire route based on semantic intent when there is no clear destination.

[0091] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0092] This embodiment also provides a navigation system for electric two-wheeled vehicles based on multimodal semantic enhancement. This system is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0093] Figure 5 This is a structural block diagram of an electric two-wheeled vehicle navigation system based on multimodal semantic enhancement according to an embodiment of this application, such as... Figure 5 As shown, the system includes: The semantic parsing module 51 is used to obtain user input information and parse the input information based on a pre-built multimodal large language model to obtain multimodal semantic information.

[0094] The spatiotemporal trajectory representation generation module 52 is used to fuse multimodal semantic information with location data, vehicle power data and time data based on the semantically enhanced spatiotemporal trajectory model to generate a semantically enhanced spatiotemporal trajectory representation, which includes energy consumption constraints and safety preferences.

[0095] The path generation module 53 is used to generate a navigation path that meets energy consumption constraints and safety preferences based on the spatiotemporal trajectory representation, and output the navigation path in a visual form. The semantically enhanced spatiotemporal trajectory model supports the inclusion of unstructured roads in the output path.

[0096] In some embodiments, the multimodal semantic information includes the target point for path planning; the semantic parsing module 51 includes: The parsing module is used to parse the input information based on a multimodal large language model to obtain semantic parsing results.

[0097] The first target point determination module is used to use the destination as the target point for path planning when the semantic parsing result contains a clear destination.

[0098] The second target point determination module is used to determine the target point for route planning based on the semantic parsing results, the geographic semantic knowledge base, and power data when the semantic parsing results do not contain a clear destination.

[0099] In some embodiments, the second target point determination module includes: The candidate set generation module is used to obtain battery swapping facility distribution data from the geographic semantic knowledge base, and determine candidate destinations based on semantic parsing results, geographic semantic knowledge base, battery swapping facility distribution, weather data, and electricity data.

[0100] The target point determination module is used to sort candidate destinations based on reachability, task suitability, and endurance safety, and determine the target point for route planning from the candidate destinations according to the sorting results.

[0101] In some embodiments, the spatiotemporal trajectory representation generation module 52 includes: The trajectory sequence generation module is used to generate a riding trajectory sequence with time-series battery information based on multimodal semantic information, location data, battery data of the two-wheeled vehicle, and time data.

[0102] The spatiotemporal coding module is used to encode cycling trajectory sequences with time-series battery information based on the spatiotemporal trajectory encoder.

[0103] The semantic encoding module is used to encode multimodal semantic information based on the semantic feature encoder.

[0104] The fusion module is used to deeply fuse the encoding of multimodal semantic information with the encoding of cycling trajectory sequence through a cross-modal attention mechanism to generate a semantically enhanced spatiotemporal trajectory representation.

[0105] In some embodiments, the path generation module 53 includes: The decoding module is used to decode the spatiotemporal trajectory representation through a Transformer-based decoder.

[0106] The prediction module is used to input the decoded spatiotemporal trajectory representation into the safety prediction head, displacement prediction head, and power consumption prediction head respectively, and obtain three corresponding output results.

[0107] The candidate point determination module is used to integrate the three output results, determine the candidate trajectory points for the next position, and form a complete navigation path in an autoregressive manner.

[0108] In some embodiments, the safety prediction head is used to perform a binary classification task to predict the safety factor of a road segment; The displacement prediction head is used to perform multi-class classification tasks to predict the displacement direction and distance at the next moment.

[0109] The power consumption prediction head is used to perform regression tasks to estimate power loss in road sections.

[0110] In some embodiments, the system further includes: The power consumption prediction module is used to calculate the estimated remaining power based on the output of the power consumption prediction head.

[0111] The replenishment point selection module is used to search for the nearest energy replenishment point based on the current location when the estimated remaining power is less than a preset power threshold.

[0112] The route replanning module is used to insert energy supply points as temporary destinations into the current route planning sequence, triggering route replanning with the temporary destination as the target.

[0113] In some embodiments, the system further includes: The traffic acquisition module is used to obtain key waypoints in the navigation path, match the key waypoints with the geographic knowledge base, and obtain traffic data related to the navigation path.

[0114] The traffic alert module is used to generate and output traffic alert information based on traffic data.

[0115] The system described above parses user input information based on a pre-built multimodal large language model, and is compatible with various input formats such as voice, text, and images. The semantically enhanced spatiotemporal trajectory model supports outputting paths that include unstructured roads, allowing electric two-wheelers to travel on routes that motor vehicles cannot access, such as alleys and pedestrian streets. This solves the problem of existing navigation systems being unsuitable for electric two-wheelers, enabling shorter-distance and more flexible navigation, avoiding congested main roads, and improving travel efficiency.

[0116] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0117] This embodiment also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0118] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0119] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program: S1: Obtain user input information, parse the input information based on a pre-built multimodal large language model, and obtain multimodal semantic information.

[0120] S2, based on a semantically enhanced spatiotemporal trajectory model, integrates multimodal semantic information with location data, vehicle battery data, and time data to generate a semantically enhanced spatiotemporal trajectory representation, which includes energy consumption constraints and safety preferences.

[0121] S3 generates navigation paths that meet energy consumption constraints and safety preferences based on spatiotemporal trajectory representations, and outputs the navigation paths in a visual form. The semantically enhanced spatiotemporal trajectory model supports the inclusion of unstructured roads in the output paths.

[0122] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0123] In one embodiment, Figure 6 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application, such as... Figure 6 As shown, an electronic device is provided, which can be a server, and its internal structure diagram can be as follows. Figure 6 As shown, the electronic device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a navigation method for an electric two-wheeled vehicle based on multimodal semantic enhancement.

[0124] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0125] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0126] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0127] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A navigation method for electric two-wheeled vehicles based on multimodal semantic enhancement, characterized in that, The method includes: The user's input information is obtained, and the input information is parsed based on a preset multimodal large language model to obtain multimodal semantic information; Based on the semantically enhanced spatiotemporal trajectory model, the multimodal semantic information is fused with location data, vehicle battery data and time data to generate a semantically enhanced spatiotemporal trajectory representation, which includes energy consumption constraints and safety preferences. Based on the spatiotemporal trajectory representation, a navigation path that conforms to the energy consumption constraints and the safety preferences is generated and output in a visual form. The semantically enhanced spatiotemporal trajectory model supports the inclusion of unstructured roads in the output path.

2. The method according to claim 1, characterized in that, The multimodal semantic information includes the target point for path planning; The input information is parsed based on a pre-built multimodal large language model to obtain multimodal semantic information, including: Based on the multimodal large language model, the input information is parsed to obtain semantic parsing results; If the semantic parsing result contains a clear destination, the destination will be used as the target point for route planning. If the semantic parsing result does not contain a specific destination, the target point for route planning is determined based on the semantic parsing result, the geographic semantic knowledge base, and the power consumption data.

3. The method according to claim 2, characterized in that, The step of determining the target point for route planning based on the semantic parsing results, the geographic semantic knowledge base, and the electricity data includes: Obtain the distribution data of battery swapping facilities from the geographic semantic knowledge base; Based on the semantic parsing results, the geographic semantic knowledge base, the distribution of the battery swapping facilities, weather data, and the electricity data, candidate destinations are determined. The candidate destinations are ranked based on accessibility, task suitability, and range safety. The target point for the route planning is then determined from the candidate destinations based on the ranking results.

4. The method according to claim 1, characterized in that, The process of fusing the multimodal semantic information with location data, the battery data of the two-wheeled vehicle, and time data to generate a semantically enhanced spatiotemporal trajectory representation includes: Based on the multimodal semantic information, the location data, the battery data of the two-wheeled vehicle, and the time data, a riding trajectory sequence with time-series battery information is generated; Encode the cycling trajectory sequence with time-series battery information based on the spatiotemporal trajectory encoder; The multimodal semantic information is encoded based on a semantic feature encoder; By using a cross-modal attention mechanism, the encoding of the multimodal semantic information is deeply fused with the encoding of the cycling trajectory sequence to generate the semantically enhanced spatiotemporal trajectory representation.

5. The method according to claim 1, characterized in that, The step of generating a navigation path that conforms to the energy consumption constraints and the safety preferences based on the spatiotemporal trajectory representation includes: The spatiotemporal trajectory representation is decoded using a Transformer-based decoder. The decoded spatiotemporal trajectory representation is input into the safety prediction head, displacement prediction head, and power consumption prediction head respectively, and three corresponding output results are obtained. By combining the three output results, candidate trajectory points for the next position are determined, and a complete navigation path is formed in an autoregressive manner.

6. The method according to claim 5, characterized in that, The safety prediction head is used to perform a binary classification task to predict the safety factor of road segments; The displacement prediction head is used to perform a multi-classification task to predict the displacement direction and distance at the next moment; The power consumption prediction head is used to perform regression tasks to estimate power loss in road sections.

7. The method according to claim 5, characterized in that, The method further includes: Based on the output of the power consumption prediction head, the estimated remaining power is calculated; If the estimated remaining power is less than a preset power threshold, the nearest energy replenishment point is retrieved based on the current location. Insert the energy replenishment point as a temporary destination into the current path planning sequence, triggering path replanning with the temporary destination as the target.

8. The method according to claim 1, characterized in that, The method further includes: Obtain key waypoints in the navigation path, match the key waypoints with a geographic knowledge base, and obtain traffic data related to the navigation path; Traffic condition alerts are generated based on the traffic condition data, and then output.

9. A navigation system for an electric two-wheeled vehicle based on multimodal semantic enhancement, characterized in that, The system includes: The semantic parsing module is used to acquire user input information and parse the input information based on a preset multimodal large language model to obtain multimodal semantic information. The spatiotemporal trajectory representation generation module is used to fuse the multimodal semantic information with location data, vehicle power data and time data based on the semantically enhanced spatiotemporal trajectory model to generate a semantically enhanced spatiotemporal trajectory representation, which includes energy consumption constraints and safety preferences. The path generation module is used to generate a navigation path that conforms to the energy consumption constraints and the safety preferences based on the spatiotemporal trajectory representation, and output the navigation path in a visual form. The semantically enhanced spatiotemporal trajectory model supports the inclusion of unstructured roads in the output path.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the electric two-wheeler navigation method based on multimodal semantic enhancement as described in any one of claims 1 to 8.

Images