Navigation instruction generation method and device, equipment and medium

Abstract

The invention relates to the technical field of natural language generation, and discloses a navigation instruction generation method and device, equipment and a medium. The method comprises the following steps: acquiring a navigation track, user motion parameters, and a color image and a depth image of a current navigation position; sensing regional attribute features of the current navigation position based on the color image; sensing room attribute features of the current navigation position based on the color image; sensing object attribute characteristics of an object in a target area based on the color image and the depth image, wherein the target area is a range extending by a preset threshold distance by taking the current navigation position as a center; sensing motion attribute characteristics of the user based on a video stream formed by the color images and the user motion parameters; and generating a navigation instruction based on the navigation track, the region attribute feature, the room attribute feature, the object attribute feature and the motion attribute feature to obtain a target navigation instruction. According to the embodiment of the invention, the key information of the navigation track can be perceived hierarchically so as to generate various types of navigation instructions.

Description

Technical Field

[0001] This application relates to the field of natural language generation technology, and in particular to a method, apparatus, device and medium for generating navigation instructions. Background Technology

[0002] Navigation command generation is an important research area in command navigation. The generated navigation commands can be used to evaluate existing command navigation models, or as training data to further improve the performance of command navigation models.

[0003] Currently, navigation command generation mainly relies on manual annotation. Some works attempt to explore training task-specific navigation command generation models based on existing navigation command data. These works require complete human involvement or rely on human annotation to even begin the process. Furthermore, navigation command generation models are limited by training data and often can only generate navigation commands of a specific form and style, making them unable to adequately meet the needs of model training and evaluation. Summary of the Invention

[0004] The purpose of this application is to provide a navigation instruction generation method, apparatus, device, and medium that can hierarchically perceive key information of the navigation trajectory to generate various types of navigation instructions.

[0005] This application provides a navigation instruction generation method, including: Acquire navigation trajectory, user motion parameters, and color and depth images of the current navigation location; Based on the color image, the regional attribute features of the current navigation location are perceived; Based on the color image, the room attribute features of the current navigation location are perceived; Based on the color image and the depth image, the object attribute features of objects within the target area are perceived; the target area is a range extending from the current navigation position to a preset threshold distance. The user's motion attribute characteristics are perceived based on the video stream composed of the color images and the user's motion parameters; Based on the navigation trajectory, the area attribute features, the room attribute features, the object attribute features, and the motion attribute features, navigation instructions are generated to obtain target navigation instructions.

[0006] In some embodiments, the navigation instruction generation method further includes: Obtain navigation tasks; A path that conforms to the navigation task is generated using a path planning method, thus obtaining the navigation trajectory.

[0007] In some embodiments, perceiving the regional attribute features of the current navigation location based on the color image includes: The color images acquired from multiple angles are stitched together to obtain a panoramic image; The panoramic image is mapped onto a plane using an equidistant cylindrical projection method to obtain a projected image. The visual attribute features in the projected image are mapped to a preset regional attribute semantic space to generate the regional attribute features based on a preset environmental function description paradigm.

[0008] In some embodiments, perceiving room attribute features of the current navigation location based on the color image includes: The color images acquired from multiple angles are stitched together to obtain a stitched image whose angles are the same as those of the human eye. The illumination and hue features of the stitched image are normalized to obtain a normalized image; Extract the visual features of the furniture from the normalized image; The visual features of furniture in the normalized image are mapped to a preset room attribute semantic space to generate the room attribute features based on the co-occurrence features of furniture in the normalized image.

[0009] In some embodiments, the step of perceiving object attribute features of objects within the target region based on the color image and the depth image includes: The color image and the depth image are fused within the target area to obtain a fused image; Extract the visual features of objects in the fused image; The visual features of objects in the fused image are mapped to a preset object attribute semantic space, and the visual features of objects in the fused image are aligned with the corresponding feature text based on an attention mechanism to generate the object attribute features.

[0010] In some embodiments, perceiving the user's motion attribute features based on the video stream composed of the color images and the user motion parameters includes: Extract the inter-frame variation region features of the video stream; The inter-frame change region features and the user motion parameters are mapped to a preset motion attribute semantic space to generate the motion attribute features based on the co-occurrence features of the inter-frame change region features and the user motion parameters.

[0011] In some embodiments, the generation of navigation instructions based on the navigation trajectory, the area attribute features, the room attribute features, the object attribute features, and the motion attribute features includes: Generate an action sequence that conforms to the navigation trajectory based on the area attribute features, room attribute features, object attribute features, and motion attribute features; Generate corresponding natural language based on the action sequence; The natural language is then subjected to context alignment processing to obtain the target navigation instructions.

[0012] This application embodiment also provides a navigation instruction generation device, including: The first module is used to acquire navigation trajectory, user motion parameters, and color and depth images of the current navigation location; The second module is used to perceive the regional attribute features of the current navigation location based on the color image; The third module is used to perceive the room attribute features of the current navigation location based on the color image; The fourth module is used to perceive the object attribute features of objects within the target area based on the color image and the depth image; the target area is a range extending from the current navigation position to a preset threshold distance. The fifth module is used to perceive the user's motion attribute characteristics based on the video stream composed of the color images and the user's motion parameters; The sixth module is used to generate navigation instructions based on the navigation trajectory, the area attribute features, the room attribute features, the object attribute features, and the motion attribute features, to obtain target navigation instructions.

[0013] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described navigation instruction generation method.

[0014] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described navigation instruction generation method.

[0015] The beneficial effects of this application are as follows: By perceiving the regional attribute features and room attribute features of the current navigation location, the object attribute features of objects near the current navigation location, and the motion attribute features of the user, and combining the navigation trajectory, regional attribute features, room attribute features, object attribute features, and motion attribute features, the natural language generation module transforms them into target navigation instructions containing orientation indicators and environmental references, which significantly improves the adaptability of instruction generation. Especially in complex indoor scenes, it can accurately identify the spatial position of temporary obstacles, avoid instruction errors caused by changes in object position, and can hierarchically perceive key information of the navigation trajectory to generate various types of navigation instructions. Attached Figure Description

[0016] Figure 1 This is a diagram illustrating the application environment of the navigation instruction generation method provided in the embodiments of this application.

[0017] Figure 2 This is a flowchart of the navigation instruction generation method provided in the embodiments of this application.

[0018] Figure 3 This is a flowchart of the specific method for step S202 provided in the embodiments of this application.

[0019] Figure 4 This is a flowchart of the specific method of step S203 provided in the embodiments of this application.

[0020] Figure 5 This is a flowchart of the specific method for step S204 provided in the embodiments of this application.

[0021] Figure 6 This is a flowchart of the specific method of step S205 provided in the embodiments of this application.

[0022] Figure 7 This is a flowchart of the specific method of step S206 provided in the embodiments of this application.

[0023] Figure 8 This is a schematic diagram of the navigation instruction generation device provided in the embodiments of this application.

[0024] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0026] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and drawings are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0028] In related technologies, navigation command generation mainly relies on manual annotation or task-specific trained models, which suffers from high labor costs and limited command formats. Existing methods struggle to adapt to dynamic environmental changes and cannot effectively integrate multimodal perception information, resulting in generated navigation commands lacking spatial semantic understanding and prone to path description deviations in complex indoor scenes. For example, in smart home environments, when users need to navigate around temporarily placed obstacles, traditional methods cannot identify object attributes in real time, leading to mismatches between generated turning commands and the actual situation.

[0029] To address these issues, researchers found that manually annotated navigation commands struggle to cover all environmental variables, while single-sensor data cannot accurately reflect spatial structural features. Analysis revealed that insufficient semantic understanding of environmental functional areas is the primary cause of command deviations. Further research showed that fusing depth information and visual features can improve object recognition accuracy, and correlation analysis between user motion states and video stream changes can predict action intentions. Based on this, a spatial cognitive model is proposed to be constructed through the fusion of multi-dimensional environmental features, enabling the automated generation of navigation commands.

[0030] Based on this, embodiments of this application provide a navigation instruction generation method, apparatus, device, and medium. By fusing multi-dimensional environmental features to construct a spatial cognition model, the automated generation of navigation instructions is achieved. It can hierarchically perceive key information of the navigation trajectory and generate various types of navigation instructions.

[0031] The navigation instruction generation method provided in this application can be executed by a computer device, which can be a terminal device or a server. Terminal devices include, but are not limited to, mobile phones, computers, smart home appliances, vehicle terminals, and aircraft. The server can be a standalone physical server, a server cluster consisting of multiple physical servers, a distributed system, or a cloud server. Furthermore, the information, data, and signals involved in this application's embodiments are all authorized by the relevant parties or have received full authorization from all parties, and the collection, use, and processing of related data comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0032] Figure 1 This diagram illustrates the application environment of the navigation instruction generation method provided in this embodiment. (See attached diagram.) Figure 1This method is applied to a navigation command generation system. The system includes a terminal 110 and a server 120. The terminal 110 and server 120 are connected via a network. The terminal 110 can be a desktop terminal or a mobile terminal; the mobile terminal can be at least one of a mobile phone, tablet, or laptop. The server 120 can be a standalone server or a server cluster consisting of multiple servers. The terminal 110 sends navigation trajectory, user motion parameters, and a color image and depth image of the current navigation location to the server 120. The server 120 acquires the navigation trajectory, user motion parameters, and the color and depth images of the current navigation location. Based on the color images, it perceives the regional attribute features of the current navigation location, the room attribute features of the current navigation location, and the object attribute features of objects within the target area. Based on the video stream composed of color images and the user's motion parameters, it perceives the user's motion attribute features. Based on the navigation trajectory, regional attribute features, room attribute features, object attribute features, and motion attribute features, it generates navigation commands to obtain the target navigation command. The target area is a range extending from the current navigation location at a preset threshold distance.

[0033] It should be understood that Figure 1 The application scenarios shown are merely examples. In practical applications, the navigation instruction generation method provided in this application embodiment can also be applied to other scenarios. For example, the above-described navigation instruction generation method can be directly applied to terminal 110. Terminal 110 is used to acquire navigation trajectory, user motion parameters, color image and depth image of the current navigation position, perceive regional attribute features of the current navigation position based on the color image, perceive room attribute features of the current navigation position based on the color image, perceive object attribute features of objects within the target area based on the color image and depth image, perceive user motion attribute features based on the video stream composed of color images and user motion parameters, and generate navigation instructions based on navigation trajectory, regional attribute features, room attribute features, object attribute features and motion attribute features to obtain target navigation instructions.

[0034] To facilitate understanding of the navigation instruction generation method provided in this application embodiment, the following example uses server 120 as the execution subject to illustrate the application scenario of the navigation instruction generation method.

[0035] Figure 2 This is a flowchart of the navigation instruction generation method provided in an embodiment of this application. See also... Figure 2 In one embodiment, the method includes, but is not limited to, steps S201 to S206.

[0036] Step S201: Obtain the navigation trajectory, user motion parameters, color image and depth image of the current navigation location.

[0037] A navigation trajectory refers to the path a user plans or actually walks from the starting point to the destination. A navigation trajectory includes not only the geometry of the path but also time and attitude information along that path.

[0038] User motion parameters refer to the physical quantities that represent the user's motion state as they walk along a navigation trajectory. These parameters change in real time and are used to accurately understand and control motion. User motion parameters can include linear velocity, angular velocity, acceleration, and displacement distance, recorded by an inertial measurement unit (IMU), and are the basis for real-time positioning and path estimation.

[0039] The current navigation location refers to the user's current location while walking along the navigation path.

[0040] The color image of the current navigation location refers to the image captured from the visual view of the current navigation location. The depth image of the current navigation location refers to the image that evaluates the distance between objects and the camera (which can be the current navigation location). The value of each pixel in the depth image is not color, but rather the distance from the scene point corresponding to that pixel to the camera. The depth image directly provides three-dimensional geometric information of the environment.

[0041] As examples, acquiring navigation trajectory, user motion parameters, color image and depth image of the current navigation location can be achieved by acquiring the start and end points uploaded by the user to generate the corresponding navigation trajectory, acquiring the corresponding color image from a camera deployed at the current navigation location and generating the corresponding depth image through corresponding image processing of the color image, and acquiring user motion parameters obtained by acquiring the user's real-time motion parameters from a motion parameter acquisition device.

[0042] Step S202: Perceive the regional attribute features of the current navigation location based on the color image.

[0043] It is understandable that regional attribute features refer to the semantic description of spatial functional zoning. Specifically, panoramic stitching and equidistant columnar projection technology can be used to convert multi-angle color images into planar projections, and functional areas such as corridors and living rooms can be identified through preset environmental functional description paradigms.

[0044] As some examples, perceiving the regional attribute features of the current navigation location can be achieved by inputting a color image into a pre-defined regional attribute perception network. Based on the mapping relationship between the visual attribute features in the color image and pre-defined regional attribute description text, the network determines the corresponding regional attribute description text and regional function description text, thus obtaining the corresponding regional attribute features. For instance, when the current navigation location is identified as an entrance based on the visual attribute features in the color image, descriptive text or feature vectors such as "entrance," "welcoming guests," and / or "changing shoes" are output according to the mapping relationship.

[0045] Preferably, the region attribute perception network can be the Qwen2.5-VL visual language model.

[0046] Step S203: Perceive the room attribute features of the current navigation location based on the color image.

[0047] It is understandable that room attribute features refer to the identification of the functional attributes of an interior space. Specifically, this can be achieved by stitching together images that conform to the human visual field, extracting furniture co-occurrence patterns, and matching them with a preset semantic space. For example, it can identify the combination features of beds and wardrobes in a bedroom.

[0048] As examples, perceiving the room attribute features of the current navigation location can be achieved by inputting a color image into a pre-defined room attribute perception network. The network then determines the corresponding room attribute description text based on the mapping relationship between the visual features of furniture in the color image and pre-defined room attribute description text, thus obtaining the corresponding room attribute features. For instance, if the visual features of furniture in the color image indicate that the current navigation location contains furniture such as a stove, cabinets, and a refrigerator, the current navigation location is identified as a kitchen, and the description text or feature vector for "kitchen" is output based on the mapping relationship.

[0049] Preferably, the room attribute perception network can be the Qwen2.5-VL visual language model.

[0050] Step S204: Perceive the object attribute features of objects within the target area based on color and depth images.

[0051] It can be understood that object attribute features refer to the identification results of obstacles or landmarks within the target area. Specifically, color image and depth image fusion technology is used to construct three-dimensional spatial information, and attention mechanism is used to align the visual features of objects with semantic descriptions, such as identifying the height and position information of a chair.

[0052] The target area is the range extending from the current navigation position to a preset threshold distance. In other words, the target area refers to the range extending from the current navigation position to a preset threshold distance, such as a circular area with a radius of 2 meters. This can be achieved by setting a distance threshold and combining it with coordinate system transformation, and is used to limit the local spatial range for object attribute perception.

[0053] As some examples, perceiving the object attribute features of objects within a target area can be achieved by inputting color images and depth images into a preset object attribute perception network, and determining the corresponding object attribute description text based on the mapping relationship between the visual features (such as color, shape, material, style, etc.) of objects within a preset distance in the color image and the preset object attribute description text, thereby obtaining the corresponding object attribute features.

[0054] Preferably, the room attribute perception network can be the Qwen2.5-VL visual language model.

[0055] Step S205: Based on the video stream composed of color images and user motion parameters, perceive the user's motion attribute characteristics.

[0056] As can be understood, motion attribute features refer to the dynamic analysis of a user's action state. Specifically, this involves detecting changes in the area between video stream frames and combining motion sensor data to identify the trends in the user's walking speed and direction.

[0057] As examples, perceiving user motion attribute features can be achieved by sorting color images of the current navigation location according to their acquisition time and generating corresponding video streams. After aligning the video streams with user motion parameters, the data is input into a pre-defined motion attribute perception network. The corresponding motion attribute description text is determined based on the mapping relationship between the visual change features of two adjacent color images in the video stream, the user motion parameters, and pre-defined motion attribute description text, thus obtaining the corresponding motion attribute features. For example, based on visual change features in two adjacent color images in the video stream, such as changes in door frame position, the appearance of stairs, or sudden changes in spatial brightness, the network outputs descriptive text or feature vectors such as "turn left" or "pass through the doorway" according to the mapping relationship.

[0058] Preferably, the room attribute perception network can be the Qwen2.5-VL visual language model.

[0059] Step S206: Based on the navigation trajectory, area attribute features, room attribute features, object attribute features and motion attribute features, navigation instructions are generated to obtain target navigation instructions.

[0060] As some examples, navigation instruction generation can be achieved by inputting navigation trajectory, regional attribute features, room attribute features, object attribute features, and motion attribute features into a preset navigation instruction generation network. The regional attribute features, room attribute features, object attribute features, and motion attribute features are then concatenated into a spatially continuous natural language using a context-guided method. The concatenated natural language is then adjusted to conform to the navigation trajectory and the current navigation location to obtain the target navigation instruction.

[0061] Preferably, the navigation instruction generation network can be a Grok 4 language model.

[0062] The navigation instruction generation method provided in this application constructs a panoramic view of the current navigation location using multi-angle color images while the user walks along the navigation trajectory. It extracts spatial layout features using projection transformation to extract regional and room attribute features of the current area. Simultaneously, it processes the depth image of the current navigation location to obtain the three-dimensional coordinates of objects. It combines visual features to determine the passability of obstacles to extract object attribute features of objects within the target area. It also combines video stream analysis and user motion parameters to detect deviations between the user's movement trajectory and the predetermined path, and perceives user gait and turning data to extract user motion attribute features. Combining the navigation trajectory, regional attribute features, room attribute features, object attribute features, and motion attribute features, it encodes multi-dimensional information such as regional attribute semantics, room attribute semantics, object distribution semantics, and user behavior semantics into an action sequence. This sequence is then converted into a target navigation instruction containing orientation indicators and environmental references by a natural language generation module. For example, when the user is detected to have deviated from the predetermined path, the method combines the features of the right wall and the distance to obstacles in front to generate a target navigation instruction such as "Please proceed along the left wall and go around the coffee table two meters away." Therefore, by perceiving the regional and room attribute characteristics of the current navigation location, the object attribute characteristics of objects near the current navigation location, and the user's motion attribute characteristics, and combining the navigation trajectory, regional attribute characteristics, room attribute characteristics, object attribute characteristics, and motion attribute characteristics, the natural language generation module transforms them into target navigation instructions containing orientation indicators and environmental references, significantly improving the adaptability of instruction generation. Especially in complex indoor scenes, it can accurately identify the spatial location of temporary obstacles, avoid instruction errors caused by changes in object position, and can hierarchically perceive key information of the navigation trajectory to generate various types of navigation instructions.

[0063] In some embodiments, the navigation instruction generation method further includes: obtaining a navigation task; generating a path that conforms to the navigation task using a path planning method, thereby obtaining a navigation trajectory.

[0064] Navigation tasks refer to the target location a user needs to reach or the type of task they need to complete. This can be achieved through voice recognition or text input, such as recognizing a user's need to go to the kitchen or find a specific item through voice commands. Path planning methods are algorithms that calculate feasible routes based on environmental spatial information. These can be implemented using algorithms like A* or Dijkstra's algorithm, generating the optimal path by analyzing obstacle distribution and feasible areas in the environmental map.

[0065] After acquiring the navigation task, the system determines the destination location or task type based on the task objective, such as the user needing to go to the bedroom or move items. Subsequently, a feasible path from the starting point to the destination is generated using a path planning algorithm combined with environmental map information; for example, in an environment with multiple obstacles, the shortest path around the obstacles is planned. The generated navigation trajectory includes a sequence of path points and movement direction information, such as representing the path as a sequence of coordinate points. This trajectory is then input into the navigation instruction generation module for further processing. Thus, by introducing a path planning algorithm, a more adaptable navigation trajectory can be automatically generated based on real-time task requirements, such as dynamically adjusting the path in complex indoor environments to avoid temporary obstacles, thereby reducing reliance on manual annotation. When a user needs to complete a multi-objective navigation task, the system can automatically plan the optimal path sequence based on task priority and generate corresponding navigation instructions, improving the flexibility and usability of the navigation system.

[0066] Figure 3 This is a flowchart illustrating the specific method of step S202 provided in the embodiments of this application. See also... Figure 3 In one embodiment, the method includes, but is not limited to, steps S301 to S303.

[0067] Step S301: Perform panoramic stitching on the color images acquired from multiple angles to obtain a panoramic image.

[0068] It is understandable that panoramic stitching refers to forming a complete image covering a 360-degree field of view by geometric transformation and pixel fusion of two-dimensional images acquired from multiple perspectives. Specifically, it can be achieved by using feature point matching algorithms combined with image registration technology, such as using SIFT feature detectors to extract key points and then performing perspective transformation fusion.

[0069] Step S302: The panoramic image is mapped onto a plane using the equidistant cylindrical projection method to obtain the projected image.

[0070] It is understandable that the equidistant cylindrical projection method refers to the projection method of converting a panoramic image in a spherical coordinate system into a two-dimensional planar coordinate system. Specifically, it can be implemented by using a pixel mapping algorithm with equal angular intervals of latitude and longitude. For example, the spherical longitude is mapped to the horizontal coordinate of the plane and the latitude is mapped to the vertical coordinate.

[0071] Step S303: Map the visual attribute features in the projected image to a preset regional attribute semantic space to generate regional attribute features based on a preset environmental function description paradigm.

[0072] It can be understood that the semantic space of regional attributes refers to a vector space composed of predefined environmental function categories. Specifically, this can be achieved by aligning the feature vectors output by the semantic segmentation model with the embedding space of the text labels. For example, contrastive learning can be used to map image features to a vector space containing semantic categories such as "corridor," "living room," and "kitchen." The environmental function description paradigm refers to a rule template used to structurally describe the uses of spatial regions. Specifically, this can be achieved using a semantic reasoning framework based on knowledge graphs. For example, functional attributes such as "dining area" or "work area" can be inferred based on the categories of objects present in the region and their spatial layout relationships.

[0073] When perceiving the regional attribute features of the current navigation location, the acquisition of multi-angle color images and panoramic stitching can eliminate the visual limitations of a single viewpoint, fully covering the 360° spatial information surrounding the current navigation location. Isometric cylindrical projection converts the spherical image into a planar unfolded map, allowing subsequent processing to directly apply planar image analysis methods. Visual attribute features in the projected image are extracted through a pre-trained regional attribute perception network. Based on the mapping relationship between visual attribute features in the color image and preset regional attribute description text, the corresponding regional attribute description text and regional function description text are determined, such as identifying visual elements like floor materials, wall decorations, or fixed facilities. After these features are mapped to the regional attribute semantic space, combined with logical rules defined in the environmental function description paradigm, such as "areas with dining tables and chairs are classified as dining areas," regional attribute features that conform to human cognitive habits are automatically generated. Thus, through panoramic image stitching and semantic space mapping, visual features strongly correlated with scene functions can be automatically extracted, such as distinguishing corridors from rooms based on floor material texture, or identifying living rooms from entryways based on wall decoration features. It achieves automated identification of navigation environment area attributes, solving the problems of low efficiency of manual annotation and insufficient flexibility of template generation. By integrating multi-view image information and semantic reasoning mechanism, it can accurately identify the functional attributes of different areas in complex scenes, such as distinguishing between work areas and rest areas in open layouts, thereby providing reliable environmental semantic information support for navigation command generation.

[0074] Figure 4 This is a flowchart illustrating the specific method of step S203 provided in the embodiments of this application. (See attached document.) Figure 4 In one embodiment, the method includes, but is not limited to, steps S401 to S404.

[0075] Step S401: The color images acquired from multiple angles are stitched together to obtain a stitched image with the same image angle as human eye vision.

[0076] Step S402: Normalize the illumination and hue features of the stitched image to obtain a normalized image.

[0077] Step S403: Extract the visual features of the furniture from the normalized image.

[0078] Step S404: Map the visual features of furniture in the normalized image to a preset room attribute semantic space to generate room attribute features based on the co-occurrence features of furniture in the normalized image.

[0079] It can be understood that the room attribute semantic space refers to a high-dimensional vector space containing scene labels such as bedroom, living room, and kitchen, constructed through a pre-trained model. Specifically, Word2Vec or BERT models can be used for embedding and mapping to convert visual features into semantic descriptions.

[0080] It is understandable that co-occurrence features refer to the probability distribution of different furniture appearing together in a specific scenario. Specifically, they can be modeled using association rule mining or probabilistic graphical models to infer the functional attributes of a room based on the furniture combination relationship.

[0081] When perceiving the room attribute features of the current navigation location, a complete spatial view consistent with human visual perception can be constructed through multi-angle color image stitching. For example, a stitched image with a hemispherical frame of approximately 180° from the front can be used to avoid furniture occlusion caused by viewing angle limitations. Subsequent normalization of the stitched image for illumination and tone effectively eliminates ambient light interference and improves the robustness of furniture feature extraction in different scenarios. A pre-defined room attribute perception network extracts the visual features of furniture from the normalized image, accurately capturing its shape and material information. These extracted visual features are then mapped to the room attribute semantic space. During this process, co-occurrence features among furniture are used to correct misjudgments that may arise from single features. For example, detecting the simultaneous presence of a bed and a wardrobe can increase the confidence level of the bedroom scene. Therefore, by using multi-view fusion and feature normalization processing, the usability of image data is enhanced. At the same time, by combining co-occurrence features to optimize the scene judgment logic, the accuracy of room attribute recognition is significantly improved, ensuring the stability of feature extraction under different environments. Through co-occurrence feature modeling, the room attribute judgment is made more in line with human cognitive habits of spatial functions, ultimately providing reliable semantic input for navigation command generation.

[0082] Figure 5 This is a flowchart illustrating the specific method of step S204 provided in the embodiments of this application. See also... Figure 5 In one embodiment, the method includes, but is not limited to, steps S501 to S503.

[0083] Step S501: Fuse the color image and the depth image within the target area to obtain a fused image.

[0084] Image fusion can be understood as the pixel-level overlay of color information from a color image with three-dimensional spatial information from a depth image. This can be achieved using multimodal image registration algorithms to enhance the accuracy of object contour and spatial location recognition. Visual features refer to the object's shape, texture, and spatial distribution information extracted from the fused image. This can be achieved using convolutional neural networks for feature extraction to characterize the object's appearance attributes.

[0085] Step S502: Extract the visual features of objects in the fused image.

[0086] Step S503: Map the visual features of objects in the fused image to a preset object attribute semantic space, and align the visual features of objects in the fused image with the corresponding feature text based on an attention mechanism to generate object attribute features.

[0087] It can be understood that the object attribute semantic space refers to the set of mapping relationships between object categories and attribute descriptions constructed by a pre-trained model. Specifically, it can be implemented using a vision-language joint embedding model to transform visual features into understandable semantic information.

[0088] After acquiring color and depth images of the current navigation location, the spatial boundaries of the target area are first determined through coordinate transformation. Multi-angle images collected within this area are then registered and fused to form stereoscopic visual data containing color and depth information. Subsequently, a feature extraction network captures the geometric features and surface texture of the object, and these feature vectors are projected into a predefined object attribute semantic space. During this process, an attention mechanism dynamically calculates the relevance weights between different object features and textual descriptions. For example, for key navigation references like door handles, their circular outline features are assigned higher attention scores, ensuring that the generated object attribute features accurately reflect the object's actual role in the navigation scenario. Thus, through multimodal data fusion and attention alignment mechanisms, not only is the completeness of object attribute recognition improved in complex scenes, but the semantic gap between visual features and linguistic descriptions is also effectively eliminated. For instance, when recognizing stair railings, depth information helps distinguish their spatial relationship with wall decorative lines, while the attention mechanism ensures that the accurate description of "left-side metal railing" replaces the vague "next-door object" description.

[0089] Figure 6 This is a flowchart illustrating the specific method of step S205 provided in the embodiments of this application. (See attached document.) Figure 6 In one embodiment, the method includes, but is not limited to, steps S601 to S602.

[0090] Step S601: Extract the inter-frame change region features of the video stream.

[0091] Step S602: Map the inter-frame change region features and user motion parameters to a preset motion attribute semantic space to generate motion attribute features based on the co-occurrence features of the inter-frame change region features and user motion parameters.

[0092] It is understandable that inter-frame change region features refer to image regions in a video stream that undergo significant changes between adjacent frames. Specifically, they can be extracted using optical flow or background subtraction algorithms to capture dynamic visual information caused by user movement.

[0093] It can be understood that the motion attribute semantic space refers to the vector space that jointly maps visual features with motion parameters. Specifically, it can be constructed using a multimodal fusion network to establish the correlation between visual dynamic changes and the user's motion state.

[0094] When perceiving the user's motion attributes, the video stream is decomposed into a sequence of consecutive frames. By calculating the pixel differences or optical flow vector fields between adjacent frames, the boundaries of regions with significant changes are identified. These region boundaries, along with motion parameters collected by the user's sensors, are synchronously input into a multimodal encoder for feature alignment and joint embedding in the motion attribute semantic space. Co-occurrence features are generated through a cross-modal attention mechanism, capturing the spatiotemporal correlation between visual dynamic changes and motion parameters, ultimately outputting motion attribute features describing the user's motion trends. Thus, by fusing visual dynamic changes and physical motion parameters, a cross-modal correlation is established at the semantic level, enabling more accurate identification of the user's motion intentions and avoiding misjudgments caused by sensor errors or visual occlusion. This achieves refined perception of the user's motion state, allowing generated navigation commands to dynamically adapt to the user's actual movement characteristics. For example, when the user suddenly accelerates or turns, the system can adjust the command content in a timely manner to ensure that navigation guidance remains synchronized with the actual movement trajectory, effectively improving the adaptability of the command generation model in complex motion scenarios.

[0095] Figure 7 This is a flowchart illustrating the specific method of step S206 provided in the embodiments of this application. See also... Figure 7 In one embodiment, the method includes, but is not limited to, steps S701 to S703.

[0096] Step S701: Generate an action sequence that conforms to the navigation trajectory based on the region attribute features, room attribute features, object attribute features and motion attribute features.

[0097] Step S702: Generate the corresponding natural language based on the action sequence.

[0098] Step S703: Perform context alignment processing on the natural language to obtain target navigation instructions.

[0099] It can be understood that an action sequence refers to a step-by-step navigation path planned based on environmental characteristics and user state. Specifically, it can be implemented using path planning algorithms combined with multimodal feature matching. For example, spatial topological mapping can be performed between regional attribute features and the navigation trajectory, and step-by-step path nodes can be generated through dynamic time warping algorithms. This feature is used to transform discrete environmental attributes into executable navigation steps.

[0100] Context alignment processing can be understood as eliminating logical breaks and referential ambiguities in instructions. This can be achieved using attention mechanisms or discourse coherence models, such as maintaining consistency of directional references through cross-sentence entity linking techniques. This feature ensures that the generated instructions remain coherent in both temporal and spatial dimensions.

[0101] After obtaining regional attribute features, room attribute features, object attribute features, and motion attribute features, an action sequence containing elements such as turning angle, movement distance, and obstacle avoidance strategy is generated by fusing spatial function division from regional attribute features, furniture layout from room attribute features, obstacle distribution from object attribute features, and user behavior patterns from motion attribute features. Subsequently, each action step is mapped to a short natural language sentence containing directional words, action words, and reference points, such as "Go straight for three meters, then turn left to avoid the sofa." Finally, by analyzing the semantic relationships between preceding and following sentences, pronoun references and directional descriptions are adjusted to form a coherent and complete navigation command. Thus, by fusing multi-dimensional environmental features to generate dynamic action sequences, combined with language models and context alignment techniques, natural language commands that conform to real-world scenarios and are logically coherent can be automatically generated. This allows for adaptation to different environmental layouts and user movement patterns without relying on manually labeled data. It achieves automated generation of navigation commands, dynamically adjusting command content based on real-time environmental features. The generated commands possess natural language diversity and contextual coherence, effectively solving the technical problems of existing methods such as single command form and lack of environmental adaptability, and providing high-quality data support for the training and evaluation of command navigation models.

[0102] See Figure 8 This application also provides a navigation instruction generation apparatus that can implement the above-described navigation instruction generation method. The apparatus includes: The first module 801 is used to acquire navigation trajectory, user motion parameters, and color and depth images of the current navigation location; The second module 802 is used to perceive the regional attribute features of the current navigation location based on the color image; The third module 803 is used to perceive the room attribute features of the current navigation location based on color images; The fourth module 804 is used to perceive the object attribute features of objects within the target area based on color and depth images; the target area is the range extending from the current navigation position to a preset threshold distance. The fifth module 805 is used to perceive the user's motion attribute characteristics based on the video stream composed of color images and the user's motion parameters; The sixth module 806 is used to generate navigation instructions based on navigation trajectory, area attribute features, room attribute features, object attribute features and motion attribute features, to obtain target navigation instructions.

[0103] The specific implementation of the navigation instruction generation device is basically the same as the specific embodiment of the navigation instruction generation method described above, and will not be repeated here.

[0104] Figure 9 This is a block diagram illustrating an electronic device according to an exemplary embodiment.

[0105] The following reference Figure 9 To describe an electronic device 900 according to such an embodiment of the present disclosure. Figure 9 The electronic device 900 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0106] like Figure 9 As shown, the electronic device 900 is presented in the form of a general-purpose computing device. The components of the electronic device 900 may include, but are not limited to: at least one processing unit 910, at least one storage unit 920, a bus 930 connecting different system components (including storage unit 920 and processing unit 910), a display unit 940, etc.

[0107] The storage unit stores program code, which can be executed by the processing unit 910, causing the processing unit 910 to perform the steps described in the navigation instruction generation method section of this specification according to various exemplary embodiments of this disclosure.

[0108] Storage unit 920 may include readable media in the form of volatile storage units, such as random access memory (RAM) 9201 and / or cache memory 9202, and may further include read-only memory (ROM) 9203.

[0109] Storage unit 920 may also include a program / utility 9204 having a set (at least one) program module 9205, such program module 9205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0110] Bus 930 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0111] Electronic device 900 can also communicate with one or more external devices 900' (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 900, and / or with any device that enables electronic device 900 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 950. Furthermore, electronic device 900 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 960. Network adapter 960 can communicate with other modules of electronic device 900 via bus 930. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0112] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0113] The navigation instruction generation method, apparatus, device, and medium provided in this application embodiment perceive the regional attribute features and room attribute features of the current navigation location, the object attribute features of objects near the current navigation location, and the motion attribute features of the user. Combining the navigation trajectory, regional attribute features, room attribute features, object attribute features, and motion attribute features, the instructions are converted into target navigation instructions containing orientation indicators and environmental references by a natural language generation module. This significantly improves the adaptability of instruction generation. Especially in complex indoor scenes, it can accurately identify the spatial position of temporary obstacles, avoid instruction errors caused by changes in object position, and can hierarchically perceive key information of the navigation trajectory to generate various types of navigation instructions.

[0114] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, or network device, etc.) to execute the methods described above according to the embodiments of this disclosure.

[0115] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0116] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0117] Those skilled in the art will understand that the above modules can be distributed in the device as described in the embodiments, or they can be modified accordingly and placed in one or more devices that are unique to this embodiment. The modules in the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0118] Exemplary embodiments of this disclosure have been specifically shown and described above. It should be understood that this disclosure is not limited to the detailed structures, arrangements, or implementations described herein; rather, this disclosure is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A method for generating navigation instructions, characterized in that, include: Acquire navigation trajectory, user motion parameters, and color and depth images of the current navigation location; Based on the color image, the regional attribute features of the current navigation location are perceived; Based on the color image, the room attribute features of the current navigation location are perceived; Based on the color image and the depth image, the object attribute features of objects within the target area are perceived; the target area is a range extending from the current navigation position to a preset threshold distance. The user's motion attribute characteristics are perceived based on the video stream composed of the color images and the user's motion parameters; Based on the navigation trajectory, the area attribute features, the room attribute features, the object attribute features, and the motion attribute features, navigation instructions are generated to obtain target navigation instructions.

2. The navigation instruction generation method according to claim 1, characterized in that, Also includes: Obtain navigation tasks; A path that meets the navigation task is generated using a path planning method, thus obtaining the navigation trajectory.

3. The navigation instruction generation method according to claim 1, characterized in that, The method of perceiving the regional attribute features of the current navigation location based on the color image includes: The color images acquired from multiple angles are stitched together to obtain a panoramic image; The panoramic image is mapped onto a plane using an equidistant cylindrical projection method to obtain a projected image. The visual attribute features in the projected image are mapped to a preset regional attribute semantic space to generate the regional attribute features based on a preset environmental function description paradigm.

4. The navigation instruction generation method according to claim 1, characterized in that, The process of perceiving room attribute features of the current navigation location based on the color image includes: The color images acquired from multiple angles are stitched together to obtain a stitched image whose angles are the same as those of the human eye. The illumination and hue features of the stitched image are normalized to obtain a normalized image; Extract the visual features of the furniture from the normalized image; The visual features of furniture in the normalized image are mapped to a preset room attribute semantic space to generate the room attribute features based on the co-occurrence features of furniture in the normalized image.

5. The navigation instruction generation method according to claim 1, characterized in that, The method of perceiving object attribute features of objects within the target region based on the color image and the depth image includes: The color image and the depth image are fused within the target area to obtain a fused image; Extract the visual features of objects in the fused image; The visual features of objects in the fused image are mapped to a preset object attribute semantic space, and the visual features of objects in the fused image are aligned with the corresponding feature text based on an attention mechanism to generate the object attribute features.

6. The navigation instruction generation method according to claim 1, characterized in that, The method of perceiving the user's motion attribute features based on the video stream composed of the color images and the user's motion parameters includes: Extract the inter-frame variation region features of the video stream; The inter-frame change region features and the user motion parameters are mapped to a preset motion attribute semantic space to generate the motion attribute features based on the co-occurrence features of the inter-frame change region features and the user motion parameters.

7. The navigation instruction generation method according to claim 1, characterized in that, The generation of navigation instructions based on the navigation trajectory, the area attribute features, the room attribute features, the object attribute features, and the motion attribute features includes: Generate an action sequence that conforms to the navigation trajectory based on the area attribute features, room attribute features, object attribute features, and motion attribute features; Generate corresponding natural language based on the action sequence; The natural language is then subjected to context alignment processing to obtain the target navigation instructions.

8. A navigation instruction generation device, characterized in that, include: The first module is used to acquire navigation trajectory, user motion parameters, and color and depth images of the current navigation location; The second module is used to perceive the regional attribute features of the current navigation location based on the color image; The third module is used to perceive the room attribute features of the current navigation location based on the color image; The fourth module is used to perceive the object attribute features of objects within the target area based on the color image and the depth image; the target area is a range extending from the current navigation position to a preset threshold distance. The fifth module is used to perceive the user's motion attribute characteristics based on the video stream composed of the color images and the user's motion parameters; The sixth module is used to generate navigation instructions based on the navigation trajectory, the area attribute features, the room attribute features, the object attribute features, and the motion attribute features, to obtain target navigation instructions.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the navigation instruction generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the navigation instruction generation method according to any one of claims 1 to 7.

Images