A semantic navigation method and system based on three-dimensional Gaussian splash

Abstract

The application discloses a kind of semantic navigation methods and systems based on three-dimensional Gaussian splash. It aims at improving the navigation accuracy and efficiency of robot in indoor environment. Method is through data acquisition and preprocessing, indoor environment image is obtained using high-precision RGB-D camera, then three-dimensional Gaussian splash model training is carried out, and three-dimensional scene containing geometry, texture and appearance information is constructed. Further, a deep learning model is used for semantic segmentation, and each part of the scene is given an accurate semantic label. Finally, in the semantic navigation implementation stage, the system quickly locates the target object according to the user input target, and plans the optimal path to achieve accurate navigation. The present application can update the three-dimensional semantic field in real time to adapt to the dynamic changes of indoor environment, and improve the real-time performance and reliability of navigation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of semantic navigation, in particular to a semantic navigation method and system based on three-dimensional Gaussian splatting. BACKGROUND

[0002] With the development of artificial intelligence, sensor technology and robot technology, embodied intelligence technology endows robots with a "body", enabling them to perceive the environment, understand tasks, and autonomously navigate and perform tasks in complex environments. In particular in home environments, the accurate navigation capability of mobile robots or humanoid robots becomes increasingly important.

[0003] Traditional two-dimensional semantic maps cannot fully present the three-dimensional structure of the environment, resulting in information loss, labeling errors, ambiguity and other problems, which limits the accuracy and efficiency of navigation. Semantic segmentation technology has limitations in precision, detail and individual distinction, and has high computational load. Patent CN111645073B proposes a robot visual semantic navigation method, but has problems such as dependence on semantic atlas, high demand for computing resources, limitations of voice instructions, poor environmental adaptability, insufficient model updating and generalization ability, and complexity of semantic understanding. SUMMARY

[0004] Therefore, the embodiments of the present application provide a semantic navigation method and system based on three-dimensional Gaussian splatting, aiming to build a three-dimensional semantic field to achieve accurate and efficient open text query and apply it to the field of indoor navigation, etc., to solve the limitations of existing technologies in speed and accuracy.

[0005] In a first aspect, a semantic navigation method based on three-dimensional Gaussian splatting is provided, the method comprising:

[0006] using a mobile device with a high-precision RGB-D camera to collect RGB images and depth images at a stable frame rate according to a predetermined path, ensuring coverage of the entire indoor space, and pre-processing the collected RGB images and depth images;

[0007] confirming a three-dimensional Gaussian splatting model architecture, initializing model parameters, inputting the pre-processed image data into the model, optimizing the model parameters using the stochastic gradient descent method, calculating the mean square error of the predicted scene and the actual scene, and updating the parameters in reverse according to the error; wherein the model parameters at least include the mean value, covariance matrix and color of the initialized Gaussian distribution; the three-dimensional Gaussian splatting model architecture is LangSplat;

[0008] Confirming a semantic segmentation model architecture, collecting a large number of indoor scene labeled images, training the model with a cross-entropy loss function, and applying the trained semantic segmentation model to a three-dimensional Gaussian splash reconstructed three-dimensional scene to assign accurate semantic labels to each pixel or three-dimensional Gaussian distribution; wherein the semantic segmentation model includes U-Net or DeepLab;

[0009] The user inputs a navigation target through a voice command or a graphical interface, analyzes the navigation target according to the trained semantic segmentation model, determines the target semantic category and the region where it is located, and locates the three-dimensional coordinates of the target object according to the target semantic label and the position information; path planning, navigation execution and implementation update are performed based on the located three-dimensional coordinates.

[0010] Optionally, path planning, navigation execution and implementation update based on the located three-dimensional coordinates, specifically including:

[0011] Based on the three-dimensional scene semantic information and the target position, an A* path planning algorithm is used to calculate the optimal path, and the path is converted into a sequence of executable actions for a mobile device or a robot; the mobile device or the robot executes navigation according to the planned path, collects real-time surrounding environment image data, and inputs the new data into the three-dimensional Gaussian splash and semantic segmentation model to update the three-dimensional scene representation and semantic information in real time, and adjusts the path planning in a timely manner according to the updated information.

[0012] Optionally, a mobile device with a high-precision RGB-D camera is used to collect RGB images and depth images at a stable frame rate according to a predetermined path, and to ensure that the entire indoor space is covered, and the collected RGB images and depth images are preprocessed, including:

[0013] Collecting multiple times under different heights, multiple angles and different lighting conditions; wherein the heights include at least 0.5 meters, 1 meter and 1.5 meters, the multiple angles include a horizontal direction interval of 30 degrees, and the lighting conditions include daytime natural light and nighttime light;

[0014] Precise registration and normalization of image and depth information, and removal of depth image noise using median filtering and other methods.

[0015] Optionally, inputting the preprocessed image data into the model further includes:

[0016] Using data augmentation techniques, including randomly cropping, rotating and flipping images to increase the diversity of training data, and using a validation set to monitor model performance to prevent overfitting.

[0017] Optionally, the model parameters are optimized using the stochastic gradient descent method, including:

[0018] Calculating the mean square error of the predicted scene and the actual scene and updating the parameters in reverse, wherein N is the number of samples, yi is a true value, is a predicted value.

[0019] Optionally, the model is trained with a cross-entropy loss function, comprising:

[0020] with a cross-entropy loss function training the model, wherein C is the number of categories, y i is a true value, is a predicted value, updating the model parameters by back propagation, and adjusting the training parameters according to the validation set to optimize the model effect.

[0021] In a second aspect, a semantic navigation system based on three-dimensional Gaussian splatting is provided, comprising:

[0022] A data acquisition and preprocessing module is configured to acquire RGB images and depth images at a stable frame rate according to a predetermined path using a mobile device with a high-precision RGB-D camera, to ensure coverage of the entire indoor space, and to preprocess the acquired RGB images and depth images.

[0023] A three-dimensional Gaussian splatting model training module is configured to confirm a three-dimensional Gaussian splatting model architecture, initialize model parameters, input preprocessed image data into the model, optimize model parameters using a stochastic gradient descent method, calculate the mean square error of the predicted scene and the actual scene, and update the parameters in reverse according to the error; wherein the model parameters at least include the mean, covariance matrix, and color of the initialized Gaussian distribution; and the three-dimensional Gaussian splatting model architecture is LangSplat.

[0024] A semantic segmentation model training and application module is configured to confirm a semantic segmentation model architecture, collect a large number of indoor scene labeled images, train the model with a cross-entropy loss function, apply the trained semantic segmentation model to the three-dimensional scene reconstructed by the three-dimensional Gaussian splatting, and assign accurate semantic labels to each pixel or three-dimensional Gaussian distribution; wherein the semantic segmentation model includes U-Net or DeepLab.

[0025] A semantic navigation implementation module is configured to input a navigation target through a voice command or a graphical interface, analyze the navigation target according to the trained semantic segmentation model, determine the target semantic category and the region where the target semantic category is located, locate the three-dimensional coordinates of the target object based on the target semantic label and the position information, and perform path planning, navigation execution, and implementation update based on the located three-dimensional coordinates.

[0026] In a third aspect, an electronic device is provided, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the semantic navigation method of any one of the first aspect when executing the computer program.

[0027] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the semantic navigation method according to any one of the first aspect.

[0028] In a fifth aspect, a computer program product is provided, and the computer program product comprises computer programs / instructions. The computer programs / instructions are executed by a processor to implement the semantic navigation method according to any one of the first aspect.

[0029] The technical scheme provided by the embodiments of the present application has at least the following beneficial effects:

[0030] With the aid of three-dimensional Gaussian splashing and semantic segmentation, the object boundary can be accurately outlined, and the LERF dataset visualization result shows that the present application is superior to the traditional method. For example, in a complex indoor scene, the boundaries of furniture and decorations can be clearly distinguished, confusion is avoided, a reliable foundation is provided for navigation and other operations, and it is of great significance for accurate operation of a robot.

[0031] Multi-scale semantic levels are obtained by using SAM, and the structural relationship of an object is comprehensively grasped. Taking a table as an example, the whole, the tabletop, the table leg and the texture can be recognized, the semantic understanding ability is improved, which is helpful for more accurate path planning in navigation and improves safety and accuracy.

[0032] Combined with semantic information, an optimized path is planned, passable areas and obstacles are identified, detours are reduced in a complex layout, and navigation efficiency and fluency are improved.

[0033] A three-dimensional semantic field is updated in real time, and indoor dynamic changes such as furniture movement and personnel movement are adapted to, so as to ensure accurate and reliable navigation and be suitable for scenes such as an office and a conference room.

[0034] Good adaptability to light changes and occlusions is achieved, uneven light and object occlusion problems are overcome, stable navigation function is maintained in a complex actual environment, and system robustness is improved. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present application or the technical scheme in the prior art, the drawings needed to be used in the following embodiment or prior art description will be briefly introduced. Obviously, the drawings in the following description are only exemplary, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0036] Figure 1 A step flow chart of a semantic navigation method based on three-dimensional Gaussian splashing provided by the embodiments of the present application is provided.

[0037] Figure 2 A block diagram of a semantic navigation system based on three-dimensional Gaussian splashing provided by the embodiments of the present application is provided.

[0038] Figure 3 A schematic diagram of an electronic device is provided for embodiments of the present application. DETAILED DESCRIPTION

[0039] In order to make the objectives, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application.

[0040] In the description of the present application, the terms "comprising", "having" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to only those steps or units explicitly listed, but can also include other steps or units inherent to these processes, methods, products or devices, or steps or units added based on further optimization of the concept of the present application.

[0041] With the rapid progress of technology, embodied intelligence technology, which combines artificial intelligence, sensor technology, robotics and other multi-disciplinary knowledge, is constantly developing and improving. The core of this technology is to give robots a "body" so that they can perceive the environment, understand tasks, and navigate and perform tasks autonomously in complex environments, just like humans. In particular, in the home environment, the accurate navigation capability of mobile robots or humanoid robots becomes increasingly important.

[0042] In modern home life, people's demand for intelligent services is growing. Whether it is a cleaning robot, a companion robot or a home security monitoring robot, they all need to have the ability to move freely in the home environment. This not only involves the robot's recognition and memory of space, but also includes the ability to adapt to dynamic environments, such as avoiding sudden obstacles, identifying the location of family members, and predicting and planning the best path, etc.

[0043] Three-dimensional Gaussian splatting:

[0044] With the continuous development of computer graphics, computer vision and virtual reality, the demand for efficient and realistic presentation of three-dimensional scenes is growing. Traditional three-dimensional modeling and rendering methods have limitations in some aspects, such as time-consuming and laborious complex scene modeling, and not enough delicate and realistic rendering effects. In this context, three-dimensional Gaussian splatting technology has emerged, aiming to process the representation and rendering of three-dimensional scenes in a more flexible, efficient and high-quality visual effect way.

[0045] Three-dimensional Gaussian Spray is a representation method that uses Gaussian distribution functions as its core elements. Each Gaussian distribution represents a local feature in the scene. By combining and adjusting multiple Gaussian distributions, it can accurately describe various shapes, colors, and lighting information in a three-dimensional scene. This Gaussian-based representation allows it to capture scene details, both surface textures and lighting variations, in great detail. Compared to traditional three-dimensional model representation methods such as polygon meshes, Three-dimensional Gaussian Spray has better storage efficiency. It does not need to store a large amount of complex geometric mesh information, but instead uses relatively simple Gaussian parameters (such as mean and covariance) to represent the scene. In terms of computation, its rendering process takes advantage of the properties of Gaussian distribution to achieve fast calculation to some extent, reducing the time and resource consumption required for rendering. This technology can adaptively adjust the density and accuracy of Gaussian distribution based on factors such as viewing angle and scene importance. When the viewing angle is close to an object, it can increase the density of Gaussian distribution in that area to show more detailed details; in areas far away or less important, it can appropriately reduce the density to save resources while still maintaining the visual coherence of the overall scene.

[0046] The current application scenarios of Three-dimensional Gaussian Spray mainly focus on the fields of Virtual Reality (VR) and Augmented Reality (AR), film and television, game modeling, and architecture. In VR and AR experiences, users have very high requirements for immersion and realism of the scene. Three-dimensional Gaussian Spray technology can quickly generate realistic three-dimensional scenes, making users feel as if they are in the scene. For example, in VR games, it can be used to create detailed game scenes, from fantastical castles to dense forests, with every detail presented in high quality, enhancing the player's gaming experience. In AR applications, when virtual objects are superimposed on real scenes, it can ensure that the rendering effect of virtual objects matches the real environment, providing a more natural fusion effect.

[0047] Existing solution two-dimensional semantic map:

[0048] Two-dimensional semantic map is a map form that annotates and describes semantic information of the environment on a two-dimensional plane. It plays an important role in many fields such as robot navigation, autonomous driving, indoor positioning, etc. Two-dimensional semantic map is based on a two-dimensional coordinate system, and different areas, objects, etc. in the environment are divided and labeled according to specific semantic categories to construct. It not only presents the geometric shape of the environment, but more importantly, gives each part a clear semantic meaning.

[0049] A two-dimensional semantic map presents environmental information in an intuitive and understandable way, whether for human operators or robots or autonomous vehicles that need to navigate based on the map. For example, in indoor navigation, people can clearly see from the map the layout of the room, the location of the door, and the path to the destination, etc. Two-dimensional semantic maps can be easily extended and updated. As the environment is perceived in depth or new objects appear, new geometric shapes and semantic labels can be added to the map at any time. For example, in an office, when a new device is purchased, it can be easily added to the map as a new geometric shape and given the corresponding semantic label.

[0050] For indoor service robots, industrial robots, etc., two-dimensional semantic maps are an important basis for their navigation, obstacle avoidance, and task completion in complex environments. Robots can identify passable areas such as corridors and doors, and objects to avoid such as tables and chairs based on semantic information on the map, thereby planning a reasonable path and successfully completing the task. In indoor environments, two-dimensional semantic maps combined with other positioning technologies such as WiFi positioning and Bluetooth positioning can achieve more accurate indoor positioning. By comparing the current location of the user or device with the labeled information on the map, the specific location can be determined, such as in a large shopping mall, customers can find their floor and store location through a mobile app combined with a two-dimensional semantic map.

[0051] Three-dimensional Gaussian splatting has the following advantages in navigation:

[0052] First, new perspective rendering:

[0053] In the navigation scene, whether it is an indoor environment such as a large shopping mall, office building, etc. or an outdoor scene such as a city street, park, etc., it is very important to be able to switch perspectives flexibly according to the needs of users or navigation devices. Three-dimensional Gaussian splatting technology performs well in new perspective rendering based on its unique principles. It can quickly and realistically re-render the overall scene at different perspectives by representing each element in a three-dimensional scene with a Gaussian distribution. For example, when pedestrians navigate inside a shopping mall, they may need to switch from a level view perspective to an overhead perspective to better view the layout of the entire floor and the location relationship of each store. Three-dimensional Gaussian splatting can easily achieve this change in perspective and can delicately present the details of the mall decorations, shelf arrangements, and crowd distribution seen from different heights, just like actually looking down or looking straight ahead, providing more intuitive and comprehensive visual information for navigation.

[0054] The navigation environment is often complex and diverse, containing objects of various shapes, colors, and materials. Three-dimensional Gaussian splatting can handle this complexity well. For objects with irregular shapes, such as the unique appearance of old buildings or the strange sculptures in the park, it can accurately capture their shape characteristics using Gaussian distribution and present them completely when rendering from a new perspective. At the same time, for objects of different materials, such as the reflective effect of glass curtain walls or the texture of wooden furniture, it can also realistically simulate their visual effects by adjusting the parameters of the Gaussian function. This allows users to clearly perceive the true appearance of the surrounding environment during navigation, whether in bright daylight or dim light at night, and to obtain accurate visual guidance, thus more smoothly completing the navigation task.

[0055] II. Multi-frame consistency:

[0056] During navigation, especially for dynamic scenes (such as pedestrians walking on the street, vehicles driving on the road, etc.), it is necessary to ensure that the scene presents consistency in consecutive multiple frames. Three-dimensional Gaussian splatting technology has a significant advantage in this regard. Since it describes elements in the scene with Gaussian distribution, as long as the objects in the scene do not change substantially (such as position movement, shape change, etc.) between different frames, through continuous processing and rendering of the same Gaussian distribution, it can ensure that the basic features of the scene in each frame remain consistent. For example, when an autonomous vehicle is driving, it continuously acquires images of the surrounding environment through on-board cameras for navigation. Three-dimensional Gaussian splatting can ensure that the appearance and positional relationship of roads, buildings, traffic signs, etc. remain stable in these consecutive image frames, and there will be no situation where a corner of a building suddenly disappears or deforms in the previous frame, etc. This provides a reliable visual reference for the autonomous driving system, helping it to accurately judge the road conditions and make correct driving decisions.

[0057] In the navigation scene, in addition to static environmental objects, there are often dynamic objects such as pedestrians, other vehicles, etc. The multi-frame consistency feature of three-dimensional Gaussian splatting helps to track these dynamic objects.

[0058] By maintaining the identification and processing of the Gaussian distribution corresponding to the same dynamic object in consecutive frames, the motion trajectory of the dynamic object can be clearly observed. For example, in the navigation scene of an airport hall, staff can use three-dimensional Gaussian splatting technology to track the walking route of a passenger through monitoring equipment. Even if the passenger is moving in the crowd and changing direction, the Gaussian distribution corresponding to the passenger can be accurately identified in each frame, thus keeping track of their movements in real time, providing convenience for navigation guidance or security monitoring, etc.

[0059] III. Can build a three-dimensional semantic map:

[0060] Three-dimensional semantic maps can provide rich semantic information for navigation, and when combined with three-dimensional Gaussian splashes, they can better leverage their advantages.

[0061] In a navigation environment, each Gaussian distribution can not only represent the geometric shape and appearance characteristics of an object, but also be assigned a specific semantic label, such as "wall", "door", "road", "vehicle", "pedestrian", etc. In this way, during navigation, both human users and navigation devices (such as autonomous vehicles, service robots, etc.) can more targetedly plan paths, avoid obstacles, etc. based on these semantic information.

[0062] For example, in indoor office environment navigation, through the three-dimensional semantic map with semantic labels generated by three-dimensional Gaussian splashes, a service robot can easily identify which areas are corridors, which are offices, and which are conference rooms, etc., thereby accurately finding the target location and avoiding obstacles such as desks and chairs, and efficiently completing tasks.

[0063] When three-dimensional semantic maps are constructed based on three-dimensional Gaussian splashes, they can provide more accurate spatial understanding. Since Gaussian distributions can finely describe the shape and position of an object, combined with semantic labels, navigation users or devices can have a deeper understanding of the spatial structure of the entire navigation environment. For example, in the field of urban planning and architectural design, designers can use the three-dimensional semantic map generated by this technology to evaluate the impact of building layout on the passage of pedestrians, vehicles, etc. For example, by observing the three-dimensional Gaussian splash model with semantic labels, one can clearly see the connection between roads and buildings, the passage settings between different floors, etc., thereby optimizing the spatial layout and improving the navigation convenience of the entire environment.

[0064] The main content of patent CN111645073B is a robot visual semantic navigation method, device and system. The abstract of this invention is as follows:

[0065] The present application relates to the technical field of robot navigation, and discloses a robot visual semantic navigation method, comprising the following steps: establishing a set of scene images and a set of voice instructions; labeling the image features of each scene image in the set of scene images, and labeling the voice features of each voice instruction in the set of voice instructions; constructing a semantic atlas by combining the image features and voice features of the same time, obtaining a set of semantic atlases, and labeling the semantic features of each semantic atlas in the set of semantic atlases; constructing a state vector by fusing the image features, voice features and semantic features of the same time, obtaining a set of state vectors; labeling the action sequence corresponding to each state vector in the set of state vectors, using the set of state vectors as training samples to train a deep reinforcement learning model, obtaining a navigation model; and controlling the robot according to the navigation model. The present application can realize the navigation of objects that are not in the field of view of the robot.

[0066] The advantages of the present application are as follows:

[0067] Navigation of objects not in the field of view: By establishing a navigation model containing semantic features, combining image features, speech features, and extracted semantic features, the association between different objects can be understood, thereby establishing the relationship between the target object and the existing objects in the scene image for the robot, and realizing navigation based on semantic prior knowledge.

[0068] Integration of deep reinforcement learning and knowledge graph: The knowledge graph can encode various forms of knowledge, making it possible for semantic prior knowledge to participate in the training of the deep reinforcement learning model, so that the navigation model can accurately define the relationship between the target object and other objects in the scene according to semantic features, improve the generalization ability for invisible environments and target objects, and improve the navigation performance.

[0069] Use multiple modeling methods to improve efficiency: For example, when extracting real-time image features, a visual neural network is trained using an established set of scene images to obtain an image feature extraction model, without the need to rebuild the model, improving efficiency; when extracting real-time semantic features, the model is also trained using an established set of semantic graphs.

[0070] Use pre-trained models to accelerate training: When training the image feature extraction model, first use the ImageNet dataset to pre-train the ResNet-50 network to obtain a pre-trained model with certain object recognition capabilities, and then use the scene image set for training, reducing the training difficulty and reducing the quantity and quality requirements for the scene image set.

[0071] Real-time navigation control: After the navigation model is established, real-time image, speech, and semantic features can be extracted from real-time scene images, speech instructions, and semantic graphs, and then a real-time state vector can be obtained, which is input into the navigation model to obtain a real-time action sequence to control the robot and help it find the target object.

[0072] Improve the representation ability of instruction features and map features: By establishing the correspondence between the relevant objects in the instruction and the semantic map, alignment and fusion of two modalities (text and vision) are realized.

[0073] III. Defects in the background art

[0074] There are many shortcomings in two-dimensional semantic maps. As a two-dimensional plane map, it cannot fully present the three-dimensional structure of the environment. For some environments with height changes, such as the interior of a multi-story building, a terrain with slopes, etc., a two-dimensional semantic map can only provide information on a plane, which may result in partial information loss and affect the overall understanding of the environment. For example, in actual scenarios, objects are often stacked on a table, such as a stack of books, a pile of files, etc. Since the two-dimensional semantic map does not have height information, for these stacked objects, it is not possible to determine which layer belongs to which specific object, nor is it clear how they are arranged in the vertical direction. For example, a stack of five books with different contents may just be represented as a fuzzy rectangle or polygon (representing the projection of the book stack on the table) on a two-dimensional semantic map, and it is not possible to segment and identify each book as an independent individual, because there is no height information to assist in distinguishing their specific positions and attribution in the stack.

[0075] Although the accuracy of semantic labeling is continuously improved through various technical means, there are still cases of labeling errors or inaccuracies. For example, due to the limited resolution of visual sensors, the limitations of image recognition technology, etc., a similar shape but different semantic object may be incorrectly labeled, or when labeling complex objects, it may not be possible to accurately subdivide their semantic categories.

[0076] At the same time, two-dimensional semantic maps have ambiguity. When performing semantic labeling, some of the words used may have multiple meanings, leading to ambiguity in map interpretation. For example, the word "passageway" may have different meanings in different environments. In an indoor environment, it may refer to a passageway between rooms for passage; while in an outdoor environment, such as in a large park, it may refer to a path for pedestrians to stroll or a road for vehicles to travel. When labeled as "passageway" on a two-dimensional semantic map, it is difficult to accurately determine which type of passageway it refers to without additional context information. In addition, insufficient semantic subdivision may also cause ambiguity. Sometimes, the labeled semantic categories may not be detailed enough, leading to ambiguity. For example, only "table" is labeled, but there may be multiple types of tables in the environment, such as office tables, dining tables, laboratory tables, etc., which may differ in size, shape, purpose, etc. If the map user needs more precise information, such as finding a table with a specific purpose, labeling only "table" cannot meet the demand, and may lead to different understandings of the relevant areas on the map due to such general labeling.

[0077] The semantic segmentation technology also has many shortcomings. First, the traditional navigation technology combined with semantic information may have the problem of high computational load. The traditional semantic vSLAM system has a large amount of calculation when processing image features, especially in the aspects of feature point extraction, descriptor generation and pose estimation. At the same time, due to the high consumption of computing resources, it is difficult to realize real-time semantic segmentation in many practical application scenarios, and the real-time performance cannot be guaranteed.

[0078] In addition, semantic segmentation has shortcomings in precision and detail. When segmenting objects, especially the edge parts of objects, it is difficult to achieve very accurate division. For example, when distinguishing a cat lying on the grass, the boundary between the cat's body hair and the surrounding grass may not be clear, and the semantic segmentation model may not be able to accurately determine the boundary pixel attribution of the cat and the grass, resulting in a blurred segmentation result at the boundary, with some grass pixels being misclassified into the cat category, or some hair pixels at the edge of the cat being misclassified into the grass category. This is the common boundary blurring problem. In addition, for objects with complex and detailed structures, semantic segmentation is difficult to accurately present the details. Taking a Gothic church in a building scene as an example, the facade of the church often has many exquisite carvings, complex spire structures and small stained glass windows, and the semantic segmentation model may blur these fine structures, failing to accurately segment each carving element, different parts of the spire and the outline of the stained glass window, but rather roughly classifying them as a whole building category.

[0079] At the same time, semantic segmentation itself has limitations. Semantic segmentation can provide object types at the pixel level, but cannot distinguish different individuals in the same category, which limits its application range in more complex scenes. Specifically: when different categories of objects are similar in appearance, color and texture, the semantic segmentation model is prone to confusion. For example, in an image containing snow and a white bedsheet drying scene, the white snow and the white bedsheet may have similar colors and some degree of texture similarity, and the model may mistakenly segment some of the snow as the bedsheet category, or vice versa, making it difficult to accurately distinguish between these two different semantic objects with similar visual features. At the same time, in real-world scenarios, multiple category objects often overlap with each other, making it extremely challenging to accurately segment each category. For example, in an image containing a person holding a flower, the person's hand and the flower overlap, and the model has difficulty accurately separating the pixels of the hand and the flower to determine which pixels belong to the hand and which pixels belong to the flower, often resulting in segmentation errors or unclear segmentation in the overlapping area.

[0080] The defects of the patent CN111645073B are as follows:

[0081] Dependence on semantic graph: The effectiveness of the method depends on the construction and accuracy of the semantic graph. If the construction of the semantic graph is biased or incomplete, it may affect the accuracy of navigation.

[0082] Computational resource requirements: Involving deep reinforcement learning, neural network training, and real-time image, speech, and semantic feature processing, it may require more computational resources. For some robots with limited computing power, there may be real-time or performance challenges.

[0083] Limitations of voice instructions: The accuracy of understanding and converting voice instructions into natural language information may be limited by speech recognition technology. Complex or unclear voice instructions may result in inaccurate extracted speech features.

[0084] Environmental adaptability: Although the method considers the navigation of objects not in the field of view to some extent, its performance may be affected in extreme or special environmental conditions, such as severe image data loss or excessive noise interference.

[0085] Model updating and generalization ability: In practical applications, the environment may change, and it is necessary to consider how to update the semantic graph and navigation model in a timely manner to maintain good generalization ability and navigation effect.

[0086] Complexity of semantic understanding: Natural language semantic understanding itself has complexity and ambiguity, and there may be cases where the semantics of some instructions or scenarios are difficult to accurately capture and express.

[0087] Fourth, the purpose of the invention:

[0088] The purpose of the invention is to build a three-dimensional semantic field to achieve accurate and efficient open text queries and apply it to fields such as indoor navigation. Specifically, by combining three-dimensional Gaussian splashing technology, using SegmentAnything Model (SAM) to learn hierarchical semantics, and introducing a semantic autoencoder for specific scenarios, the limitations of existing methods in terms of speed and accuracy are addressed, enabling the model to respond to text-based queries and quickly and accurately locate and identify target objects in three-dimensional space, providing support for indoor navigation. Compared to methods based on NeRF, the invention maintains rendering efficiency while improving query speed and the accuracy of semantic fields, bringing better solutions for applications such as robot navigation, 3D editing, and augmented reality.

[0089] Fifth, the scheme of the invention:

[0090] (1) Overall technical scheme overview:

[0091] The present application focuses on precise semantic navigation in indoor environment, and the core is to fuse three-dimensional Gaussian splatting technology and semantic segmentation technology, and combine mature navigation means. The overall scheme covers key steps such as environment information collection, three-dimensional Gaussian splatting model training, semantic segmentation and final semantic navigation implementation, and each step closely cooperates to achieve the goal of efficient and accurate semantic navigation.

[0092] (2) Technical principle:

[0093] (I) Three-dimensional Gaussian splatting principle:

[0094] Three-dimensional Gaussian splatting represents a three-dimensional scene by a set of three-dimensional Gaussian distributions, where each three-dimensional Gaussian is determined by the mean vector and the covariance matrix ∑, and its expression is

[0095]

[0096] This representation can effectively capture the geometry, texture and appearance information of the scene, realize efficient rendering and processing, and at the same time preserve rich details. When optimizing the parameters of three-dimensional Gaussian, it is often rendered to a two-dimensional image plane, and the rendering efficiency is improved by using tile-based rasterization technology, and the rendering color calculation formula is

[0097]

[0098] Where c i is the color of the i-th Gaussian, N represents the set of Gaussians within the tile, C(v) is the rendering color at pixel v, o i is the opacity of the i-th Gaussian, and proj

[0099] (II) Semantic segmentation principle:

[0100] Semantic segmentation uses deep learning models such as convolutional neural networks (CNN) to train a large number of labeled images to learn the feature representation of different objects, so as to realize accurate classification of image pixels. The core is to build a suitable network structure to enable the model to automatically extract semantic information in the image and map the pixels to predefined semantic categories.

[0101] (III) Semantic navigation principle based on three-dimensional Gaussian splatting:

[0102] By integrating three-dimensional Gaussian splatting and semantic segmentation, semantic labels are assigned to Gaussian distributions in three-dimensional scenes to achieve semantic understanding of the scene. In the navigation process, according to the semantic attributes of the target, combined with existing path planning algorithms (such as A* algorithm), the optimal path is planned in three-dimensional space, while avoiding obstacles with real-time perception and scene updating, realizing precise semantic navigation. A* algorithm selects nodes by evaluating function

[0103] f(n)=g(n)+h(n) (3)

[0104] where g(n) represents the actual cost from the starting node to node n, and h(n) is the estimated cost from node n to the target node.

[0105] (3) Implementation process:

[0106] Please refer to Figure 1 , which shows a flowchart of a semantic navigation method based on three-dimensional Gaussian splatting according to an embodiment of the present application, which can include the following steps:

[0107] S1 Data acquisition and preprocessing: using a mobile device with a high-precision RGB-D camera, RGB images and depth images are collected at a stable frame rate according to a predetermined path, ensuring coverage of the entire indoor space, and the collected RGB images and depth images are preprocessed.

[0108] S2 Training of three-dimensional Gaussian splatting model: confirm the architecture of the three-dimensional Gaussian splatting model, initialize the model parameters, input the preprocessed image data into the model, optimize the model parameters using stochastic gradient descent method, calculate the mean square error of the predicted scene and the actual scene, and update the parameters accordingly.

[0109] wherein the model parameters at least include the mean value, covariance matrix and color of the initialized Gaussian distribution; the architecture of the three-dimensional Gaussian splatting model is LangSplat.

[0110] S3 Training and application of semantic segmentation model: confirm the architecture of the semantic segmentation model, collect a large number of indoor scene labeled images, train the model with cross-entropy loss function, and apply the trained semantic segmentation model to the three-dimensional scene reconstructed by three-dimensional Gaussian splatting, to assign accurate semantic labels to each pixel or three-dimensional Gaussian distribution.

[0111] wherein the semantic segmentation model includes U-Net or DeepLab.

[0112] S4 Semantic navigation implementation: the user inputs the navigation target through voice command or graphical interface, analyzes the navigation target according to the trained semantic segmentation model, determines the target semantic category and the region, locates the three-dimensional coordinates of the target object according to the target semantic label and position information, and performs path planning, navigation execution and implementation update based on the located three-dimensional coordinates.

[0113] In this embodiment, specifically includes:

[0114] (I) Indoor environment information collection

[0115] 1. Data collection device selection

[0116] Select a mobile device or robot with a high-precision RGB-D camera, with a resolution that ensures clear images can be obtained to accurately reconstruct the three-dimensional scene, such as selecting a camera with a resolution of not less than 640x480.

[0117] 2. Collection method

[0118] The mobile device or robot moves indoors according to the predetermined path to collect RGB images and depth images at a stable frame rate (such as 30 frames / second), ensuring coverage of the entire indoor space and avoiding blind spots.

[0119] To enrich data diversity, collect multiple times at different heights (such as 0.5 meters, 1 meter, 1.5 meters), multiple angles (every 30 degrees horizontally), and different lighting conditions (such as natural light during the day and light at night), to cope with complex indoor environments.

[0120] (II) Training of three-dimensional Gaussian splatting model

[0121] 1. Data preprocessing

[0122] Accurately register the collected RGB images and depth images to ensure that the pixel and depth information match without error.

[0123] Perform normalization to map pixel values to the [0,1] range, improving model training efficiency and stability.

[0124] Use methods such as median filtering to remove depth image noise and improve depth data accuracy.

[0125] 2. Model architecture selection

[0126] Use advanced three-dimensional Gaussian splatting model architectures such as LangSplat, which perform exceptionally well in processing three-dimensional scene data and reconstruction.

[0127] 3. Training process

[0128] Initialize model parameters, including the mean μ, covariance matrix Σ, and color of the Gaussian distribution.

[0129] Input the preprocessed image data into the model to optimize the model parameters using the stochastic gradient descent method, calculate the mean square error between the predicted scene and the actual scene and update the parameters accordingly, where N is the number of samples, y iTrue value, Predicted value.

[0130] Data augmentation techniques such as random cropping, rotation, and flipping images are used to increase the diversity of training data. At the same time, the validation set is used to monitor model performance and prevent overfitting. Training is stopped when the performance of the validation set no longer improves.

[0131] (Three) Semantic Segmentation

[0132] 1. Semantic segmentation model selection and training

[0133] Select deep learning models suitable for indoor scenes such as U-Net, DeepLab, etc., which have excellent performance in semantic segmentation tasks.

[0134] Collect a large number of indoor scene labeled images, covering various types of furniture, appliances, doors, windows, etc.

[0135] Use cross-entropy loss function Train the model, where C is the number of categories, y i is the true category label, is the model prediction probability, update the model parameters through backpropagation, and adjust the training parameters according to the validation set to optimize the model effect.

[0136] 2. Semantic label assignment

[0137] Apply the trained semantic segmentation model to the three-dimensional Gaussian splash reconstruction of the three-dimensional scene, and assign accurate semantic labels to each pixel or three-dimensional Gaussian distribution, such as marking the part belonging to the table as "table" category, realizing scene semanticization, and laying the foundation for semantic navigation.

[0138] The implementation process of semantic navigation specifically includes:

[0139] 1. Target recognition and positioning

[0140] Users input navigation targets through voice commands or graphical interfaces, such as "navigate to the living room sofa." The system uses semantic understanding technology to analyze the instructions and determine the target semantic category ("sofa") and the area it is in ("living room").

[0141] In the three-dimensional semantic scene, according to the target semantic label and location information, quickly locate the three-dimensional coordinates of the target object, such as searching for "sofa" category objects in the semantic segmented scene, and combining with the room layout to determine its specific location in the living room.

[0142] 2. Path planning

[0143] Based on the three-dimensional scene semantic information and target position, the A* path planning algorithm is used to calculate the optimal path. During planning, obstacles (identified through semantic labels) and passable areas (such as corridors) are fully considered to ensure safe and efficient paths.

[0144] The planned path is converted into a sequence of actions executable by the mobile device or robot, such as forward, backward, turn, etc.

[0145] 3. Navigation execution and real-time updating

[0146] The mobile device or robot executes navigation according to the planned path, and collects surrounding environment image data in real time during movement.

[0147] The new data is input into the three-dimensional Gaussian splash and semantic segmentation model to update the three-dimensional scene representation and semantic information in real time. According to the updated information, the path planning is adjusted in time to respond to dynamic changes in the environment (such as furniture movement), ensuring accurate and reliable navigation.

[0148] The technical measures include:

[0149] (I) Data collection optimization

[0150] 1. Multi-sensor fusion

[0151] Fusion of sensor data such as laser radar and inertial measurement unit (IMU). Laser radar provides accurate distance information, especially suitable for long-distance and low-light scenes; IMU obtains device attitude information. Through sensor fusion technology, the advantages of different sensor data are integrated to improve data quality and reliability.

[0152] 2. Adaptive collection strategy

[0153] According to the characteristics of indoor environment and navigation requirements, increase the collection frequency and coverage in complex areas (such as furniture-intensive areas) or key areas (such as entrances and exits), and reduce the collection frequency in simple and open areas to improve collection efficiency.

[0154] (II) Model training improvement

[0155] 1. Transfer learning

[0156] Use the three-dimensional Gaussian splash and semantic segmentation model pre-trained on a large-scale general data set to fine-tune on a specific indoor scene data set, accelerate model training, reduce the need for labeled data, and improve the performance of the model in specific scenarios.

[0157] 2. Model compression and acceleration

[0158] Use pruning, quantization, and other model compression techniques to reduce model parameter quantity and computational complexity, improve running efficiency, and meet the real-time semantic navigation requirements of resource-constrained devices (such as mobile robots).

[0159] (Three) Semantic Navigation Enhancement

[0160] 1. Multi-modal Interaction

[0161] Support multi-modal interaction methods such as voice commands, graphical user interface (GUI), and gesture recognition. Users can select targets by touching the screen or indicating directions with gestures, providing a more flexible and convenient user experience, suitable for various usage scenarios.

[0162] 2. Intelligent Obstacle Avoidance Strategy

[0163] Combine semantic information with real-time perception data and use an intelligent obstacle avoidance strategy. Consider the semantic attributes of obstacles (movable or immovable) to make decisions, try to bypass movable obstacles (such as chairs), and strictly avoid immovable obstacles (such as walls). At the same time, based on obstacle movement trend prediction, adjust the navigation path in advance to ensure safe navigation.

[0164] The technical improvement points include:

[0165] (I) Improve the adaptability of the three-dimensional Gaussian splash model to complex indoor scenes

[0166] 1. Improve the model architecture

[0167] Optimize the architecture of the three-dimensional Gaussian splash model for complex indoor object structures and lighting changes. Add a local feature attention module to better capture furniture details and textures; introduce a lighting processing unit to adapt to scene reconstruction under different lighting conditions, improving the model's representation ability for complex indoor scenes.

[0168] 2. Optimize the training algorithm

[0169] Use adaptive learning rate adjustment strategies such as Adagrad, Adadelta, etc. to automatically adjust the learning rate based on data characteristics, improve model convergence speed and stability, and reduce gradient problems in complex scene training.

[0170] (II) Improve the accuracy and efficiency of semantic segmentation

[0171] 1. Improve network structure

[0172] Design a semantic segmentation network structure suitable for indoor scenes, combining indoor object layout and semantic relationship characteristics. Use a hierarchical structure to first coarsely segment the scene into large areas (such as rooms, corridors), and then finely segment each area to improve accuracy and efficiency. At the same time, introduce a context information fusion module to enhance the model's understanding of object semantic associations and reduce missegmentation.

[0173] 2. Optimize labeling strategy

[0174] Semi-supervised learning method is used to train with a small amount of labeled data and a large amount of unlabeled data. Automatic labeling of some data is achieved through model prediction results and consistency constraints, reducing the workload of manual labeling. Labeling quality evaluation indicators are established to evaluate and correct the labeled data in real time, ensuring accuracy.

[0175] (III) Enhancing the real-time and reliability of semantic navigation

[0176] 1. Real-time scene updating mechanism

[0177] An efficient real-time scene updating mechanism is established, using incremental updating algorithm to process only the changed part, reducing the consumption of computing resources. Parallel computing technology (such as multi-threading, GPU acceleration) is used to parallelize data processing tasks, improving the updating speed and meeting the real-time navigation requirements.

[0178] 2. Robust path planning algorithm

[0179] The path planning algorithm is improved by introducing a probability model that considers obstacle movement probability and environmental uncertainty to plan fault-tolerant paths. Path re-planning strategy is established to quickly re-plan the path when unexpected situations (such as road blockage) occur, ensuring the smooth completion of navigation tasks.

[0180] The working principle and working process specifically include:

[0181] 1. Data acquisition stage

[0182] The camera collects real-time RGB image and depth image data of the indoor environment, which is transmitted to the processor through the data transmission line. The processor performs preliminary processing such as format conversion and caching.

[0183] 2. Model processing stage

[0184] The processor calls the three-dimensional Gaussian splash model training module to input the collected data into the model for training or scene reconstruction. The model calculates the three-dimensional Gaussian distribution parameters based on the algorithm and parameters to construct the three-dimensional scene representation.

[0185] The processor inputs the reconstructed three-dimensional scene data into the semantic segmentation model, which performs semantic segmentation on the scene based on the training parameters and assigns semantic labels to each part.

[0186] 3. Navigation stage

[0187] The user inputs the navigation target through the interactive device (voice input, touch screen, etc.), and the interactive device transmits the instruction to the processor. The processor analyzes the instruction and uses the semantic navigation module to locate the target and plan the path in the three-dimensional semantic scene.

[0188] The path planning result is converted into a control signal, and the mobile device or robot is controlled to move by a driving module. During navigation, the camera continuously collects data, the processor updates the three-dimensional scene and semantic information in real time, the navigation path is adjusted according to the change of the environment, and the target position is accurately reached.

[0189] As can be seen from the above, the 3D Gauss containing semantic feature embedding is used to represent the 3D scene, the semantic feature is encoded on each 3D Gauss, and the three-dimensional semantic field is constructed in this way, so that the model can respond to the text-based query. Compared with the method based on NeRF, the rendering process is faster.

[0190] Please refer to Figure 2 , which shows a block diagram of a semantic navigation system based on three-dimensional Gaussian splash provided by an embodiment of the application. As Figure 2 shown, the system can include:

[0191] The data acquisition and preprocessing module is used for acquiring RGB images and depth images at a stable frame rate according to a predetermined path by using a mobile device with a high-precision RGB-D camera, and ensuring that the entire indoor space is covered, and pre-processing the acquired RGB images and depth images;

[0192] The three-dimensional Gaussian splash model training module is used for confirming the three-dimensional Gaussian splash model architecture, initializing the model parameters, inputting the pre-processed image data into the model, optimizing the model parameters by using the stochastic gradient descent method, calculating the mean square error of the predicted scene and the actual scene, and updating the parameters in reverse according to the error; wherein the model parameters at least include the mean value, the covariance matrix and the color of the initialized Gaussian distribution; the three-dimensional Gaussian splash model architecture is LangSplat;

[0193] The semantic segmentation model training and application module is used for confirming the semantic segmentation model architecture, collecting a large number of indoor scene labeled images, training the model with a cross-entropy loss function, applying the trained semantic segmentation model to the three-dimensional scene reconstructed by the three-dimensional Gaussian splash, and giving accurate semantic labels to each pixel or three-dimensional Gaussian distribution; wherein the semantic segmentation model includes U-Net or DeepLab;

[0194] The semantic navigation implementation module is used for inputting the navigation target by voice instructions or a graphical interface, analyzing the navigation target according to the trained semantic segmentation model, determining the target semantic category and the region where the target semantic category is located, positioning the three-dimensional coordinates of the target object according to the target semantic label and the position information, and performing path planning, navigation execution and implementation update based on the positioned three-dimensional coordinates.

[0195] The specific definitions of the semantic navigation system based on three-dimensional Gaussian splatting can refer to the definitions of the semantic navigation method based on three-dimensional Gaussian splatting in the above, which will not be repeated here. Each module in the above semantic navigation system based on three-dimensional Gaussian splatting can be realized by software, hardware and combinations thereof, in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0196] In one embodiment, an electronic device, which can be a computer, has an internal structure diagram as shown in Figure 3 The electronic device includes a processor, a memory and a network interface connected by a system bus. The processor of the device is used to provide computing and control capabilities. The memory of the device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is used for semantic navigation data based on three-dimensional Gaussian splatting. The network interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a semantic navigation method based on three-dimensional Gaussian splatting.

[0197] Those skilled in the art can understand that the structure shown in Figure 3 is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0198] In one embodiment, a computer readable storage medium having a computer program stored thereon is also provided, which relates to all or part of the processes in the above-mentioned embodiment methods.

[0199] In one embodiment, a computer program product including computer programs / instructions is also provided, which relates to all or part of the processes in the above-mentioned embodiment methods.

[0200] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, databases, or other media in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (SyMchliMk) DRAM (SLDRAM), memory bus (RaMbus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0201] Any combination of the above-described technical features of the embodiments can be combined. In order to make the description simple, all possible combinations of the technical features in the above-described embodiments are not described, but as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0202] The above-described embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the patent scope of the application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the patent protection scope of the present application should be subject to the appended claims.

Claims

1. A semantic navigation method based on three-dimensional Gaussian splashing, characterized in that: The method comprises: Use a mobile device equipped with a high-precision RGB-D camera to capture RGB and depth images along a predetermined path at a stable frame rate, ensuring coverage of the entire indoor space. Preprocess the captured RGB and depth images. Confirm the 3D Gaussian splat model architecture, initialize the model parameters, input the preprocessed image data into the model, optimize the model parameters using stochastic gradient descent, calculate the mean squared error between the predicted scene and the actual scene, and backpropagate the updated parameters accordingly. The model parameters include at least the mean, covariance matrix, and color of the initialized Gaussian distribution. The 3D Gaussian splat model architecture is LangSplat. Confirm the semantic segmentation model architecture, collect a large number of annotated images of indoor scenes, train the model using the cross-entropy loss function, and apply the trained semantic segmentation model to the 3D scene reconstructed by 3D Gaussian splatting to assign accurate semantic labels to each pixel or 3D Gaussian distribution. Semantic segmentation models include U-Net or DeepLab. The user inputs the navigation target through voice commands or a graphical interface. The navigation target is parsed according to the trained semantic segmentation model, the target semantic category and the area where it is located are determined, and the three-dimensional coordinates of the target object are located based on the target semantic label and position information. Path planning, navigation execution, and implementation updates are performed based on the located three-dimensional coordinates.

2. The semantic navigation method according to claim 1, characterized in that: Path planning, navigation execution, and implementation updates based on the three-dimensional coordinates of the positioning include: Based on the semantic information of the 3D scene and the target position, the A* path planning algorithm is used to calculate the optimal path and convert it into an executable action sequence for the mobile device or robot. The mobile device or robot navigates according to the planned path, collects image data of the surrounding environment in real time, and inputs the new data into the 3D Gaussian splatter and semantic segmentation model, updating the 3D scene representation and semantic information in real time, and adjusting the path planning in a timely manner based on the updated information.

3. The semantic navigation method according to claim 1, characterized in that: Use a mobile device equipped with a high-precision RGB-D camera to capture RGB and depth images at a stable frame rate along a predetermined path, ensuring coverage of the entire indoor space. Preprocess the captured RGB and depth images, including: Multiple acquisitions were made at different heights, angles, and lighting conditions; the heights included at least 0.5 meters, 1 meter, and 1.5 meters, the angles included 30-degree intervals horizontally, and the lighting conditions included both daytime natural light and nighttime lighting; Perform accurate registration of image and depth information, normalize the image, and use median filtering to remove depth image noise.

4. The semantic navigation method according to claim 1, characterized in that: Feeding the preprocessed image data into the model also includes: Use data augmentation techniques, including random cropping, rotation, and flipping of images, to increase the diversity of training data, and use validation sets to monitor model performance and prevent overfitting.

5. The semantic navigation method according to claim 1, characterized in that: Optimize model parameters using stochastic gradient descent, including: Calculate the mean square error between the predicted scenario and the actual scenario And update the parameters in reverse propagation, where N is the number of samples, y i is the true value, is the predicted value.

6. The semantic navigation method according to claim 1, characterized in that: The model is trained with a cross entropy loss function, including: Using cross entropy loss function Training model, where C is the number of categories, y i is the true value, To predict the value, the model parameters are updated through back propagation, and the training parameters are adjusted according to the validation set to optimize the model effect.

7. A semantic navigation system based on three-dimensional Gaussian splashing, characterized in that: The system comprises: The data acquisition and preprocessing module uses a mobile device equipped with a high-precision RGB-D camera to capture RGB and depth images along a predetermined path at a stable frame rate, ensuring coverage of the entire indoor space, and preprocesses the captured RGB and depth images. The 3D Gaussian splat model training module is used to confirm the 3D Gaussian splat model architecture, initialize the model parameters, input the preprocessed image data into the model, optimize the model parameters using stochastic gradient descent, calculate the mean squared error between the predicted scene and the actual scene, and backpropagate the updated parameters accordingly. The model parameters include at least the mean, covariance matrix, and color of the initialized Gaussian distribution. The 3D Gaussian splat model architecture is LangSplat. The semantic segmentation model training and application module is used to confirm the semantic segmentation model architecture, collect a large number of annotated indoor scene images, train the model using the cross-entropy loss function, and apply the trained semantic segmentation model to 3D scenes reconstructed using 3D Gaussian splattering, assigning accurate semantic labels to each pixel or 3D Gaussian distribution. Semantic segmentation models include U-Net or DeepLab. In the semantic navigation implementation module, users input navigation targets through voice commands or a graphical interface. The module then parses the navigation targets based on the trained semantic segmentation model, determines the target semantic category and the area where it is located, and locates the three-dimensional coordinates of the target object based on the target semantic label and position information. Path planning, navigation execution, and implementation updates are performed based on the located three-dimensional coordinates.

8. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images