Semantic navigation method and system based on three-dimensional Gaussian splashing

Abstract

The invention discloses a semantic navigation method and system based on three-dimensional Gaussian splashing. The invention aims to improve the navigation precision and efficiency of the robot in an indoor environment. According to the method, an indoor environment image is acquired by using a high-precision RGB-D camera through data acquisition and preprocessing, and then a three-dimensional scene containing geometric, texture and appearance information is constructed through training of a three-dimensional Gaussian splash model. Furthermore, a deep learning model is adopted for semantic segmentation, and accurate semantic tags are given to all parts in the scene. And finally, in a semantic navigation realization stage, the system quickly positions a target object according to a target input by a user, and plans an optimal path to realize accurate navigation. According to the invention, the three-dimensional semantic field can be updated in real time to adapt to the dynamic change of the indoor environment, and the real-time performance and reliability of navigation are improved.

Description

Technical Field

[0001] The present invention relates to the field of semantic navigation technology, and in particular to a semantic navigation method and system based on three-dimensional Gaussian splashing. Background Art

[0002] With the development of artificial intelligence, sensor technology, and robotics, embodied intelligence technology gives robots a "body" that enables them to perceive the environment, understand tasks, and autonomously navigate and perform tasks in complex environments. Especially in home environments, the accurate navigation capabilities of mobile robots or humanoid robots are becoming increasingly important.

[0003] Traditional two-dimensional semantic maps cannot fully present the three-dimensional structure of the environment, and there are problems such as information loss, labeling errors, ambiguity, etc., which limit the accuracy and efficiency of navigation. Semantic segmentation technology has deficiencies in accuracy, details and individual distinction, and has a high computational load. Patent CN111645073B proposes a robot visual semantic navigation method, but it has problems such as dependence on semantic maps, large computing resource requirements, voice command limitations, poor environmental adaptability, insufficient model update and generalization capabilities, and complexity of semantic understanding. Summary of the invention

[0004] Based on this, the embodiments of the present application provide a semantic navigation method and system based on three-dimensional Gaussian splatting, which aims to achieve accurate and efficient open text query by constructing a three-dimensional semantic field, and apply it to fields such as indoor navigation to solve the limitations of existing technologies in speed and accuracy.

[0005] In a first aspect, a semantic navigation method based on three-dimensional Gaussian splashing is provided, the method comprising:

[0006] Use a mobile device with a high-precision RGB-D camera to collect RGB images and depth images at a stable frame rate along a predetermined path, ensure that the entire indoor space is covered, and pre-process the collected RGB images and depth images;

[0007] Confirm the three-dimensional Gaussian splash model architecture, initialize the model parameters, input the preprocessed image data into the model, optimize the model parameters using the stochastic gradient descent method, calculate the mean square error between the predicted scene and the actual scene, and back-propagate and update the parameters accordingly; where the model parameters at least include the mean, covariance matrix, and color of the initialized Gaussian distribution; the three-dimensional Gaussian splash model architecture is LangSplat;

[0008] Confirm the semantic segmentation model architecture, collect a large number of annotated images of indoor scenes, train the model with the cross entropy loss function, apply the trained semantic segmentation model to the 3D scene reconstructed by 3D Gaussian splashing, and assign accurate semantic labels to each pixel or 3D Gaussian distribution; the semantic segmentation model includes U-Net or DeepLab;

[0009] The user inputs the navigation target through voice commands or a graphical interface. The navigation target is parsed according to the trained semantic segmentation model, the target semantic category and the area where it is located are determined, and the three-dimensional coordinates of the target object are located based on the target semantic label and position information. Path planning, navigation execution and implementation updates are performed based on the located three-dimensional coordinates.

[0010] Optionally, path planning and navigation execution and implementation update are performed based on the positioned three-dimensional coordinates, specifically including:

[0011] Based on the semantic information of the three-dimensional scene and the target position, the A* path planning algorithm is used to calculate the optimal path and convert it into an executable action sequence for the mobile device or robot; the mobile device or robot performs navigation according to the planned path, collects image data of the surrounding environment in real time, and inputs the new data into the three-dimensional Gaussian splash and semantic segmentation model, updates the three-dimensional scene representation and semantic information in real time, and adjusts the path planning in time according to the updated information.

[0012] Optionally, a mobile device with a high-precision RGB-D camera is used to collect RGB images and depth images at a stable frame rate along a predetermined path, and ensure that the entire indoor space is covered, and the collected RGB images and depth images are pre-processed, including:

[0013] Multiple acquisitions were made at different heights, angles and lighting conditions; the heights included at least 0.5m, 1m and 1.5m, the angles included 30-degree intervals in the horizontal direction, and the lighting conditions included natural light during the day and lights at night;

[0014] Perform accurate registration and normalization of the image and depth information, and use median filtering to remove depth image noise.

[0015] Optionally, inputting the preprocessed image data into the model further comprises:

[0016] Use data augmentation techniques, including random cropping, rotation, and flipping of images, to increase the diversity of training data, and use validation sets to monitor model performance and prevent overfitting.

[0017] Optionally, the model parameters are optimized using stochastic gradient descent, including:

[0018] Calculate the mean square error between the predicted scenario and the actual scenario And update the parameters in reverse propagation, where N is the number of samples and yi is the true value, is the predicted value.

[0019] Optionally, train the model with a cross entropy loss function, including:

[0020] Using the cross entropy loss function Training model, where C is the number of categories, y i is the true value, To predict the value, the model parameters are updated through back propagation, and the training parameters are adjusted according to the validation set to optimize the model effect.

[0021] In a second aspect, a semantic navigation system based on three-dimensional Gaussian splashing is provided, the system comprising:

[0022] The data acquisition and preprocessing module is used to use a mobile device with a high-precision RGB-D camera to collect RGB images and depth images at a stable frame rate along a predetermined path, ensure coverage of the entire indoor space, and preprocess the collected RGB images and depth images;

[0023] The 3D Gaussian splash model training module is used to confirm the 3D Gaussian splash model architecture, initialize the model parameters, input the preprocessed image data into the model, optimize the model parameters using the stochastic gradient descent method, calculate the mean square error between the predicted scene and the actual scene, and back-propagate the updated parameters accordingly; the model parameters at least include the mean, covariance matrix, and color of the initialized Gaussian distribution; the 3D Gaussian splash model architecture is LangSplat;

[0024] The semantic segmentation model training and application module is used to confirm the semantic segmentation model architecture, collect a large number of annotated images of indoor scenes, train the model with a cross-entropy loss function, apply the trained semantic segmentation model to the 3D scene reconstructed by 3D Gaussian splashing, and assign accurate semantic labels to each pixel or 3D Gaussian distribution; the semantic segmentation model includes U-Net or DeepLab;

[0025] In the semantic navigation implementation module, the user inputs the navigation target through voice commands or a graphical interface. The navigation target is parsed according to the trained semantic segmentation model, the target semantic category and the area where it is located are determined, and the three-dimensional coordinates of the target object are located according to the target semantic label and position information; path planning, navigation execution and implementation updates are performed based on the located three-dimensional coordinates.

[0026] In a third aspect, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the semantic navigation method described in any one of the first aspects is implemented.

[0027] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the semantic navigation method described in any one of the first aspects is implemented.

[0028] In a fifth aspect, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements any of the semantic navigation methods described in the first aspect.

[0029] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0030] With the help of three-dimensional Gaussian splashing and semantic segmentation, the boundaries of objects can be accurately outlined. The visualization results of the LERF dataset show that the present invention is superior to traditional methods. For example, in complex indoor scenes, the boundaries of furniture and decorations can be clearly distinguished to avoid confusion, providing a reliable basis for operations such as navigation, which is of great significance for the precise operation of robots.

[0031] SAM is used to obtain multi-scale semantic levels and fully grasp the structural relationship of objects. Taking a table as an example, it can identify the whole, tabletop, table legs and texture, etc., improve semantic understanding ability, help plan paths more accurately during navigation, and improve safety and accuracy.

[0032] Combine semantic information to plan and optimize paths, identify traversable areas and obstacles, reduce detours in complex layouts, and improve navigation efficiency and fluency.

[0033] The 3D semantic field is updated in real time to adapt to dynamic changes in the room, such as furniture movement and people walking around, ensuring accurate and reliable navigation. It is suitable for scenes such as offices and conference rooms.

[0034] It has good adaptability to lighting changes and occlusion, overcomes the problems of uneven lighting and object occlusion, maintains stable navigation function in complex actual environments, and improves system robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0036] Figure 1 A flowchart of a semantic navigation method based on three-dimensional Gaussian splashing provided in an embodiment of the present application;

[0037] Figure 2 A block diagram of a semantic navigation system based on three-dimensional Gaussian splashing provided in an embodiment of the present application;

[0038] Figure 3 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0040] In the description of the present invention, the terms "comprises", "has" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may also include other steps or units that are not explicitly listed but are inherent to these processes, methods, products or apparatuses, or steps or units added based on further optimization schemes conceived by the present invention.

[0041] With the rapid advancement of science and technology, embodied intelligence technology, which combines knowledge from multiple fields such as artificial intelligence, sensor technology, and robotics, is constantly developing and improving. The core of this technology is to give robots a "body" so that they can perceive the environment and understand tasks like humans, and autonomously navigate and perform tasks in complex environments. Especially in home environments, the accurate navigation capabilities of mobile robots or humanoid robots are becoming increasingly important.

[0042] In modern home life, people's demand for intelligent services is growing. Whether it is a cleaning robot, a companion robot or a home security monitoring robot, they all need to be able to move freely in the home environment. This not only involves the robot's recognition and memory of space, but also includes the ability to adapt to dynamic environments, such as avoiding sudden obstacles, identifying the location of family members, and predicting and planning the best path.

[0043] 3D Gaussian splatter:

[0044] With the continuous development of computer graphics, computer vision, virtual reality and other fields, the demand for efficient and realistic presentation of 3D scenes is growing. Traditional 3D modeling and rendering methods have limitations in some aspects, such as time-consuming and labor-intensive modeling of complex scenes and insufficient rendering effects. In this context, 3D Gaussian splattering technology came into being, which aims to handle the representation and rendering of 3D scenes in a more flexible, efficient and high-quality visual effect way.

[0045] 3D Gaussian splatting uses Gaussian distribution function as the core building element. Each Gaussian distribution represents a local feature in the scene. By combining and adjusting multiple Gaussian distributions, various shapes, colors, lighting and other information in the 3D scene can be accurately described. This Gaussian-based expression enables it to delicately capture the details of the scene, whether it is the surface texture of the object or the changes in light and shadow. Compared with some traditional 3D model representation methods, such as polygonal meshes, 3D Gaussian splatting has better storage efficiency. It does not need to store a large amount of complex geometric mesh information, but represents the scene through relatively simple Gaussian parameters (such as mean, covariance, etc.). In terms of calculation, its rendering process utilizes the characteristics of Gaussian distribution, which can achieve fast calculation to a certain extent, reducing the time and resource consumption required for rendering. The technology can adaptively adjust the density and accuracy of Gaussian distribution according to factors such as the viewing angle and the importance of the scene. When the viewing angle is close to an object, it can increase the density of Gaussian distribution in the area, thereby presenting finer details; while at a farther angle or in less important areas, the density is appropriately reduced to save resources, while still maintaining the visual coherence of the overall scene.

[0046] The current application scenarios of 3D Gaussian splash are mainly concentrated in the fields of virtual reality (VR) and augmented reality (AR), film and television, game modeling and architecture. In VR and AR experiences, users have extremely high requirements for the immersion and realism of the scene. 3D Gaussian splash technology can quickly generate realistic 3D scenes, making users feel as if they are in the scene. For example, in VR games, it can be used to create delicate game scenes, from fantasy castles to dense forests, every detail can be presented with high quality, enhancing the player's gaming experience. In AR applications, when virtual objects are superimposed on real scenes, it can ensure that the rendering effect of virtual objects matches the real environment, providing a more natural fusion effect.

[0047] Existing two-dimensional semantic map:

[0048] A 2D semantic map is a map format that annotates and describes the semantic information of the environment on a 2D plane. It plays an important role in many fields such as robot navigation, autonomous driving, and indoor positioning. A 2D semantic map is constructed based on a 2D plane coordinate system by dividing and annotating different areas and objects in the environment according to specific semantic categories. It not only presents the geometric shape of the environment, but more importantly, gives each part a clear semantic meaning.

[0049] Two-dimensional semantic maps present environmental information in an intuitive and easy-to-understand way. Whether it is a human operator or a robot or self-driving vehicle that needs to navigate based on the map, it is relatively easy to understand what the map conveys. For example, in indoor navigation, people can clearly see the layout of the room, the location of the door, and the path to the destination from the map. Two-dimensional semantic maps can be easily expanded and updated. As the perception of the environment deepens or new objects appear, new geometric shapes and semantic annotations can be added to the map at any time. For example, in an office, when a new device is purchased, it can be easily added to the map as a new geometric shape and given a corresponding semantic label.

[0050] For indoor service robots, industrial robots, etc., two-dimensional semantic maps are an important basis for them to navigate, avoid obstacles and complete tasks in complex environments. Based on the semantic information on the map, the robot can identify passable areas such as passages and doors, as well as objects that need to be avoided, such as tables and chairs, so as to plan a reasonable path and successfully complete the task. In indoor environments, the use of two-dimensional semantic maps combined with other positioning technologies (such as WiFi positioning, Bluetooth positioning, etc.) can achieve more accurate indoor positioning. By comparing the current location of the user or device with the annotation information on the map, the specific location can be determined. For example, in large shopping malls, customers can use mobile phone applications combined with two-dimensional semantic maps to find their floor, store location, etc.

[0051] Three-dimensional Gaussian splashing has the following advantages in navigation:

[0052] 1. New perspective rendering:

[0053] In navigation scenarios, whether it is an indoor environment (such as large shopping malls, office buildings, etc.) or an outdoor scene (such as city streets, parks, etc.), it is very important to be able to flexibly switch perspectives according to the needs of users or navigation devices. Based on its unique principle, the 3D Gaussian splash technology performs well in rendering from a new perspective. It represents each element in the 3D scene with a Gaussian distribution, and can quickly and realistically re-render the full picture of the scene from different perspectives. For example, when pedestrians navigate inside a shopping mall, they may need to switch from a head-on perspective to a bird's-eye view in order to better view the layout of the entire floor and the location relationship of each store. 3D Gaussian splash can easily achieve this perspective change, and can delicately present details such as mall decorations, shelf placement, and crowd distribution seen from different heights, just like looking down from the air or looking straight up, providing more intuitive and comprehensive visual information for navigation.

[0054] Navigation environments are often complex and diverse, containing objects of various shapes, colors, and materials. Three-dimensional Gaussian splashing can cope with this complexity well. For objects with irregular shapes, such as the unique appearance of old buildings and bizarre sculptures in parks, it can use Gaussian distribution to accurately capture their shape characteristics and fully present them when rendered from a new perspective. At the same time, for objects of different materials, such as the reflective effect of glass curtain walls and the texture of wooden furniture, their visual effects can also be realistically simulated by adjusting the parameters of the Gaussian function. This allows users to clearly perceive the true appearance of the surrounding environment during navigation, and obtain accurate visual guidance whether in bright daytime or dim night, so as to complete the navigation task more smoothly.

[0055] 2. Multi-frame consistency:

[0056] During navigation, especially for dynamic scenes (such as pedestrians walking on the street, vehicles driving on the road, etc.), it is necessary to ensure that the scene presentation is consistent in multiple consecutive frames. Three-dimensional Gaussian splashing technology has significant advantages in this regard. Since it uses Gaussian distribution to describe the elements in the scene, between different frames, as long as the objects in the scene do not undergo substantial changes (such as position movement, shape change, etc.), the basic features of the scene in each frame can be ensured to remain consistent through continuous processing and rendering of the same Gaussian distribution. For example, when an autonomous driving car is driving, it continuously obtains images of the surrounding environment through the on-board camera for navigation. Three-dimensional Gaussian splashing can ensure that the appearance and position relationship of roads, buildings, traffic signs, etc. remain stable in these consecutive image frames, and there will be no sudden disappearance or deformation of a corner of a building in the previous frame, etc., which provides a reliable visual reference for the autonomous driving system, helping it to accurately judge road conditions and make correct driving decisions.

[0057] In navigation scenarios, in addition to static environmental objects, dynamic objects are often encountered, such as pedestrians, other vehicles, etc. The multi-frame consistency feature of 3D Gaussian splashing helps to track these dynamic objects.

[0058] By maintaining the recognition and processing of the Gaussian distribution corresponding to the same dynamic object in consecutive frames, the motion trajectory of the dynamic object can be clearly observed. For example, in the navigation scene of the airport hall, the staff can use the three-dimensional Gaussian splash technology through the monitoring equipment to track the walking route of the passengers. Even if the passengers shuttle through the crowd, change the walking direction, etc., they can accurately identify the Gaussian distribution corresponding to the passenger in each frame, so as to grasp their movements in real time, which facilitates the work of providing navigation guidance or security monitoring.

[0059] 3. A three-dimensional semantic graph can be constructed:

[0060] Three-dimensional semantic graphs can provide rich semantic information for navigation, and when three-dimensional Gaussian splashing is combined with three-dimensional semantic graphs, its advantages can be better utilized.

[0061] In the navigation environment, each Gaussian distribution can not only represent the geometric shape and appearance characteristics of the object, but also be assigned specific semantic labels, such as "wall", "door", "road", "vehicle", "pedestrian", etc. In this way, during the navigation process, both human users and navigation devices (such as self-driving cars, service robots, etc.) can plan paths and avoid obstacles more specifically based on this semantic information.

[0062] For example, in indoor office environment navigation, through the 3D semantic map with semantic labels generated by 3D Gaussian splashing, the service robot can easily identify which areas are passages, which are offices, which are meeting rooms, etc., so as to accurately find the target location and avoid obstacles such as tables and chairs, and complete the task efficiently.

[0063] When a 3D semantic map is constructed based on 3D Gaussian splash, it can provide a more accurate spatial understanding. Since Gaussian distribution can delicately describe the shape and position of an object, combined with semantic labels, it can allow navigation users or devices to have a deeper understanding of the spatial structure of the entire navigation environment. For example, in the fields of urban planning and architectural design, designers can use the 3D semantic map generated by this technology to evaluate the impact of building layout on the passage of pedestrians, vehicles, etc. For example, by observing the 3D Gaussian splash model with semantic labels, you can clearly see the connection between roads and buildings, the channel settings between different floors, etc., so as to optimize the spatial layout and improve the navigation convenience of the entire environment.

[0064] The main content of patent CN111645073B is a robot visual semantic navigation method, device and system. The abstract of the invention is as follows:

[0065] The present invention relates to the field of robot navigation technology, and discloses a robot visual semantic navigation method, comprising the following steps: establishing a scene image set and a voice instruction set; marking the image features of each scene image in the scene image set, marking the voice features of each voice instruction in the voice instruction set; combining the image features and voice features at the same time to construct a semantic map to obtain a semantic map set, marking the semantic features of each semantic map in the semantic map set; fusing the image features, voice features and semantic features at the same time to construct a state vector to obtain a state vector set; marking the action sequence corresponding to each state vector in the state vector set, using the state vector set as a training sample to train a deep reinforcement learning model to obtain a navigation model; and controlling the robot according to the navigation model. The present invention can realize the navigation of objects that are not within the robot's field of view.

[0066] The advantages of the invention are as follows:

[0067] Realize navigation of objects that are not in the field of view: By establishing a navigation model that includes semantic features, combining image features, speech features and extracted semantic features, it is possible to understand the association between different objects, thereby establishing the relationship between the target object and the objects existing in the scene image for the robot, and realizing navigation based on semantic prior knowledge.

[0068] Integrate deep reinforcement learning with knowledge graphs: Knowledge graphs can encode various forms of knowledge, making it possible for semantic prior knowledge to participate in the training of deep reinforcement learning models, so that navigation models can accurately define the relationship between the target object and other objects in the scene based on semantic features, improve the generalization ability of invisible environments and target objects, and improve navigation performance.

[0069] A variety of modeling methods are used to improve efficiency: for example, when extracting real-time image features, the established scene image set is used to train the visual neural network to obtain the image feature extraction model, without the need to rebuild the model, which improves efficiency; when extracting real-time semantic features, the established semantic atlas set is also used for model training.

[0070] Use pre-trained models to speed up training: When training the image feature extraction model, first use the ImageNet dataset to pre-train the ResNet-50 network to obtain a pre-trained model with certain object recognition capabilities, and then use the scene image set for training, which reduces the difficulty of training and reduces the quantity and quality requirements of the scene image set.

[0071] Real-time navigation control: After the navigation model is established, the real-time image, voice and semantic features can be extracted according to the real-time scene image, voice commands and semantic map to obtain the real-time state vector. The real-time action sequence is input into the navigation model to control the robot and help it find the target object.

[0072] Improve the representation capabilities of instruction features and map features: By establishing the correspondence between relevant objects in the instruction and the semantic map, the alignment and fusion of the two modalities (text and vision) are achieved.

[0073] 3. Defects of Background Technology

[0074] There are many shortcomings in two-dimensional semantic maps. As a two-dimensional plane map, it cannot fully present the three-dimensional structure of the environment. For some environments with height changes, such as the interior of a multi-story building, sloping terrain, etc., the two-dimensional semantic map can only provide information on the plane, which may cause some information to be lost and affect the comprehensive understanding of the environment. For example, in actual scenes, objects are often stacked on the table, such as a stack of books, a pile of documents, etc. Since the two-dimensional semantic map has no height information, for these stacked objects, it is impossible to determine which layer belongs to which specific object, nor is it clear how they are arranged in the vertical direction. For example, a stack of five books with different contents may only appear as a fuzzy rectangle or polygon on the two-dimensional semantic map (representing the projection of the entire stack of books on the table), and each book cannot be segmented and identified as an independent individual because there is no height information to assist in distinguishing their specific position and belonging in the stack.

[0075] Although the accuracy of semantic annotation is continuously improved through various technical means, there are still cases of incorrect or inaccurate annotation. For example, due to the limited resolution of visual sensors and the limitations of image recognition technology, an object with similar shape but different semantics may be incorrectly labeled, or the semantic category of complex objects cannot be accurately subdivided when annotating them.

[0076] At the same time, two-dimensional semantic maps are ambiguous. When performing semantic annotation, some of the words used may be polysemous, which may lead to ambiguity in map interpretation. For example, the word "channel" may have different meanings in different environments. In an indoor environment, it may refer to a corridor between rooms for passage; in an outdoor environment, such as in a large park, it may refer to a path for pedestrians to stroll or a road where vehicles can drive. When it is marked as "channel" on a two-dimensional semantic map, it is difficult to accurately determine what type of channel it refers to without additional contextual information. In addition, ambiguity may also be caused by insufficient semantic segmentation. Sometimes, the marked semantic categories may not be detailed enough, causing ambiguity. For example, only "table" is marked, but in fact there may be many different types of tables in the environment, such as office desks, dining tables, laboratory tables, etc., which may differ in size, shape, purpose, etc. If the map user needs more precise information, such as looking for a table for a specific purpose, simply marking it as "table" will not meet the needs, and may have different understandings of the relevant areas on the map due to this general marking.

[0077] Semantic segmentation technology also has many shortcomings. First, traditional navigation technology that combines semantic information may have the problem of high computational load. Traditional semantic vSLAM systems have a large amount of computation when processing image features, especially in feature point extraction, descriptor generation, and pose estimation. At the same time, due to the high consumption of computing resources, it is difficult to achieve real-time semantic segmentation in many practical application scenarios, and real-time performance cannot be guaranteed.

[0078] In addition, semantic segmentation is deficient in accuracy and detail. When segmenting objects, especially the edges of objects, it is difficult to achieve very accurate segmentation. For example, when distinguishing a cat lying on the grass, the boundary between the hair on the edge of the cat's body and the surrounding grass may not be clear. The semantic segmentation model may not be able to accurately define the boundary pixels of the cat and the grass, resulting in a fuzzy segmentation result at the boundary, where some grass pixels are misclassified into the cat category, or some hair pixels on the edge of the cat are misclassified into the grass category. This is a common boundary fuzziness problem. In addition, for objects with complex and fine structures, semantic segmentation is difficult to fully and accurately present their details. Take the Gothic church in the architectural scene as an example. The facade of the church often has many exquisite carvings, complex spire structures, and small stained glass windows. The semantic segmentation model may blur these fine structures and cannot accurately segment each carving element, different parts of the spire, and the outline of the stained glass window. Instead, it roughly divides it as a whole architectural category.

[0079] At the same time, semantic segmentation itself has limitations. Semantic segmentation can provide pixel-level object types, but cannot distinguish different individuals of the same type, which limits its application scope in more complex scenes. Specifically: when objects of different categories are similar in appearance, color, texture and other features, the semantic segmentation model is prone to confusion. For example, in an image containing snow and white sheets drying, the white snow and white sheets may have similar colors and a certain degree of texture similarity. The model may mistakenly segment part of the snow into the sheet category, or vice versa, it is difficult to accurately distinguish these two objects with different semantics but similar visual features. At the same time, in real scenes, multiple categories of objects often overlap with each other, and it becomes extremely challenging to accurately segment each category. For example, in an image containing a person holding a flower, the person's hand and the flower partially overlap. It is difficult for the model to accurately separate the pixels of the hand and the flower, and determine which pixels belong to the hand and which belong to the flower. Segmentation errors or unclear segmentation often occur in the overlapping area.

[0080] The defects of patent CN111645073B are as follows:

[0081] Dependence on semantic graph: The effectiveness of the method depends to a certain extent on the construction and accuracy of the semantic graph. If the construction of the semantic graph is biased or incomplete, it may affect the accuracy of navigation.

[0082] Computing resource requirements: Deep reinforcement learning, neural network training, and real-time image, speech, and semantic feature processing may require more computing resources. For some robotic devices with limited computing power, there may be real-time or performance challenges.

[0083] Limitations of voice commands: The accuracy of understanding and converting voice commands into natural language information may be limited by speech recognition technology. Complex or unclear voice commands may result in inaccurate extracted speech features.

[0084] Environmental adaptability: Although this method takes into account the navigation of objects that are not within the field of view to a certain extent, its performance may be affected in some extreme or special environmental conditions, such as severe lack of image data and excessive noise interference.

[0085] Model update and generalization capability: In actual applications, the environment may change, and it is necessary to consider how to update the semantic graph and navigation model in a timely manner to maintain good generalization capability and navigation effect.

[0086] Complexity of semantic understanding: The semantic understanding of natural language is inherently complex and ambiguous. There may be situations where the semantics of some instructions or scenarios are difficult to accurately capture and express.

[0087] 4. Purpose of the present invention:

[0088] The purpose of the present invention is to construct a three-dimensional semantic field to achieve accurate and efficient open text query, and apply it to fields such as indoor navigation. Specifically, by combining three-dimensional Gaussian splattering technology, using SegmentAnything Model (SAM) to learn hierarchical semantics, and introducing semantic autoencoders for specific scenarios, the limitations of existing methods in speed and accuracy are solved, so that the model can respond to text-based queries, quickly and accurately locate and identify target objects in three-dimensional space, and provide support for indoor navigation. Compared with the NeRF-based method, the present invention improves the query speed and the accuracy of the semantic field while maintaining rendering efficiency, bringing better solutions to applications such as robot navigation, 3D editing, and augmented reality.

[0089] V. The solution of the present invention:

[0090] (1) Overview of the overall technical solution:

[0091] This invention focuses on accurate semantic navigation in indoor environments. The core lies in the integration of 3D Gaussian splash technology and semantic segmentation technology, combined with mature navigation methods. The overall solution covers key steps such as environmental information collection, 3D Gaussian splash model training, semantic segmentation, and final semantic navigation implementation. Each step works closely together to achieve efficient and accurate semantic navigation goals.

[0092] (2) Technical principle:

[0093] (I) Three-dimensional Gaussian splashing principle:

[0094] The 3D Gaussian splashing is performed by representing the 3D scene with a set of 3D Gaussian distributions, where each 3D Gaussian is represented by a mean and covariance matrix ∑, its expression is

[0095]

[0096] This representation can effectively capture the geometry, texture, and appearance information of the scene, achieve efficient rendering and processing, and retain rich details. When optimizing the parameters of a 3D Gaussian, it is often rendered to a 2D image plane, and tile-based rasterization technology is used to improve rendering efficiency. The formula for calculating the rendering color is:

[0097]

[0098] where c i is the color of the i-th Gaussian, N represents the Gaussian set in the tile, C(v) is the rendered color at pixel v, o i is the opacity of the ith Gaussian, Represents the function of the i-th Gaussian projection to two dimensions.

[0099] (II) Principle of semantic segmentation:

[0100] Semantic segmentation uses deep learning models, such as convolutional neural networks (CNNs), to train a large number of labeled images to learn the feature representations of different objects, thereby achieving accurate classification of image pixels. The core of this is to build a suitable network structure so that the model can automatically extract semantic information from the image and map pixels to predefined semantic categories.

[0101] (III) Semantic navigation principle based on three-dimensional Gaussian splashing:

[0102] By integrating 3D Gaussian splashing and semantic segmentation, semantic labels are assigned to Gaussian distributions in 3D scenes to achieve semantic understanding of scenes. In the navigation process, based on the semantic attributes of the target, combined with existing path planning algorithms (such as the A* algorithm), the optimal path is planned in 3D space. At the same time, with the help of real-time perception and scene updates, obstacles are avoided to achieve precise semantic navigation. The A* algorithm evaluates the function

[0103] f(n)=g(n)+h(n) (3)

[0104] to select a node, where g(n) represents the actual cost from the starting node to node n, and h(n) is the estimated cost from node n to the target node.

[0105] (3) Implementation process:

[0106] Please refer to Figure 1 , which shows a flowchart of a semantic navigation method based on three-dimensional Gaussian splashing provided by an embodiment of the present application, which may include the following steps:

[0107] S1 data collection and preprocessing uses a mobile device with a high-precision RGB-D camera to collect RGB images and depth images at a stable frame rate along a predetermined path, ensuring coverage of the entire indoor space, and preprocessing the collected RGB images and depth images.

[0108] S2 3D Gaussian splash model training, confirm the 3D Gaussian splash model architecture, initialize the model parameters, input the preprocessed image data into the model, optimize the model parameters using the stochastic gradient descent method, calculate the mean square error between the predicted scene and the actual scene, and backpropagate to update the parameters accordingly.

[0109] The model parameters include at least the mean, covariance matrix, and color of the initialized Gaussian distribution; the three-dimensional Gaussian splash model architecture is LangSplat.

[0110] S3 semantic segmentation model training and application: confirm the semantic segmentation model architecture, collect a large number of annotated images of indoor scenes, train the model with the cross entropy loss function, apply the trained semantic segmentation model to the three-dimensional scene reconstructed by three-dimensional Gaussian splashing, and assign accurate semantic labels to each pixel or three-dimensional Gaussian distribution.

[0111] Among them, semantic segmentation models include U-Net or DeepLab.

[0112] S4 semantic navigation is implemented in which the user inputs the navigation target through voice commands or a graphical interface. The navigation target is parsed according to the trained semantic segmentation model, the target semantic category and the area where it is located are determined, and the three-dimensional coordinates of the target object are located according to the target semantic label and position information. Path planning, navigation execution and implementation updates are performed based on the located three-dimensional coordinates.

[0113] In this embodiment, it specifically includes:

[0114] (I) Indoor environment information collection

[0115] 1. Data acquisition equipment selection

[0116] Choose a mobile device or robot with a high-precision RGB-D camera. The resolution should ensure that clear images can be obtained to accurately reconstruct the three-dimensional scene. For example, choose a camera with a resolution of no less than 640×480.

[0117] 2. Collection method

[0118] The mobile device or robot moves indoors along a predetermined path, capturing RGB images and depth images at a stable frame rate (e.g., 30 frames per second) to ensure coverage of the entire indoor space and avoid blind spots.

[0119] In order to enrich the data diversity, multiple data collection is carried out at different heights (such as 0.5 meters, 1 meter, and 1.5 meters), multiple angles (every 30 degrees in the horizontal direction), and different lighting conditions (such as natural light during the day and lights at night) to cope with complex indoor environments.

[0120] (II) 3D Gaussian splash model training

[0121] 1. Data preprocessing

[0122] The collected RGB image and depth image are accurately registered to ensure that the pixel and depth information match correctly.

[0123] Perform normalization operations to map pixel values ​​to the range of [0,1] to improve model training efficiency and stability.

[0124] Use methods such as median filtering to remove depth image noise and improve the accuracy of depth data.

[0125] 2. Model architecture selection

[0126] It uses advanced 3D Gaussian splash model architectures such as LangSplat, which excels in processing 3D scene data and reconstruction.

[0127] 3. Training process

[0128] Initialize model parameters, including the mean μ of the Gaussian distribution, the covariance matrix Σ, color, etc.

[0129] The preprocessed image data is input into the model, the model parameters are optimized by stochastic gradient descent method, and the mean square error between the predicted scene and the actual scene is calculated. And update the parameters in reverse propagation, where N is the number of samples, y iis the true value, is the predicted value.

[0130] Use data enhancement techniques, such as random cropping, rotation, and flipping of images, to increase the diversity of training data. At the same time, use the validation set to monitor model performance to prevent overfitting, and stop training when the validation set performance no longer improves.

[0131] 3. Semantic Segmentation

[0132] 1. Semantic segmentation model selection and training

[0133] We selected deep learning models suitable for indoor scenes, such as U-Net and DeepLab, which have excellent performance in semantic segmentation tasks.

[0134] Collect a large number of annotated images of indoor scenes, covering various types of furniture, electrical appliances, doors and windows, and other object categories.

[0135] Using the cross entropy loss function Training model, where C is the number of categories, y i is the true category label, The model predicts probability, updates the model parameters through back propagation, and adjusts the training parameters based on the validation set to optimize the model effect.

[0136] 2. Semantic Label Assignment

[0137] The trained semantic segmentation model is applied to the 3D scene reconstructed by 3D Gaussian splattering, and accurate semantic labels are given to each pixel or 3D Gaussian distribution. For example, the part belonging to the table is marked as the "table" category, thus realizing the semanticization of the scene and laying the foundation for semantic navigation.

[0138] The semantic navigation implementation process specifically includes:

[0139] 1. Target identification and positioning

[0140] The user inputs the navigation target through voice commands or graphical interfaces, such as "navigate to the sofa in the living room". The system uses semantic understanding technology to parse the command and determine the target semantic category ("sofa") and the area ("living room").

[0141] In a three-dimensional semantic scene, the three-dimensional coordinates of the target object can be quickly located based on the target semantic label and position information. For example, in the scene after semantic segmentation, objects of the "sofa" category can be searched, and their specific location in the living room can be determined based on the room layout.

[0142] 2. Path Planning

[0143] Based on the semantic information of the 3D scene and the target location, the optimal path is calculated using path planning algorithms such as A*. When planning, obstacles (identified by semantic tags) and traversable areas (such as corridors) are fully considered to ensure a safe and efficient path.

[0144] Convert the planned path into an executable action sequence for the mobile device or robot, such as forward, backward, turning, etc.

[0145] 3. Navigation execution and real-time update

[0146] The mobile device or robot performs navigation according to the planned path and collects real-time image data of the surrounding environment while moving.

[0147] The new data is input into the 3D Gaussian splash and semantic segmentation model to update the 3D scene representation and semantic information in real time. The path planning is adjusted in time according to the updated information to cope with dynamic changes in the environment (such as furniture movement) to ensure accurate and reliable navigation.

[0148] Technical measures specifically include:

[0149] (I) Data collection optimization

[0150] 1. Multi-sensor fusion

[0151] Fusion of sensor data such as LiDAR and Inertial Measurement Unit (IMU). LiDAR provides accurate distance information, especially for long-distance and low-light scenarios; IMU obtains device posture information. Sensor fusion technology integrates the advantages of different sensor data to improve data quality and reliability.

[0152] 2. Adaptive collection strategy

[0153] According to the characteristics of the indoor environment and navigation requirements, the collection frequency and coverage are increased in complex areas (such as furniture-dense areas) or key areas (such as entrances and exits), and the collection frequency is reduced in simple and open areas to improve the collection efficiency.

[0154] 2. Model training improvements

[0155] 1. Transfer Learning

[0156] The 3D Gaussian splatter and semantic segmentation models pre-trained on large-scale general datasets are fine-tuned on specific indoor scene datasets to accelerate model training, reduce the need for labeled data, and improve the performance of the model in specific scenarios.

[0157] 2. Model compression and acceleration

[0158] Model compression technologies such as pruning and quantization are used to reduce the number of model parameters and computational complexity, improve operational efficiency, and meet the real-time semantic navigation needs of resource-constrained devices (such as mobile robots).

[0159] 3. Semantic Navigation Enhancement

[0160] 1. Multimodal interaction

[0161] It supports multi-modal interaction methods such as voice commands, graphical user interface (GUI) and gesture recognition. Users can select targets by touching the screen or indicate directions with gestures, providing a more flexible and convenient user experience to adapt to various usage scenarios.

[0162] 2. Intelligent obstacle avoidance strategy

[0163] Combining semantic information with real-time perception data, an intelligent obstacle avoidance strategy is adopted. The semantic attributes of obstacles (movable or immovable) are considered for decision-making, and movable obstacles (such as chairs) are tried to be bypassed, and immovable obstacles (such as walls) are strictly avoided. At the same time, according to the prediction of obstacle movement trends, the navigation path is adjusted in advance to ensure safe navigation.

[0164] The technical improvements include:

[0165] (I) Improving the adaptability of the 3D Gaussian splash model to complex indoor scenes

[0166] 1. Improve model architecture

[0167] In view of the complex structure of indoor objects and changes in lighting, the 3D Gaussian splash model architecture is optimized. A local feature attention module is added to better capture the detailed texture of furniture; an illumination processing unit is introduced to adapt to scene reconstruction under different lighting conditions and improve the model's ability to represent complex indoor scenes.

[0168] 2. Optimize training algorithms

[0169] Adaptive learning rate adjustment strategies, such as Adagrad, Adadelta and other algorithms, are used to automatically adjust the learning rate according to data characteristics, improve model convergence speed and stability, and reduce gradient problems in complex scenario training.

[0170] 2. Improving the accuracy and efficiency of semantic segmentation

[0171] 1. Improve the network structure

[0172] Design a semantic segmentation network structure suitable for indoor scenes, combining the layout and semantic relationship characteristics of indoor objects. Use a hierarchical structure to first roughly segment the scene into large areas (such as rooms and corridors), and then finely segment each area to improve accuracy and efficiency. At the same time, introduce a context information fusion module to enhance the model's understanding of the semantic association of objects and reduce mis-segmentation.

[0173] 2. Optimize labeling strategy

[0174] Use semi-supervised learning methods to combine a small amount of labeled data with a large amount of unlabeled data for training. Automatically label some data through model prediction results and consistency constraints to reduce the workload of manual labeling. Establish labeling quality evaluation indicators to evaluate and correct labeled data in real time to ensure accuracy.

[0175] 3. Enhance the real-time and reliability of semantic navigation

[0176] 1. Real-time scene update mechanism

[0177] Establish an efficient real-time scene update mechanism, adopt an incremental update algorithm, process only the changed parts, and reduce computing resource consumption. Use parallel computing technology (such as multi-threading, GPU acceleration) to parallelize data processing tasks, increase the update speed, and meet real-time navigation needs.

[0178] 2. Robust path planning algorithm

[0179] Improve the path planning algorithm, introduce a probabilistic model, consider the probability of obstacle movement and environmental uncertainty, and plan a fault-tolerant path. Establish a path replanning strategy to quickly replan the path when encountering unexpected situations (such as road blockage) to ensure the smooth completion of the navigation task.

[0180] The working principle and working process specifically include:

[0181] 1. Data collection stage

[0182] The camera collects RGB images and depth image data of the indoor environment in real time and transmits them to the processor through the data transmission line. The processor performs preliminary processing such as format conversion and caching on the data.

[0183] 2. Model processing stage

[0184] The processor calls the 3D Gaussian splash model training module and inputs the collected data into the model for training or scene reconstruction. The model calculates the 3D Gaussian distribution parameters based on the algorithm and parameters to construct a 3D scene representation.

[0185] The processor inputs the reconstructed 3D scene data into the semantic segmentation model, which performs semantic segmentation on the scene based on the training parameters and assigns semantic labels to each part.

[0186] 3. Navigation phase

[0187] The user inputs the navigation target through the interactive device (voice input, touch screen, etc.), and the interactive device transmits the command to the processor. The processor interprets the command and uses the semantic navigation module to locate the target and plan the path in the three-dimensional semantic scene.

[0188] The path planning results are converted into control signals, which are then used by the drive module to control the movement of the mobile device or robot. During the navigation process, the camera continuously collects data, and the processor updates the three-dimensional scene and semantic information in real time, adjusting the navigation path according to environmental changes to ensure accurate arrival at the target location.

[0189] In summary, it can be seen that the present invention uses 3D Gaussians containing semantic feature embedding to represent 3D scenes, encodes semantic features onto each 3D Gaussian, and constructs a three-dimensional semantic field in this way, so that the model can respond to text-based queries. Compared with the NeRF-based method, its rendering process is faster.

[0190] Please refer to Figure 2 , which shows a block diagram of a semantic navigation system based on three-dimensional Gaussian splashing provided by an embodiment of the present application. Figure 2 As shown, the system may include:

[0191] The data acquisition and preprocessing module is used to use a mobile device with a high-precision RGB-D camera to collect RGB images and depth images at a stable frame rate along a predetermined path, ensure coverage of the entire indoor space, and preprocess the collected RGB images and depth images;

[0192] The 3D Gaussian splash model training module is used to confirm the 3D Gaussian splash model architecture, initialize the model parameters, input the preprocessed image data into the model, optimize the model parameters using the stochastic gradient descent method, calculate the mean square error between the predicted scene and the actual scene, and back-propagate the updated parameters accordingly; the model parameters at least include the mean, covariance matrix, and color of the initialized Gaussian distribution; the 3D Gaussian splash model architecture is LangSplat;

[0193] The semantic segmentation model training and application module is used to confirm the semantic segmentation model architecture, collect a large number of annotated images of indoor scenes, train the model with a cross-entropy loss function, apply the trained semantic segmentation model to the 3D scene reconstructed by 3D Gaussian splashing, and assign accurate semantic labels to each pixel or 3D Gaussian distribution; the semantic segmentation model includes U-Net or DeepLab;

[0194] In the semantic navigation implementation module, the user inputs the navigation target through voice commands or a graphical interface. The navigation target is parsed according to the trained semantic segmentation model, the target semantic category and the area where it is located are determined, and the three-dimensional coordinates of the target object are located according to the target semantic label and position information; path planning, navigation execution and implementation updates are performed based on the located three-dimensional coordinates.

[0195] For the specific definition of the semantic navigation system based on three-dimensional Gaussian splashing, please refer to the definition of the semantic navigation method based on three-dimensional Gaussian splashing in the above text, which will not be repeated here. The various modules in the above-mentioned semantic navigation system based on three-dimensional Gaussian splashing can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0196] In one embodiment, an electronic device is provided. The electronic device may be a computer, and its internal structure diagram may be as follows: Figure 3 As shown. The electronic device includes a processor, a memory and a network interface connected via a system bus. The processor of the device is used to provide computing and control capabilities. The memory of the device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used for semantic navigation data based on three-dimensional Gaussian splashing. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a semantic navigation method based on three-dimensional Gaussian splashing is implemented.

[0197] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0198] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which involves all or part of the processes in the above-mentioned embodiment method.

[0199] In one embodiment, a computer program product is also provided, including a computer program / instruction, which involves all or part of the process in the above embodiment method.

[0200] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in M ​​forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (SyMchliMk) DRAM (SLDRAM), memory bus (RaMbus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0201] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0202] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.

Claims

1. A semantic navigation method based on three-dimensional Gaussian splashing, characterized in that: The method comprises: Use a mobile device with a high-precision RGB-D camera to collect RGB images and depth images at a stable frame rate along a predetermined path, ensure that the entire indoor space is covered, and pre-process the collected RGB images and depth images; Confirm the three-dimensional Gaussian splash model architecture, initialize the model parameters, input the preprocessed image data into the model, optimize the model parameters using the stochastic gradient descent method, calculate the mean square error between the predicted scene and the actual scene, and back-propagate and update the parameters accordingly; where the model parameters at least include the mean, covariance matrix, and color of the initialized Gaussian distribution; the three-dimensional Gaussian splash model architecture is LangSplat; Confirm the semantic segmentation model architecture, collect a large number of annotated images of indoor scenes, train the model with the cross entropy loss function, apply the trained semantic segmentation model to the 3D scene reconstructed by 3D Gaussian splashing, and assign accurate semantic labels to each pixel or 3D Gaussian distribution; the semantic segmentation model includes U-Net or DeepLab; The user inputs the navigation target through voice commands or a graphical interface. The navigation target is parsed according to the trained semantic segmentation model, the target semantic category and the area where it is located are determined, and the three-dimensional coordinates of the target object are located based on the target semantic label and position information. Path planning, navigation execution and implementation updates are performed based on the located three-dimensional coordinates.

2. The semantic navigation method according to claim 1, characterized in that: Path planning and navigation execution and implementation updates based on the three-dimensional coordinates of positioning, including: Based on the semantic information of the three-dimensional scene and the target position, the A* path planning algorithm is used to calculate the optimal path and convert it into an executable action sequence for the mobile device or robot; the mobile device or robot performs navigation according to the planned path, collects image data of the surrounding environment in real time, and inputs the new data into the three-dimensional Gaussian splash and semantic segmentation model, updates the three-dimensional scene representation and semantic information in real time, and adjusts the path planning in time according to the updated information.

3. The semantic navigation method according to claim 1, characterized in that: Use a mobile device with a high-precision RGB-D camera to collect RGB images and depth images at a stable frame rate along a predetermined path, and ensure that the entire indoor space is covered. Pre-process the collected RGB images and depth images, including: Multiple acquisitions were made at different heights, angles and lighting conditions; the heights included at least 0.5m, 1m and 1.5m, the angles included 30-degree intervals in the horizontal direction, and the lighting conditions included natural light during the day and lights at night; Perform accurate registration and normalization of the image and depth information, and use median filtering to remove depth image noise.

4. The semantic navigation method according to claim 1, characterized in that: Feeding the preprocessed image data into the model also includes: Use data augmentation techniques, including random cropping, rotation, and flipping of images, to increase the diversity of training data, and use validation sets to monitor model performance and prevent overfitting.

5. The semantic navigation method according to claim 1, characterized in that: Use stochastic gradient descent to optimize model parameters, including: Calculate the mean square error between the predicted scenario and the actual scenario And update the parameters in reverse propagation, where N is the number of samples and y i is the true value, is the predicted value.

6. The semantic navigation method according to claim 1, characterized in that: The model is trained with a cross entropy loss function, including: Using the cross entropy loss function Training model, where C is the number of categories, y i is the true value, To predict the value, the model parameters are updated through back propagation, and the training parameters are adjusted according to the validation set to optimize the model effect.

7. A semantic navigation system based on three-dimensional Gaussian splashing, characterized in that: The system comprises: The data acquisition and preprocessing module is used to use a mobile device with a high-precision RGB-D camera to collect RGB images and depth images at a stable frame rate along a predetermined path, ensure coverage of the entire indoor space, and preprocess the collected RGB images and depth images; The 3D Gaussian splash model training module is used to confirm the 3D Gaussian splash model architecture, initialize the model parameters, input the preprocessed image data into the model, optimize the model parameters using the stochastic gradient descent method, calculate the mean square error between the predicted scene and the actual scene, and back-propagate the updated parameters accordingly; the model parameters at least include the mean, covariance matrix, and color of the initialized Gaussian distribution; the 3D Gaussian splash model architecture is LangSplat; The semantic segmentation model training and application module is used to confirm the semantic segmentation model architecture, collect a large number of annotated images of indoor scenes, train the model with a cross-entropy loss function, apply the trained semantic segmentation model to the 3D scene reconstructed by 3D Gaussian splashing, and assign accurate semantic labels to each pixel or 3D Gaussian distribution; the semantic segmentation model includes U-Net or DeepLab; In the semantic navigation implementation module, the user inputs the navigation target through voice commands or a graphical interface. The navigation target is parsed according to the trained semantic segmentation model, the target semantic category and the area where it is located are determined, and the three-dimensional coordinates of the target object are located according to the target semantic label and position information; path planning, navigation execution and implementation updates are performed based on the located three-dimensional coordinates.

8. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images