Anti-shake semantic SLAM method and device for biped humanoid robot and medium

Abstract

The invention discloses an anti-shake semantic SLAM method and device for a biped humanoid robot and a medium, and belongs to the technical field of semantic map construction. The anti-jitter semantic SLAM method comprises the following steps: S1, acquiring continuous video frames captured by a camera, and performing semantic segmentation based on an open vocabulary on each video frame to obtain a segmentation instance result of each frame; s2, constructing a Gaussian ball in a three-dimensional space based on continuous video frames captured by a camera, and carrying out initialization processing on the Gaussian ball; and S3, performing optimization processing on the preliminarily estimated six-degree-of-freedom pose of the biped humanoid robot and the initially constructed three-dimensional scene semantic map, and outputting the optimized six-degree-of-freedom pose and three-dimensional scene semantic map. Through semantic understanding and the semantic segmentation method based on the open vocabulary, the positioning precision and the map construction quality are improved, and the flexibility and generalization ability of the method in real world application are enhanced.

Description

Technical Field

[0001] The present application relates to the technical field of semantic map construction, and specifically to an anti-shake semantic SLAM method, device, and medium for a bipedal humanoid robot. Background Art

[0002] Existing technologies have made some progress in the field of autonomous operation of bipedal robots, especially in the application of SLAM (Simultaneous Localization and Mapping) algorithms. SLAM algorithms enable robots to build environmental maps and update their positions in real time, which is essential for achieving autonomous navigation without intervention. Vision-based SLAM algorithms are popular due to their cost-effectiveness, miniaturization, and simple sensor configuration.

[0003] Existing inventions focus on wheeled robots in specific scenarios, and use different variants of semantic SLAM technology to improve the robot's positioning, map construction, target recognition and classification capabilities in a limited environment. For example, the Chinese patent application document with publication number CN115962769A discloses a lightweight semantic map construction method and system for a lawn mowing robot, which combines an RGB camera and a depth camera to achieve semantic extraction of targets and three-dimensional position estimation of the lawn mowing robot. The Chinese patent application document with publication number CN115249300A discloses a semantic SLAM method for a wheeled robot for a hospital scene, which uses an RGB-D camera to collect RGB images and depth information of the environment, and builds a local visual semantic map by fusing IMU data to achieve high-precision positioning, especially for inherent semantic features in the hospital environment, such as nurse stations, guide signs, etc., to perform map construction and positioning. The Chinese patent application document with publication number CN111360780A discloses a garbage picking robot based on visual semantic SLAM, which integrates visual information and IMU data to build a semantic map, identify and classify garbage, and guide garbage recycling operations. Wheeled robots may encounter relatively fewer challenges in limited scenarios, while humanoid robots need to deal with more degrees of freedom of movement and environmental uncertainties, which requires more advanced perception, planning and decision-making capabilities.

[0004] For bipedal humanoid robots, although early research has made some progress in implementing SLAM technology on bipedal humanoid robots, using advanced algorithms such as fine map construction using depth camera sensors and fusion of deep learning, there are still several key challenges:

[0005] 1. The impact of movement and shaking: When a humanoid robot walks, its body structure and movement pattern will cause significant jitter and instability in the data captured by the depth camera. This dynamic change will interfere with the accuracy of the SLAM algorithm, affect the stability of positioning and map construction, and thus reduce the robustness and efficiency of the entire system.

[0006] 2. Challenges in feature-sparse environments: Feature-based SLAM algorithms rely on extracting unique visual features from the environment, such as corners, edges, or textures. However, in some environments, such as long corridors, blank walls, or natural landscapes, there may not be enough features for mileage calculations to use, which makes it difficult for SLAM algorithms to locate.

[0007] 3. Difficulty in recognizing new categories of semantic entities: In addition to basic positioning and map construction, humanoid robot SLAM systems also need to have semantic understanding capabilities, that is, to recognize and classify objects, areas or events in the environment. When the robot encounters semantic entities that have not been seen before or are not recorded in the database, recognition and classification become particularly difficult. This limits the robot's autonomy and intelligence level in unknown or changing environments.

[0008] Therefore, there is an urgent need to develop an anti-shake semantic SLAM method for bipedal humanoid robots to ensure that the bipedal humanoid robots can effectively perform SLAM and downstream tasks under various conditions. Summary of the invention

[0009] In view of this, the present application aims to solve at least one of the above-mentioned technical problems in the related art to a certain extent. To this end, the present application provides a method, device, and medium for anti-shake semantic SLAM of a bipedal humanoid robot, which improves the positioning accuracy and the quality of map construction through semantic understanding and a semantic segmentation method based on an open vocabulary, and enhances its flexibility and generalization ability in real-world applications.

[0010] To achieve the above objectives, in a first aspect, the present application provides an anti-shake semantic SLAM method for a bipedal humanoid robot, comprising the following steps:

[0011] S1, obtaining continuous video frames captured by a camera, inputting them into a semantic segmentation model, performing semantic segmentation based on an open vocabulary on each of the video frames, and obtaining a segmentation instance result of each frame, wherein the video frame contains color information and depth information of the scene;

[0012] S2, constructing a Gaussian sphere in a three-dimensional space based on continuous video frames captured by the camera, and performing initialization processing on the Gaussian sphere; wherein the step of performing initialization processing on the Gaussian sphere includes performing preliminary estimation of the six-degree-of-freedom posture of the bipedal humanoid robot and initial construction of a semantic map of the three-dimensional scene;

[0013] S3, optimizing the six-degree-of-freedom posture of the biped humanoid robot preliminarily estimated in step S2 and the three-dimensional scene semantic map initially constructed, and outputting the optimized six-degree-of-freedom posture and three-dimensional scene semantic map;

[0014] Among them, the optimization processing step includes estimating the impact of camera shake by constructing a motion blur reconstruction error and minimizing the motion blur reconstruction error. The blurred image corresponding to the motion blur reconstruction error is represented by the integral of the irradiance of the time-varying six-degree-of-freedom posture within the exposure time.

[0015] Preferably, the step of performing open vocabulary-based semantic segmentation on each of the video frames includes: using a pre-trained visual language model as a teacher model to provide guidance for the semantic segmentation model, and the semantic segmentation model refines its learned visual concepts into a set of segmentation tokens, wherein each set of segmentation tokens summarizes a local area of ​​the video frame, and the visual concepts include the shape, texture, and color of the object.

[0016] Preferably, the step of performing semantic segmentation based on an open vocabulary on each of the video frames further comprises: based on a three-dimensional vision algorithm Bundle Adjustment, using the entire observation sequence to perform mutual verification between multiple frames.

[0017] Preferably, the step of using the entire observation sequence to perform mutual verification between multiple frames includes: adding visual Figure 1 The consistency score index is used to measure whether two masks belong to the same object, where the visual Figure 1 The consistency score metric is defined as the support rate for merging two masks in a sequence of observations.

[0018] Preferably, after step S3, the method further includes: S4, acquiring new continuous video frames as the camera moves, updating the three-dimensional scene semantic map based on the new continuous video frames, and outputting a real-time running trajectory map of the bipedal humanoid robot.

[0019] Preferably, the optimization processing step in step S3 also includes: constructing an image error, a depth reconstruction error and a semantic reconstruction error relative to the current video frame, and optimizing the running trajectory by minimizing the image error, the depth reconstruction error and the semantic reconstruction error.

[0020] Preferably, in step S2 of constructing a Gaussian sphere in a three-dimensional space based on continuous video frames captured by the camera and performing initialization processing on the Gaussian sphere, the Gaussian sphere in the three-dimensional space is represented as:

[0021] (1)

[0022] in,f 3D ( x ) represents the Gaussian sphere, x Represents the coordinates in space, σ Represents transparency, μ represents the center position of the Gaussian sphere, r represents the radius of the Gaussian sphere;

[0023] To optimize the parameters of the Gaussian distribution to represent the scene, the Gaussian is rendered as a 2D image in a differentiable way:

[0024] (2)

[0025] represents the two-dimensional projection of the center of the Gaussian sphere on the rendering plane, represents the two-dimensional projection of the Gaussian sphere radius on the rendering plane, K Represents the camera internal parameters, represents the focal length, d Represents depth information. z The weight of the axis; E t Represents the six-degree-of-freedom pose of the camera.

[0026] Preferably, the step of performing optimization processing in step S3 further includes: arranging all Gaussian spheres in depth order, and performing front and back volume rendering using a maximum volume rendering formula.

[0027] Preferably, the image error formula for calculating pixel p=(u, v) on each frame is:

[0028] (3)

[0029] in, n Represents the number of Gaussians on the spatial ray of the current pixel point, i is the index in the summation symbol, j is an index into the product notation, where i From 1 to n changes, indicating the spherical harmonic expansion of i coefficients, j From 1 to i −1 changes, used to multiply the previous coefficients, C(p) Representing pixels p The image error on c i Represents RGB color using spherical harmonic coefficients, , f i ( p ) means in pixels p The iThe value of the Gaussian function, o represents the amplitude or weight of the Gaussian function, Indicates j Gaussian function at pixel p The complement of Represents pixels p and mean μ The square of the Euclidean distance between represents the product symbol;

[0030] The depth reconstruction error is expressed as:

[0031] (4)

[0032] in, D(p) Representing pixels p The depth reconstruction error on d i The representative depth is expressed by spherical harmonic coefficients;

[0033] The semantic reconstruction error is expressed as:

[0034] (5)

[0035] S(p) Representing pixels p The semantic reconstruction error on s i The semantics of the representation is expressed by spherical harmonic coefficients;

[0036] Motion blurred images are considered Overlap of real images of several poses:

[0037] (6)

[0038] in, B represents a motion blurred image, Represents a time variable, used to define the exposure time interval, represent Pixels of the moment, and Respectively represent the start and end time of the exposure time interval, Indicates at time The image acquired at each moment is in pixels The intensity or color value at Indicated in i Pixels in the frame The intensity or color value at N Represents the total number of frames.

[0039] In a second aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to implement the steps of any of the above-mentioned anti-shake semantic SLAM methods.

[0040] In a third aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the steps of any of the above-mentioned anti-shake semantic SLAM methods.

[0041] In a fourth aspect, the present application provides a bipedal humanoid robot, which executes any of the above-mentioned anti-shake semantic SLAM methods to achieve simultaneous positioning and map construction.

[0042] Based on the above technical solution, it can be seen that the data encryption method and system of the present application have at least one of the following beneficial effects compared with the prior art:

[0043] 1. This application improves the positioning accuracy and the quality of map construction through semantic understanding and semantic segmentation methods based on open vocabulary, enhances its flexibility and generalization ability in real-world applications, and ensures that the bipedal humanoid robot can effectively perform SLAM and downstream tasks under various conditions.

[0044] 2. This application solves the impact of movement and shaking of bipedal humanoid robots. For the high-jitter motion scenes of bipedal humanoid robots, this application proposes a method for estimating the moving six-degree-of-freedom camera pose from motion blurred images and restoring clear three-dimensional scenes, thereby improving the accuracy and stability of SLAM algorithms in bipedal humanoid robot applications.

[0045] 3. This application enhances the SLAM system by combining an object semantic segmentation algorithm to provide a richer environmental description than traditional feature points to supplement the lack of traditional feature points; by integrating object semantic information into the SLAM algorithm, the system can generate more detailed maps in environments with sparse features and maintain more accurate robot positioning.

[0046] Other features and advantages of the present application will be described in the subsequent description, and in part will become apparent from the description, or it may be understood through the implementation of the present application that the objects and other advantages of the present application can be realized and obtained through the written description and the structures particularly pointed out in the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flow chart of the anti-shake semantic SLAM method in Example 1 of the present application;

[0048] Figure 2This is a flow chart of the anti-shake semantic SLAM method in Example 2 of the present application;

[0049] Figure 3 This is a schematic diagram of the execution of an anti-shake semantic SLAM method for a bipedal humanoid robot in this application. DETAILED DESCRIPTION

[0050] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0051] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application are also intended to include plural forms, unless the context clearly indicates other meanings.

[0052] In order to improve the data security of the decryption process, the applicant has conducted in-depth research on positioning technology and map construction technology. By integrating multiple technologies and methods, this patent proposes an anti-shake semantic SLAM method for bipedal humanoid robots to improve the positioning, map construction and semantic understanding capabilities of high-shake bipedal humanoid robots in complex environments. The following solutions are proposed:

[0053] 1. Solve the impact of bipedal humanoid robot movement and shaking: Bipedal humanoid robots will produce unstable swaying and uncertain movements when walking, resulting in high-jitter motion that causes depth camera motion blur, and the blurred image obtained introduces noise into the SLAM algorithm. This application is aimed at high-jitter motion scenes of bipedal humanoid robots, and proposes a method for estimating the motion of a six-degree-of-freedom camera from motion blurred images and restoring a clear three-dimensional scene, thereby improving the accuracy and stability of the SLAM algorithm in the application of bipedal humanoid robots.

[0054] 2. Improve the positioning accuracy of bipedal humanoid robots in feature-sparse environments: In environments with sparse features, existing SLAM algorithms will encounter performance bottlenecks because they usually rely on rich features in the environment (such as corners, textures, etc.) to estimate the robot's pose and build an environmental map. When these features are insufficient, such as in scenes with long corridors, blank walls, or large areas of homogeneous surfaces, the positioning accuracy and stability of the SLAM algorithm will decrease. To address this challenge, this application enhances the SLAM system by combining an object semantic segmentation algorithm to provide a richer description of the environment than traditional feature points to supplement the lack of traditional feature points. By incorporating object semantic information into the SLAM algorithm, the system is able to generate more detailed maps in environments with sparse features and maintain more accurate robot positioning.

[0055] 3. Solve the problem of identifying and understanding new categories of semantic entities: Existing semantic segmentation technology has poor recognition ability for objects that are not in its initial training data set, that is, new categories of objects. This application introduces semantic segmentation technology based on an open vocabulary to enable the robot to recognize and classify unseen objects, enhance its semantic understanding in unknown or changing environments, thereby improving its autonomy and intelligence level, and enabling the robot to better understand and adapt to complex real-world scenarios.

[0056] Embodiment 1

[0057] like Figure 1 As shown, the present application provides an anti-shake semantic SLAM method for a biped humanoid robot, comprising the following steps:

[0058] S1, obtaining continuous video frames captured by a camera, inputting them into a semantic segmentation model, performing semantic segmentation based on an open vocabulary on each of the video frames, and obtaining a segmentation instance result of each frame, wherein the video frame contains color information and depth information of the scene;

[0059] S2, constructing a Gaussian sphere in a three-dimensional space based on continuous video frames captured by the camera, and performing initialization processing on the Gaussian sphere; wherein the step of performing initialization processing on the Gaussian sphere includes performing preliminary estimation of the six-degree-of-freedom posture of the bipedal humanoid robot and initial construction of a semantic map of the three-dimensional scene;

[0060] S3, optimizing the six-degree-of-freedom posture of the biped humanoid robot preliminarily estimated in step S2 and the three-dimensional scene semantic map initially constructed, and outputting the optimized six-degree-of-freedom posture and three-dimensional scene semantic map;

[0061] Among them, the optimization processing step includes estimating the impact of camera shake by constructing a motion blur reconstruction error and minimizing the motion blur reconstruction error. The blurred image corresponding to the motion blur reconstruction error is represented by the integral of the irradiance of the time-varying six-degree-of-freedom posture within the exposure time.

[0062] The above-mentioned motion blur reconstruction error refers to the change in the camera posture due to the dynamic movement of the bipedal humanoid robot during the camera exposure time, which makes the captured image blurry. This blur is due to the continuous change of the six-degree-of-freedom posture (position and direction) of the camera during the exposure period, resulting in the accumulation of light information at different times and positions on the same pixel. The motion blur reconstruction error is an error indicator used to quantify the degree of image blur caused by motion. By reducing the error caused by motion blur, the performance of the SLAM algorithm can be optimized. In addition, the motion blur reconstruction error, as a feedback mechanism, can help the robot better adapt to these disturbances and enhance the robustness of the entire system.

[0063] Preferably, the step of performing open vocabulary-based semantic segmentation on the video frame includes using a pre-trained visual language model as a teacher model to provide guidance for the semantic segmentation model, and the semantic segmentation model refines its learned visual concepts into a set of segmentation tokens, wherein each set of segmentation tokens summarizes a local area of ​​the video frame, and the visual concepts include the shape, texture, and color of the object. Specifically, the teacher model provides guidance to the semantic segmentation model through the knowledge learned in the pre-training process, and this guidance can be direct parameter sharing or by transferring the knowledge of the teacher model to the semantic segmentation model through knowledge distillation.

[0064] Specifically, a segmentation token refers to a set of parameters or features in the model that are used to identify and segment specific regions in an image. These tokens can be viewed as "learning units" in the model, with each unit responsible for identifying a local feature or object in an image. Under the guidance of the teacher model, the segmentation token learns how to accurately capture local regions in an image and perform effective segmentation. The goal of the segmentation token is to accurately identify different objects and scene elements in an image and separate them from the background.

[0065] Preferably, the step of performing semantic segmentation of the video frame based on an open vocabulary also includes mutual verification between multiple frames using the entire observation sequence based on a 3D vision algorithm Bundle Adjustment. Bundle Adjustment (BA) is a 3D vision algorithm widely used in computer vision and photogrammetry, and is used to optimize the camera pose and position of 3D points in 3D reconstruction and motion recovery structure problems.

[0066] Preferably, the step of using the entire observation sequence to perform mutual verification between multiple frames includes: adding visual Figure 1 The consistency score index is used to measure whether two masks belong to the same object, where the visual Figure 1 The consistency score metric is defined as the support rate for merging two masks in a sequence of observations.

[0067] Specifically, the input RGBD video frame is first semantically segmented based on an open vocabulary to obtain the segmentation instance results of each frame. The introduction of open vocabulary is mainly to help identify new categories that have not been seen in the environment. The main method is to use a pre-trained visual-language model (such as CLIP) as a "teacher model" to refine its learned visual concepts into a set of segmentation tokens, each of which summarizes a local area of ​​the target image. This process does not require any manual labels, so the knowledge of models such as CLIP can be directly transferred to the semantic segmentation task. The output of the existing semantic segmentation model is inconsistent between different frames. In the instance segmentation of the same object in different frames, although the mask segmentation is correct, the object ID does not correspond. Therefore, the core lies in establishing the correspondence between the two-dimensional mask ID of different frames. This application draws on the classic three-dimensional vision algorithm Bundle Adjustment and uses the entire observation sequence to perform mutual verification between multiple frames to improve global consistency.

[0068] Specifically, we add visual Figure 1 The consistency score is used to measure whether two masks belong to the same 3D object. Figure 1 The consistency score is defined as the support rate of supporting the merging of the two masks in a certain observation sequence (within the sliding window). First, find all frames in the sliding window that contain both masks. For any frame, if the two masks are contained in the same mask in the frame, then the frame supports their merging. Figure 1 The consistency score is the proportion of support for merging. Figure 1 When the consistency score is higher than the set threshold, an edge is added between them, indicating that they may belong to the same object. Then, a mask graph is constructed with all the masks in the sliding window as nodes. Figure 1 The node with the highest consistency score is selected and the view is recalculated. Figure 1 The consistency score is iterated until there is no edge. Finally, the earliest video frame in the sliding window is popped out and the next video frame is added.

[0069] Therefore, this application introduces semantic segmentation technology based on an open vocabulary to enable the robot to recognize and classify unseen objects, enhance its semantic understanding in unknown or changing environments, thereby improving autonomy and intelligence, and enabling the robot to better understand and adapt to complex real-world scenarios. And by combining the object semantic segmentation algorithm to enhance the SLAM system, it provides a richer description of the environment than traditional feature points to supplement the lack of traditional feature points. By incorporating object semantic information into the SLAM algorithm, the system can generate more detailed maps in environments with sparse features and maintain more accurate robot positioning.

[0070] Embodiment 2

[0071] like Figure 2 As shown, the present application provides an anti-shake semantic SLAM method for a biped humanoid robot, comprising the following steps:

[0072] S1, obtaining continuous video frames captured by a camera, inputting them into a semantic segmentation model, performing semantic segmentation based on an open vocabulary on each of the video frames, and obtaining a segmentation instance result of each frame, wherein the video frame contains color information and depth information of the scene;

[0073] S2, constructing a Gaussian sphere in a three-dimensional space based on continuous video frames captured by the camera, and performing initialization processing on the Gaussian sphere; wherein the step of performing initialization processing on the Gaussian sphere includes performing preliminary estimation of the six-degree-of-freedom posture of the bipedal humanoid robot and initial construction of a semantic map of the three-dimensional scene;

[0074] S3, optimizing the six-degree-of-freedom posture of the biped humanoid robot preliminarily estimated in step S2 and the three-dimensional scene semantic map initially constructed, and outputting the optimized six-degree-of-freedom posture and three-dimensional scene semantic map;

[0075] S4, as the camera moves, new continuous video frames are obtained, the three-dimensional scene semantic map is updated based on the new continuous video frames, and a real-time running trajectory map of the bipedal humanoid robot is outputted;

[0076] Among them, the optimization processing step includes estimating the impact of camera shake by constructing a motion blur reconstruction error and minimizing the motion blur reconstruction error. The blurred image corresponding to the motion blur reconstruction error is represented by the integral of the irradiance of the time-varying six-degree-of-freedom posture within the exposure time.

[0077] The difference between this embodiment and the first embodiment is that the present application further adds step S4 after step S3, obtains new continuous video frames as the camera moves, updates the three-dimensional scene semantic map based on the new continuous video frames, and outputs the real-time running trajectory map of the bipedal humanoid robot. In the SLAM system, the running trajectory of the robot can be generated by estimating the change of the camera's six-degree-of-freedom posture over time.

[0078] Preferably, the optimization step in step S3 also includes constructing an image error, a depth reconstruction error, and a semantic reconstruction error relative to the current video frame, and optimizing the running trajectory by minimizing the image error, the depth reconstruction error, and the semantic reconstruction error. Image error: refers to the difference between the current video frame and the image reconstructed according to the currently estimated camera pose and map information. This difference may be caused by inaccurate camera pose estimation or incomplete map information.

[0079] Depth reconstruction error: refers to the difference between the depth information in the current video frame and the depth information reconstructed based on the current estimated camera pose and map information. This difference reflects the reconstruction quality of the depth information.

[0080] Semantic reconstruction error: refers to the difference between the semantic information (such as object category, identity, etc.) in the current video frame and the semantic information reconstructed based on the current estimated camera pose and map information. This difference reflects the reconstruction quality of the semantic information.

[0081] The premise of optimizing the running trajectory is to first optimize the six-degree-of-freedom posture and the three-dimensional scene semantic map. The running trajectory refers to the path that the robot or camera moves in the environment. In the SLAM system, the robot's running trajectory can be generated by estimating the changes in the six-degree-of-freedom posture of the camera.

[0082] like Figure 3 As shown in the figure, it is a schematic diagram of the execution of the anti-shake semantic SLAM method. The data input part that can be obtained according to the continuous video frames captured by the camera includes: RGB color image, which is used to capture the color information of the scene; depth image, which is used to capture the distance information of each point in the scene to the camera; semantic segmentation result, semantic segmentation is performed through an open vocabulary, and a semantic label is assigned to each pixel in the image, such as furniture, ground, wall, etc.; In addition, it is also necessary to consider the image blur caused by the movement of the camera or object during the exposure time, and this blur needs to be compensated to improve the accuracy of positioning and map construction. The camera posture is tracked using RGB, depth and semantic information. This step is to estimate the motion trajectory of the camera in three-dimensional space by analyzing continuous video frames. At the same time, the posture information obtained by camera tracking is combined with RGB, depth and semantic information to perform Gaussian densification map reconstruction; the semantic map finally generated contains the three-dimensional structure of the scene and the semantic information of each pixel. This map can be used for robot navigation, path planning and environmental understanding.

[0083] This application models the 3D scene as a set of 3D Gaussians, which can be rendered into high-fidelity color depth and semantic images, so that differentiable rendering and gradient-based optimization can be directly used to optimize the camera's six-degree-of-freedom pose and dense semantic map for each frame. Specifically, the following steps are included:

[0084] (1) First, perform semantic segmentation based on open vocabulary on the input RGBD video frame to obtain the segmentation instance result of each frame. The specific segmentation process is the same as that in Example 1.

[0085] (2) Use the SFM (Structure from Motion) algorithm to initialize the point cloud and Gaussian parameters. The SFM technology extracts feature points from images of multiple perspectives and matches them. It uses a triangulation algorithm to calculate the corresponding 3D point coordinates, and then restores the initial point cloud map of the entire scene. Then, this application calculates the initial parameters of the Gaussian sphere based on the initialized point cloud. E t , μ, r, σ, c , which are the six-degree-of-freedom pose of the camera, the center position of the Gaussian sphere, the radius of the Gaussian sphere, the transparency, and the RGB color represented by the spherical harmonic coefficients.

[0086] In step S2, a Gaussian sphere in a three-dimensional space is constructed based on continuous video frames captured by the camera, and the Gaussian sphere is initialized. The Gaussian sphere in the three-dimensional space is expressed as:

[0087] (1)

[0088] f 3D represents the Gaussian sphere, x Represents the coordinates in space, σ Represents transparency, μ represents the center position of the Gaussian sphere, r represents the radius of the Gaussian sphere, Representation vector x and μ The square of the Euclidean distance between two points. The Euclidean distance is a parameter that measures the straight-line distance between two points in Euclidean space.

[0089] To optimize the parameters of the Gaussian distribution to represent the scene, the Gaussian is rendered as a 2D image in a differentiable way:

[0090] (2)

[0091] represents the two-dimensional projection of the center of the Gaussian sphere on the rendering plane, represents the two-dimensional projection of the Gaussian sphere radius on the rendering plane, K Represents the camera internal parameters, l represents the focal length, d Represents depth information. z The weight of the axis; E t Represents the six-degree-of-freedom pose of the camera (camera extrinsics).

[0092] (3) Track the six-degree-of-freedom motion trajectory of the camera to obtain the position and posture of the camera (i.e., the body of the bipedal humanoid robot) and complete the positioning. t+ 1 minimizes the image error, depth reconstruction error, motion blur irradiance error, and semantic reconstruction error of the RGB-D frame, but only evaluates the error on the pixels within the visible contour. Among them, minimizing the RGB error, depth reconstruction error, and semantic reconstruction error of the RGB-D frame is to optimize the camera trajectory, while the motion blur irradiance error is mainly to deal with the motion blurred image caused by the shaking when the bipedal humanoid robot walks. Physically, camera motion blur is caused by the integral of the irradiance during the camera movement. Therefore, the acquisition of the blurred image is represented by the integral of the irradiance of the time-varying six-degree-of-freedom camera pose during the exposure time. We simulate this motion blurred image by introducing the motion blur irradiance error, thereby estimating the six-degree-of-freedom pose of the camera under shaking.

[0093] Preferably, the step of performing the optimization process further includes: arranging all Gaussian balls in depth order, and performing front and back volume rendering using the maximum volume rendering formula. The reason for arranging in depth order is that in three-dimensional space, the objects or points represented by different Gaussian balls may be located at different depths (i.e., the distance from the observer). Arranging in depth order means sorting according to the distance from the center of these Gaussian balls to the camera. Closer objects should visually block farther objects, which helps to render the scene correctly. The front volume rendering refers to rendering from the camera to the front of the scene, considering the occlusion of the line of sight by the object; the rear volume rendering refers to rendering from the back of the scene to the camera direction, considering the occlusion of the line of sight by the object. Starting from the farthest Gaussian ball, step by step forward, calculate the contribution of each Gaussian ball to the line of sight path, and use the maximum volume rendering formula to determine the final color, in order to ensure that the color of each pixel correctly reflects the combined influence of all relevant Gaussian balls on the line of sight path from the camera to the pixel. In the above-mentioned maximum volume rendering, the final color is determined by calculating the maximum opacity when the light passes through the volume data. This means that for each pixel, the algorithm integrates the opacity of the volume data along the line of sight until a certain threshold is reached, which is usually related to the maximum opacity. This allows the generation of an accurate 3D map where the color and depth information of each pixel reflects the true layout and properties of the objects in the scene.

[0094] Calculate the pixels on each frame p=(u,v) The image error formula is:

[0095] (3)

[0096] in, n Represents the number of Gaussians on the spatial ray of the current pixel point, i is the index in the summation symbol, j is an index into the product notation, where i From 1 to nchanges, indicating the spherical harmonic expansion of i coefficients, j From 1 to i −1 changes, used to multiply the previous coefficients, C(p) Representing pixels p The image error on c i Represents RGB color using spherical harmonic coefficients , f i ( p ) means in pixels p The i The value of the Gaussian function, o represents the amplitude or weight of the Gaussian function, which controls the peak height of the Gaussian distribution. Indicates j Gaussian function at pixel p The complement at that point is 1 minus the value of the Gaussian function at that point. Represents pixels p and mean μ The square of the Euclidean distance between Represents the product symbol, used to calculate the product of a series of values; i Gaussian components for color C ( p )'s contribution, the product Used to eliminate the influence of all previous components j;

[0097] The depth reconstruction error is expressed as:

[0098] (4)

[0099] in, D(p) Representing pixels p The depth reconstruction error on d i The representative depth is expressed by spherical harmonic coefficients;

[0100] The semantic reconstruction error is expressed as:

[0101] (5)

[0102] in, S(p) Representing pixels p The semantic reconstruction error on s i The semantics of the representation is expressed by spherical harmonic coefficients;

[0103] For motion blurred images, the acquisition of blurred images is represented by the integral of the irradiance of the six-degree-of-freedom camera pose during the exposure time. The motion blurred image is regarded as Overlap of real images of several poses:

[0104] (6)

[0105] in, B represents a motion blurred image, Represents a time variable, used to define the exposure time interval, represent Pixels of the moment, and Respectively represent the start and end time of the exposure time interval, Indicates at time The image acquired at each moment is in pixels The intensity or color value at Indicated in i Pixels in the frame The intensity or color value at N Represents the total number of frames.

[0106] Since we don't care about the position at other times, we assume that t-1 and t The two moments are synthesized, so the motion blur reconstruction error is:

[0107]

[0108] B ( p ) represents pixels p We will minimize the sum of these errors and fix other parameters c , d , σ , μ , r ( c Represents RGB color using spherical harmonic coefficients, d The representative depth is expressed by spherical harmonic coefficients, σ Represents transparency, μ represents the center position of the Gaussian sphere, r represents the radius of the Gaussian sphere), only the camera pose is calculated E t .

[0109] Then, based on the obtained 6-DOF camera pose, the Gaussian function is densified to update the map. Specifically, by giving the camera from frame 1 to t +1The camera pose is updated by minimizing the RGB image, depth, semantic, and motion blur irradiance errors on all images up to t + 1. In practice, in order to keep the batch size manageable, a set of keyframes overlapping with the most recent frame is optimized. Through this step, a semantic map of the 3D scene environment is obtained. That is, we also minimize the sum of errors, but calculate instead c, d, σ, μ, r .

[0110] This SLAM solution can not only process complex visual and depth data, but also effectively deal with motion blur in dynamic environments, improving positioning accuracy and the quality of map construction. In addition, through semantic understanding and semantic segmentation methods based on open vocabulary, the system can adapt to unseen objects, enhancing its flexibility and generalization ability in real-world applications.

[0111] Embodiment 3

[0112] Based on the same technical concept, the embodiment of the present application also provides a computer device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the steps of the computer program to implement the above-mentioned anti-shake semantic SLAM method. And realize the following functions: through semantic understanding and semantic segmentation methods based on open vocabulary, the positioning accuracy and the quality of map construction are improved, and its flexibility and generalization ability in real-world applications are enhanced, ensuring that the bipedal humanoid robot can effectively perform SLAM and downstream tasks under various conditions; for the high-shake motion scene of the bipedal humanoid robot, a method for estimating the motion of the six-degree-of-freedom camera from the motion blurred image and restoring a clear three-dimensional scene is proposed, thereby improving the accuracy and stability of the SLAM algorithm in the application of the bipedal humanoid robot; combining the object semantic segmentation algorithm to enhance the SLAM system, providing a richer environmental description than the traditional feature points to supplement the lack of traditional feature points; by integrating the object semantic information into the SLAM algorithm, the system can generate a more detailed map in an environment with sparse features and maintain a more accurate robot positioning.

[0113] Embodiment 4

[0114] Based on the same technical concept, the embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the steps of the above-mentioned anti-shake semantic SLAM method. And realize the following functions: through semantic understanding and semantic segmentation methods based on open vocabulary, the positioning accuracy and the quality of map construction are improved, and its flexibility and generalization ability in real-world applications are enhanced, ensuring that the bipedal humanoid robot can effectively perform SLAM and downstream tasks under various conditions; for the high-shake motion scene of the bipedal humanoid robot, a method for estimating the motion of the six-degree-of-freedom camera pose from the motion blurred image and restoring a clear three-dimensional scene is proposed, thereby improving the accuracy and stability of the SLAM algorithm in the application of the bipedal humanoid robot; combining the object semantic segmentation algorithm to enhance the SLAM system, providing a richer environmental description than the traditional feature points to supplement the lack of traditional feature points; by integrating the object semantic information into the SLAM algorithm, the system can generate a more detailed map in an environment with sparse features and maintain a more accurate robot positioning.

[0115] Embodiment 5

[0116] Based on the same technical concept, the embodiment of the present application also provides a bipedal humanoid robot, which executes any of the anti-shake semantic SLAM methods described above to achieve simultaneous positioning and map construction. And realize the following functions: through semantic understanding and semantic segmentation methods based on open vocabulary, the positioning accuracy and the quality of map construction are improved, and its flexibility and generalization ability in real-world applications are enhanced, ensuring that the bipedal humanoid robot can effectively perform SLAM and downstream tasks under various conditions; for the high-shake motion scenes of bipedal humanoid robots, a method for estimating the motion of a six-degree-of-freedom camera from a motion-blurred image and restoring a clear three-dimensional scene is proposed, thereby improving the accuracy and stability of the SLAM algorithm in the application of bipedal humanoid robots; combining object semantic segmentation algorithms to enhance the SLAM system, providing a richer environmental description than traditional feature points to supplement the lack of traditional feature points; by integrating object semantic information into the SLAM algorithm, the system can generate more detailed maps in environments with sparse features and maintain more accurate robot positioning.

[0117] The above describes specific embodiments of the present application. In addition, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0118] The method steps in the embodiments of the present application can be implemented in hardware or in software instructions that can be executed by a processor. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory, flash memory, read-only memory, programmable read-only memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. The storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a base station, a core network device, or a network element management system. The processor and the storage medium can also exist as discrete components in a base station, a core network device, or a network element management system.

[0119] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present application is executed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device or other programmable device. The computer program or instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instruction may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium may be a magnetic medium, for example, a floppy disk, a hard disk, a tape; it may also be an optical medium, for example, a digital video disc; it may also be a semiconductor medium, for example, a solid-state hard disk. The computer-readable storage medium may be a volatile or nonvolatile storage medium, or may include both volatile and nonvolatile types of storage media.

[0120] The above description is only a preferred embodiment of the embodiments of the present application and is not intended to limit the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the embodiments of the present application.

Claims

1. A shaky semantic SLAM method for a bipedal humanoid robot, characterized in that: The steps include: S1, obtaining continuous video frames captured by a camera, inputting them into a semantic segmentation model, performing semantic segmentation based on an open vocabulary on each of the video frames, and obtaining a segmentation instance result of each frame, wherein the video frame contains color information and depth information of the scene; S2, constructing a Gaussian sphere in a three-dimensional space based on continuous video frames captured by the camera, and performing initialization processing on the Gaussian sphere; wherein the step of performing initialization processing on the Gaussian sphere includes performing preliminary estimation of the six-degree-of-freedom posture of the bipedal humanoid robot and initial construction of a semantic map of the three-dimensional scene; S3, optimizing the six-degree-of-freedom posture of the biped humanoid robot preliminarily estimated in step S2 and the three-dimensional scene semantic map initially constructed, and outputting the optimized six-degree-of-freedom posture and three-dimensional scene semantic map; Among them, the optimization processing step includes estimating the impact of camera shake by constructing a motion blur reconstruction error and minimizing the motion blur reconstruction error. The blurred image corresponding to the motion blur reconstruction error is represented by the integral of the irradiance of the time-varying six-degree-of-freedom posture within the exposure time.

2. The anti-shake semantic SLAM method according to claim 1, characterized in that: The step of performing open vocabulary-based semantic segmentation on each of the video frames includes: using a pre-trained visual language model as a teacher model to provide guidance for the semantic segmentation model, and the semantic segmentation model refines its learned visual concepts into a set of segmentation tokens, wherein each set of segmentation tokens summarizes a local area of ​​the video frame, and the visual concepts include shape, texture, and color of an object.

3. The anti-shake semantic SLAM method according to claim 2, characterized in that: The step of performing semantic segmentation based on an open vocabulary for each of the video frames also includes: based on the three-dimensional vision algorithm Bundle Adjustment, using the entire observation sequence to perform mutual verification between multiple frames.

4. The anti-shake semantic SLAM method according to claim 3, characterized in that: The step of using the entire observation sequence to perform mutual verification between multiple frames includes: adding a view consistency score indicator in the semantic segmentation model to measure whether two masks belong to the same object, wherein the view consistency score indicator is defined as a support rate supporting the merging of two masks in a segment of the observation sequence.

5. The anti-shake semantic SLAM method according to claim 4, characterized in that: After step S3, the method further includes: S4, acquiring new continuous video frames as the camera moves, updating the three-dimensional scene semantic map based on the new continuous video frames, and outputting a real-time running trajectory map of the bipedal humanoid robot.

6. The anti-shake semantic SLAM method according to claim 5, characterized in that: The optimization processing step in step S3 also includes: constructing an image error, a depth reconstruction error and a semantic reconstruction error relative to the current video frame, and optimizing the running trajectory by minimizing the image error, the depth reconstruction error and the semantic reconstruction error.

7. The anti-shake semantic SLAM method according to claim 6, characterized in that: In step S2, a Gaussian sphere in a three-dimensional space is constructed based on continuous video frames captured by the camera, and the Gaussian sphere is initialized. The Gaussian sphere in the three-dimensional space is expressed as: （1） in, f 3D （x） represents the Gaussian sphere, x Represents the coordinates in space, σ Represents transparency, μ represents the center position of the Gaussian sphere, r represents the radius of the Gaussian sphere; To optimize the parameters of the Gaussian distribution to represent the scene, the Gaussian is rendered as a 2D image in a differentiable way: （2） represents the two-dimensional projection of the center of the Gaussian sphere on the rendering plane, represents the two-dimensional projection of the Gaussian sphere radius on the rendering plane, K Represents the camera internal parameters, represents the focal length, d It represents the depth information, which is the component of the z-axis; E t Represents the six-degree-of-freedom pose of the camera.

8. The anti-shake semantic SLAM method according to claim 7, characterized in that: The step of performing optimization processing in step S3 also includes: arranging all Gaussian spheres in depth order, and performing front and back volume rendering using a maximum volume rendering formula.

9. The anti-shake semantic SLAM method according to claim 8, characterized in that: Calculate the pixels on each frame p=(u,v) The image error formula is: （3） in, n Represents the number of Gaussians on the spatial ray of the current pixel point, i is the index in the summation symbol, j is an index into the product notation, where i From 1 to n changes, indicating the spherical harmonic expansion of i coefficients, j From 1 to i −1 changes, used to multiply the previous coefficients, C(p) Representing pixels p The image error on c i Represents RGB color using spherical harmonic coefficients, , f i (p) In pixels p The i The value of the Gaussian function, o represents the amplitude or weight of the Gaussian function, Indicates j Gaussian function at pixel p The complement of Represents pixels p and mean μ The square of the Euclidean distance between represents the product symbol; The depth reconstruction error is expressed as: （4） in, D(p) Representing pixels p The depth reconstruction error on d i The representative depth is expressed by spherical harmonic coefficients; The semantic reconstruction error is expressed as: （5） in, S(p) Representing pixels p The semantic reconstruction error on s i The semantics of the representation is expressed by spherical harmonic coefficients; Motion blurred images are considered Overlap of real images of several poses: （6） in, B represents a motion blurred image, Represents a time variable, used to define the exposure time interval, represent Pixels of the moment, and Respectively represent the start and end time of the exposure time interval, Indicates at time The image acquired at each moment is in pixels The intensity or color value at Indicated in i Pixels in the frame The intensity or color value at , N represents the total number of frames.

10. A computer device comprising a memory and a processor, wherein the memory is used to store a computer program, wherein: The processor is used to execute the computer program to implement the steps of the anti-shake semantic SLAM method as described in any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program runs on a computer or a processor, the computer or the processor executes the steps of the anti-shake semantic SLAM method as described in any one of claims 1 to 9.

12. A bipedal humanoid robot, characterized in that: The bipedal humanoid robot executes the anti-shake semantic SLAM method as described in any one of claims 1 to 9 to achieve simultaneous positioning and map construction.

Images