A semantic map construction method based on RGBD visual segmentation algorithm

Abstract

The present invention discloses a method for constructing a semantic map based on an RGBD visual segmentation algorithm. First, each parameter is initialized, and a visual segmentation neural network model is constructed. Based on this, visual semantic segmentation is performed on the collected image, and the collected image is calculated, and the depth image is converted into point cloud data, and sparse processing is performed. Then, point cloud data with semantic information is generated, and after superimposing with camera extrinsics, it is projected into the actual three-dimensional space, and the spatial grid with semantic information corresponding to each moment is determined. The spatial grids obtained at different moments are superimposed and aged, and a complete semantic map is constructed based on the data collected in time series. The scheme of the present invention has low requirements on equipment, detectable range light, low computing power requirements, and strong robustness. Even after sparseness, the constructed semantic map can achieve dense perception of the surrounding space with an appropriate frame rate, and can achieve a very high degree of precision.

Description

Technical Field

[0001] The present invention belongs to the field of visual perception, and in particular relates to a semantic map construction method based on an RGBD visual segmentation algorithm. Background Art

[0002] Robots can perceive their own location, surrounding obstacles, environmental structures and other information, thereby achieving functions such as path planning, obstacle avoidance, positioning and mapping. Perception mapping technology is a subset of robot perception technology. It refers to the construction of a map model of the environment by processing and analyzing the perception data of the environment so that intelligent robots can perform positioning and path planning in it. High-level perception mapping can connect the environmental perception and understanding of robots and humans, build a unified perception map, and provide services for the actions of robots and humans. Visual perception mapping uses visual sensors such as cameras to obtain environmental information, extract and match features, and build environmental maps. This technology has great advantages in environments with large environmental changes or that require detailed map expression. At the same time, it has the characteristics of large perception range, low sensor requirements, and strong mapping robustness, so it is widely used in robot mapping and navigation.

[0003] Semantic mapping is a key technology in the field of robotics. The goal of high-order semantic mapping is to unify robots and human needs through mapping, match application needs, become easy to understand, and achieve the interactive comfort people expect. Traditional SLAM maps and feature maps are commonly used map types in traditional SLAM algorithms. They are often used in indoor navigation systems and can accurately describe environmental geometric information. However, the lack of semantic information limits the level of robot intelligence, and the results of mapping are not intuitive and cannot establish a unified understanding with humans. Geometric semantic maps have rich semantic information and can provide target object types, spatial forms and positions, as well as the positions of robots in maps. However, only a single geometric body can be used at present, and the representation of target objects is restricted. It is also limited by the computing requirements of deep learning and hardware computing power. Occupancy grid maps have the advantages of precise semantic segmentation, accurate grid occupancy maps and efficient computing power, and can be applied to autonomous driving, robot navigation, virtual reality, games and other fields. The present invention provides an efficient and available method for real-time construction of semantic maps by constructing a high-precision semantic grid map, solving the difficulties of high-order perception of robots in most scenarios. At the same time, building, storing, and using an available, universal, and concise semantic map to solve the redundancy and timeliness problems of traditional maps and to improve the robot's ability to interact with different environments is still one of the current research trends. Summary of the invention

[0004] In view of the above problems, the object of the present invention is to provide a semantic map construction method based on RGBD visual segmentation algorithm.

[0005] The specific technical solution for achieving the purpose of the present invention is:

[0006] A semantic map construction method based on RGBD visual segmentation algorithm includes the following steps:

[0007] Step 1: Initialize various parameters, including camera parameters, build an initial global semantic map, and correct various time information and robot positioning information;

[0008] Step 2: Build a visual segmentation neural network model, and perform visual semantic segmentation on the collected images based on it;

[0009] Step 3: Calculate the collected image, convert the depth image into point cloud data, and perform sparse processing;

[0010] Step 4: Mapping is performed based on the segmented image and point cloud data to generate point cloud data with semantic information;

[0011] Step 5: superimpose the point cloud data with semantic information with the camera external parameters and project them into the actual three-dimensional space to determine the spatial grid with semantic information corresponding to each moment;

[0012] Step 6: Overlay and age the spatial grids acquired at different times, and build a complete semantic map based on the time-series collected data.

[0013] Compared with the prior art, the present invention has the following beneficial effects:

[0014] (1) Compared with common laser radar mapping, the scheme of the present invention has low equipment requirements, detectable light range, low computing power requirements, strong robustness, and high real-time performance; the perception of the surrounding environment is generally based on RGBD cameras. Compared with multi-modal, OCC and other methods, it has the characteristics of relatively simple sensors, large detection range, and relatively low computing power requirements. Visual OCC may require up to 6 cameras to perform perception and mapping work at the same time, which requires extremely high computing power, but this method is relatively faster; multi-modal fusion mapping is a more traditional method with the advantages of multiple sensor fusion; but for the algorithm, the training data requires the simultaneous collection of all sensors, the data set is higher, and the algorithm robustness is weaker; during use, multiple sensor data synchronization is required, and the computing power requirements are also higher. This solution has simple equipment requirements and low computing power requirements, and can usually be deployed on mobile devices. In addition, it has the advantages of real-time mapping, and the mapping accuracy can also be guaranteed. Mapping can be quickly built with strong computing power support;

[0015] In addition, the solution of the present invention can also be easily extended to three-dimensional space mapping, and used to build high-precision three-dimensional maps or three-dimensional reconstruction;

[0016] (2) The semantic map constructed by the present invention is based on a pixel-level panoramic semantic segmentation algorithm. Even after sparse pixel-level semantic segmentation and with an appropriate frame rate, dense perception of the surrounding space can be achieved, and a very high level of precision can be achieved.

[0017] The present invention is further described below in conjunction with specific implementation modes. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flowchart of the semantic map construction method based on the RGBD visual segmentation algorithm of the present invention.

[0019] Figure 2 It is a schematic diagram of image semantic segmentation performed by a visual segmentation neural network in an embodiment of the present invention.

[0020] Figure 3 Schematic diagram of the mapping principle between world coordinate points and image points in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] Example

[0022] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an kind" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0024] Unless otherwise specifically stated, the relative arrangement, numerical expressions and numerical values ​​of the parts and steps set forth in these embodiments do not limit the scope of the present application. At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the accompanying drawings are not drawn according to the actual proportional relationship. The technology, method and equipment known to those of ordinary skill in the relevant field may not be discussed in detail, but in appropriate cases, the technology, method and equipment should be considered as a part of the authorization specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so that once a certain item is defined in an accompanying drawing, it does not need to be further discussed in subsequent drawings.

[0025] Combination Figure 1 , a semantic map construction method based on RGBD visual segmentation algorithm, comprising the following steps:

[0026] Step 1: Initialize all parameters, including camera parameters, build an initial global semantic map, and calibrate the time information and robot positioning information:

[0027] The initial global semantic map is in an unknown state, and the grid state and grid size of the global semantic map are determined;

[0028] The robot positioning information refers to the position and posture information of the robot itself, which is determined based on inertial navigation or GNSS positioning information;

[0029] The camera parameters refer to the device parameters of the monocular RGBD camera, including properties such as focal length, and the video stream refers to the video signal captured by the camera.

[0030] Step 2: Build a visual segmentation neural network model and perform visual semantic segmentation on the collected images based on it:

[0031] The visual segmentation neural network model refers to processing an image using a neural network to obtain a desired semantic segmentation result, wherein each pixel of the image is ultimately converted into a semantic representation representing the category or texture material to which the corresponding pixel in the image belongs;

[0032] The visual segmentation neural network model outputs a semantic segmentation result of the image based on RGB input or RGBD image input;

[0033] After the visual segmentation neural network model is constructed, it is pre-trained based on the data set. Based on the trained visual segmentation neural network model, the camera is connected to obtain the image captured by the camera, and the visual semantic segmentation of the image is performed to obtain the desired semantic segmentation result, wherein each pixel of the image is finally converted into a semantic representation, representing the category or texture material to which the corresponding pixel in the image belongs, such as Figure 2 As shown;

[0034] In addition, the image scale of the semantic segmentation output is consistent with the input resolution. This operation ensures that the output and the depth map taken by the RGBD camera have a pixel-level mapping relationship; here it means that the output result of the semantic segmentation and the depth map should have the same resolution so that they can match at the pixel level.

[0035] Step 3: Calculate the collected image, convert the depth image into point cloud data, and perform sparse processing:

[0036] Step 3-1, connect the camera, obtain the video stream, obtain the RGBD image or RGB image + depth map at time t collected by the camera, and intercept the depth map corresponding to the corresponding D channel;

[0037] Step 3-2: Based on the internal and external parameter information of the camera, perform point cloud conversion on each pixel in the depth map:

[0038]

[0039] Among them, x w ,y w ,z w Represents the three-dimensional coordinate point of each pixel's final transformation; u and v are the coordinate positions of the pixel in the image coordinate system; u0 and v0 are the center coordinates of the image, i.e., the camera intrinsic parameter c of the camera's internal and external parameter information. x , c y ; z c Indicates the z-axis value of the camera coordinates, that is, the distance from the target to the camera; d x / f,d y / f is the camera internal parameter f x , f y By calculating 1 / f x , 1 / f y The specific coordinate transformation derivation process is as follows:

[0040] First, the depth map needs to be converted into a point cloud: the specific process of converting the depth map into a point cloud is to transform the coordinate system, that is, the image coordinate system is converted into the world coordinate system. The constraint condition of the transformation is the camera intrinsic parameter. The formula is as follows:

[0041]

[0042] Among them, x, y, z are the point cloud coordinate system, x', y' are the image coordinate system, and D is the depth value; the derivation process is introduced below;

[0043] In the process of mapping world coordinates to images, consider the process of mapping world coordinate point M(Xw,Yw,Zw) to image point m(u,v). The mapping principle is as follows: Figure 2 As shown;

[0044] The formal representation is as follows:

[0045]

[0046] u,v are arbitrary coordinate points in the image coordinate system. u0,v0 are the center coordinates of the image; x w ,y w ,z w Represents the three-dimensional coordinate point in the world coordinate system. c Represents the z-axis value of the camera coordinates, that is, the distance from the target to the camera. R and T are the 3x3 rotation matrix and 3x1 translation matrix of the external parameter matrix respectively.

[0047] Setting of the external parameter matrix: Since the world coordinate origin and the camera origin coincide, that is, there is no rotation and translation, so:

[0048]

[0049] From the above transformation matrix formula, we can calculate the image point [u,v]T in step 3-2 to the world coordinate point [x w ,y w ,z w ]T's transformation formula.

[0050] In addition, in actual operation, the depth map can be downsampled first, that is, sparse processing, to reduce the resolution of the image, and the low-resolution depth map can be converted into a point cloud to obtain sparse point cloud data. The result after conversion contains fewer point clouds. Since each depth map pixel and RGB image are one-to-one corresponding before sparse processing, the sparse point cloud still has a corresponding relationship with the RGB image (semantic segmentation). Here, since the resolution of the camera is usually relatively high, it may be at a higher resolution even after cropping.

[0051] Step 4: Mapping is performed based on the segmented image and point cloud data to generate point cloud data with semantic information, that is, the semantics of each pixel in the semantic segmentation is assigned to the point cloud data of the corresponding pixel, and the mapping between the image and the point cloud is completed, thereby obtaining the semantic point cloud at the current moment;

[0052] In addition, if the depth map is sparsely processed in step 3, the same operation needs to be performed on the semantic segmentation result output in step 2 to maintain the mapping relationship;

[0053] The final semantic segmentation result and the sparse point cloud should have the same resolution, and the semantics of each pixel in the semantic segmentation can be assigned to the corresponding point cloud; in this way, the semantic point cloud at time t is obtained. Since the previous operation maintains the mapping relationship between the semantics and the point cloud, a simple mapping is sufficient here and no further explanation is given.

[0054] Step 5: Superimpose the point cloud data with semantic information and the camera external parameters and project them into the actual three-dimensional space to determine the spatial grid with semantic information corresponding to each moment:

[0055] According to the requirements, a local grid map is constructed, and the local grid map is obtained from the global grid map. The semantics of the local grid map established here is not unknown, but the latest local grid state is obtained from the global grid map, and it is refreshed based on the known state before time t; the local grid map here refers to the global grid map in step 1 with similar properties, but the description range is the local map centered on the robot itself at time t, and the range may be reduced to cover only the map area that needs to be updated;

[0056] The semantic point cloud at the current moment obtained in step 4 is projected into the grids of the local map, and the point clouds of each semantic in each grid are counted. Finally, the semantic with the highest statistical number is assigned to each grid to form a local semantic map;

[0057] All grid semantics of the final non-empty semantics are pushed into the global semantic map, and the global semantic map is updated.

[0058] Step 6: Overlay and age the spatial grids acquired at different times, and build a complete semantic map based on the time-series collected data:

[0059] After the system is started, the global map maintains communication with the robot positioning module, records the camera position information at each moment, and saves the information in the cache for a long enough time; here the camera position information is obtained from the positioning information connected in step 1. The maintenance of the global map requires continuous tracking of the robot's position and posture information to determine the actual area to update the local map.

[0060] The spatial grids acquired at different times are superimposed and aged, and a complete semantic map is constructed based on the time-series collected data. Specifically:

[0061] The global semantic map receives the local semantic maps at multiple times, combines the robot's positioning information, and uses the local semantic maps to update the global semantic map;

[0062] The global semantic map periodically uses an aging algorithm to perform aging operations on outdated data:

[0063] P'=δP+P•k Δt

[0064] Among them, P is the number of semantic points in the grid before updating, P' is the number of semantic points in the grid after updating, k is the aging factor, and Δt is the number of iterations.

[0065] Compared with common LiDAR mapping, this solution has low equipment requirements, detectable light range, low computing power requirements, strong robustness, and can usually be deployed on mobile devices. It has the advantages of real-time mapping and guaranteed mapping accuracy. With strong computing power support, it can quickly build maps and can be easily expanded to three-dimensional space mapping for building high-precision three-dimensional maps or three-dimensional reconstruction.

[0066] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0067] Step 1: Initialize various parameters, including camera parameters, build an initial global semantic map, and correct various time information and robot positioning information;

[0068] Step 2: Build a visual segmentation neural network model, and perform visual semantic segmentation on the collected images based on it;

[0069] Step 3: Calculate the collected image, convert the depth image into point cloud data, and perform sparse processing;

[0070] Step 4: Mapping is performed based on the segmented image and point cloud data to generate point cloud data with semantic information;

[0071] Step 5: superimpose the point cloud data with semantic information with the camera external parameters and project them into the actual three-dimensional space to determine the spatial grid with semantic information corresponding to each moment;

[0072] Step 6: Overlay and age the spatial grids acquired at different times, and build a complete semantic map based on the time-series collected data.

[0073] A computer storable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0074] Step 1: Initialize various parameters, including camera parameters, build an initial global semantic map, and correct various time information and robot positioning information;

[0075] Step 2: Build a visual segmentation neural network model, and perform visual semantic segmentation on the collected images based on it;

[0076] Step 3: Calculate the collected image, convert the depth image into point cloud data, and perform sparse processing;

[0077] Step 4: Mapping is performed based on the segmented image and point cloud data to generate point cloud data with semantic information;

[0078] Step 5: superimpose the point cloud data with semantic information with the camera external parameters and project them into the actual three-dimensional space to determine the spatial grid with semantic information corresponding to each moment;

[0079] Step 6: Overlay and age the spatial grids acquired at different times, and build a complete semantic map based on the time-series collected data.

[0080] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A semantic map construction method based on RGBD visual segmentation algorithm, characterized in that: The following steps are involved: Step 1: Initialize various parameters, including camera parameters, build an initial global semantic map, and correct various time information and robot positioning information; Step 2: Build a visual segmentation neural network model, and perform visual semantic segmentation on the collected images based on it; Step 3: Calculate the collected image, convert the depth image into point cloud data, and perform sparse processing; Step 4: Map the segmented image and point cloud data to generate point cloud data with semantic information: The semantics of each pixel in the semantic segmentation is assigned to the point cloud data of the corresponding pixel to complete the mapping between the image and the point cloud, thereby obtaining the semantic point cloud at the current moment; Step 5: Superimpose the point cloud data with semantic information and the camera external parameters and project them into the actual three-dimensional space to determine the spatial grid with semantic information corresponding to each moment: The semantic point cloud at the current moment obtained in step 4 is projected into the grid map, and the point cloud of each semantic in each grid is counted. Finally, the semantic with the highest statistical number is assigned to each grid to form a local semantic map. Push all grid semantics of the final non-empty semantics into the global semantic map and update the global semantic map; Step 6: Overlay and age the spatial grids acquired at different times, and build a complete semantic map based on the time-series collected data: The global semantic map receives the local semantic maps at multiple times, combines the robot's positioning information, and uses the local semantic maps to update the global semantic map; The global semantic map periodically uses an aging algorithm to perform aging operations on outdated data: P’=δP+P•k Δt ; Among them, P is the number of semantic points in the grid before updating, P' is the number of semantic points in the grid after updating, k is the aging factor, and Δt is the number of iterations.

2. The semantic map construction method based on the RGBD visual segmentation algorithm according to claim 1 is characterized in that: The camera parameters in step 1 include the camera's intrinsic parameters, extrinsic parameters initialization and calibration; The initial global semantic map is in an unknown state, and the grid state and grid size of the global map are determined; The robot positioning information refers to the position and posture information of the robot itself, which is determined based on inertial navigation or GNSS positioning information.

3. The semantic map construction method based on the RGBD visual segmentation algorithm according to claim 1 is characterized in that: The visual segmentation neural network model in step 2 refers to processing the image using a neural network to obtain the desired semantic segmentation result, wherein each pixel of the image is ultimately converted into a semantic representation representing the category or texture material to which the corresponding pixel in the image belongs; The visual segmentation neural network model outputs a semantic segmentation result of the image based on RGB input or RGBD image input; After the visual segmentation neural network model is constructed, pre-training is performed based on the data set. Based on the trained visual segmentation neural network model, images captured by the camera are obtained to perform visual semantic segmentation of the images. The image scale of the semantic segmentation output is kept consistent with the input resolution.

4. The semantic map construction method based on the RGBD visual segmentation algorithm according to claim 1 is characterized in that: The depth image processing in step 3 includes: Step 3-1, obtain the RGBD image captured by the camera and intercept the depth map corresponding to the D channel; Step 3-2: Based on the internal and external parameter information of the camera, perform point cloud conversion on each pixel in the depth map: ； Among them, x w ,y w ,z w Represents the three-dimensional coordinate point of each pixel's final transformation; u and v are the coordinate positions of the pixel in the image coordinate system; u0 and v0 are the center coordinates of the image, i.e., the camera intrinsic parameter c of the camera's internal and external parameter information. x , c y ; z c Indicates the z-axis value of the camera coordinates, that is, the distance from the target to the camera; d x / f,d y / f is the camera internal parameter f x , f y By calculating 1 / f x , 1 / f y get.

5. The semantic map construction method based on the RGBD visual segmentation algorithm according to claim 4 is characterized in that: In step 3-2, the depth map is first downsampled to reduce the resolution of the image, and the low-resolution depth map is converted into a point cloud to obtain sparse point cloud data.

6. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

7. A computer storable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images