Dense object semantic map construction method and device, storage medium and electronic equipment

Abstract

Embodiments of the present specification disclose a dense object semantic map construction method and device, a storage medium and an electronic device, wherein the method comprises: performing object target detection processing based on an RGB image to obtain two-dimensional detection box information and object category information, and performing visual SLAM processing based on the RGB image to obtain visual SLAM, determining three-dimensional map point cloud and camera pose in the visual SLAM, constructing an initial object semantic map based on the RGB image, the two-dimensional detection box information, the object category information, the three-dimensional map point cloud and the camera pose, constructing a global scene point cloud based on the RGB image, a Depth image and the camera pose of the visual SLAM, filtering the global scene point cloud to obtain a dense point cloud map, and performing map fitting on the initial object semantic map and the dense point cloud map to obtain a dense object semantic map in a current scene.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method, apparatus, storage medium, and electronic device for constructing dense object semantic maps. Background Technology

[0002] With the advancement of science and technology and the development of artificial intelligence, intelligent devices such as intelligent robots, intelligent sweeping robots, and intelligent mopping robots have been applied to various fields. The use of image sensors by intelligent devices to build more robust, accurate, and detailed maps is very beneficial for enhancing the application of intelligent devices in various scenarios. In practical applications, maps need to provide dense and object-level semantic mapping, that is, they need to be able to build dense object semantic maps. Summary of the Invention

[0003] This specification provides an embodiment of a method, apparatus, storage medium, and electronic device for constructing dense object semantic maps. The technical solution is as follows:

[0004] Firstly, embodiments of this specification provide a method for constructing a dense object semantic map, the method comprising:

[0005] Acquire RGBD image data of the scene, wherein the RGBD image data includes RGB images and depth images;

[0006] Based on the RGB image, object target detection processing is performed to obtain two-dimensional detection box information and object category information; and based on the RGB image, visual SLAM processing is performed to obtain visual SLAM.

[0007] The 3D map point cloud and camera pose in the visual SLAM are determined, and an initial object semantic map is constructed based on the RGB image, the 2D detection box information, the object category information, the 3D map point cloud and the camera pose.

[0008] A global scene point cloud is constructed based on the RGB image, the Depth image, and the camera pose of the visual SLAM. The global scene point cloud is then filtered to obtain a dense point cloud map.

[0009] Map fitting is performed on the initial object semantic map and the dense point cloud map to obtain a dense object semantic map of the scene.

[0010] Secondly, embodiments of this specification provide a dense object semantic map construction apparatus, the apparatus comprising:

[0011] The image acquisition module is used to acquire RGBD image data of the scene, wherein the RGBD image data includes RGB images and depth images;

[0012] The target detection module is used to perform object target detection processing based on the RGB image to obtain two-dimensional detection box information and object category information, and to perform visual SLAM processing based on the RGB image to obtain visual SLAM.

[0013] The semantic mapping module is used to determine the 3D map point cloud and camera pose in the visual SLAM, and to construct an initial object semantic map based on the RGB image, the 2D detection box information, the object category information, the 3D map point cloud and the camera pose;

[0014] The semantic mapping module is used to construct a global scene point cloud based on the RGB image, the Depth image, and the camera pose of the visual SLAM, and to filter the global scene point cloud to obtain a dense point cloud map.

[0015] The semantic mapping module is used to perform map fitting on the initial object semantic map and the dense point cloud map to obtain a dense object semantic map of the scene.

[0016] Thirdly, embodiments of this specification provide a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the above-described method steps.

[0017] Fourthly, embodiments of this specification provide an electronic device that may include: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the above-described method steps.

[0018] The beneficial effects of the technical solutions provided in some embodiments of this specification include at least the following:

[0019] In one or more embodiments of this specification, two-dimensional detection box information, object category information, and visual SLAM are first obtained based on RGB images. An initial object semantic map is then constructed based on the RGB images, the two-dimensional detection box information, the object category information, the three-dimensional map point cloud in the visual SLAM, and the camera pose. A global scene point cloud is then constructed based on the RGB images, depth images, and the camera pose. The global scene point cloud is filtered to obtain a dense point cloud map. Finally, map fitting is performed between the initial object semantic map and the dense point cloud map to obtain a dense object semantic map of the scene. This dense object semantic map combines density characteristics with object-level semantic characteristics, achieving high-precision dense object semantic mapping of the scene, while the RGBD image data used for mapping meets the requirements of a low-cost system. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a method for constructing a dense object semantic map provided in the embodiments of this specification;

[0022] Figure 2 This is a schematic diagram of a target detection processing framework provided in the embodiments of this specification;

[0023] Figure 3 This is a visual SLAM framework diagram provided in the embodiments of this specification;

[0024] Figure 4 This is a schematic diagram of an initial object semantic map provided in the embodiments of this specification;

[0025] Figure 5 This is a schematic diagram of a dense point cloud map provided in the embodiments of this specification;

[0026] Figure 6 This is a schematic diagram of a dense object semantic map provided in the embodiments of this specification;

[0027] Figure 7 This is a flowchart illustrating a lightweight object semantic map construction method provided in the embodiments of this specification.

[0028] Figure 8 This is a scene illustration of a lightweight object semantic map construction provided in the embodiments of this specification;

[0029] Figure 9 This is a flowchart illustrating a method for constructing a dense point cloud map provided in the embodiments of this specification;

[0030] Figure 10 This is a schematic diagram of a scene for constructing a dense point cloud map, as provided in the embodiments of this specification.

[0031] Figure 11 This is a schematic diagram of a loop optimization process provided in the embodiments of this specification;

[0032] Figure 12 This is a schematic diagram of the structure of a dense object semantic map construction device provided in the embodiments of this specification;

[0033] Figure 13 This is a structural block diagram of an electronic device provided in the embodiments of this specification. Detailed Implementation

[0034] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0035] In the description of this specification, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In the description of this specification, it should be noted that, unless otherwise expressly specified and limited, "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. Those skilled in the art can understand the specific meaning of the above terms in this specification based on the specific circumstances. Furthermore, in the description of this specification, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0036] In related technologies, visual semantic mapping frameworks can be mainly divided into the following categories. One category is based on visual SLAM, loosely coupled with object detection or segmentation. Common visual SLAM frameworks mainly include the ORB-SLAM and VINS series, object detection mainly uses the YOLO and SSD series, and segmentation mainly uses Mask-RCNN and DeepLab. Overall frameworks include DS-SLAM, DSP-SLAM, ORBSLAM2-SSD, and Cube-SLAM. However, this approach either does not provide a dense semantic map or does not provide specific 3D object detection boxes. Another category achieves dense semantic mapping by tightly coupling the visual SLAM system with the reconstructive neural network. The main framework is the Kimera series. This approach has weak redevelopment capabilities and requires certain computer performance. It also cannot provide 3D object detection boxes for downstream module applications, such as localization and navigation.

[0037] The technical solutions mentioned above mainly utilize the loose coupling between the visual SLAM framework and the neural network to achieve semantic mapping, but they cannot simultaneously provide both dense and object-level semantic mapping, nor can they meet the requirements of low-cost systems.

[0038] The present specification will now be described in detail with reference to specific embodiments.

[0039] In one embodiment, such as Figure 1 As shown, a method for constructing dense object semantic maps is proposed. This method can be implemented using a computer program and can run on a dense object semantic map construction device based on the von Neumann architecture. This computer program can be integrated into applications or run as a standalone utility application. The dense object semantic map construction device can be a smart device.

[0040] Specifically, the method for constructing this dense object semantic map includes:

[0041] S102: Obtain RGBD image data of the scene, wherein the RGBD image data includes RGB image and Depth image;

[0042] Optionally, RGBD image data can be acquired by the included RGBD camera, the RGBD image data including RGB images and depth images;

[0043] S104: Perform object target detection processing based on the RGB image to obtain two-dimensional detection box information and object category information, and perform visual SLAM processing based on the RGB image to obtain visual SLAM;

[0044] As an illustration, object detection processing methods can include Faster R-CNN object detection, YOLO object detection, etc.

[0045] Optionally, the following illustrates one method for object detection:

[0046] like Figure 2 As shown, Figure 2 This is a schematic diagram of an object detection processing framework; the object detection processing framework can be regarded as an object detection processing model.

[0047] 1. Using RGBD image data acquired by an RGBD camera, the RGB images in the RGBD image data are used as the input to the target detection processing framework. First, the RGB images are preprocessed to meet the input requirements of the target detection processing framework network.

[0048] 2. Secondly, the backbone network of the object detection processing model is... Figure 2The CSPDarknet53 feature extraction network shown here contains five residual networks, Resblock_body, which progressively reduces the pixel resolution of the RGB image to extract higher-level semantic information. At the same time, the last three feature layers are selected as typical feature layers for subsequent networks.

[0049] 3. The 5th feature layer is taken as the highest feature layer and subjected to SPP (Spatial Pyramid Pooling) processing. The feature images are subjected to max pooling with kernels of 13, 9, 5 and 1 respectively, and the max pooling results are stacked to expand the receptive field.

[0050] 4. Treating the 3rd and 4th feature layers as the top and middle feature layers respectively, PANet (Path Aggregation Network) is applied to these two feature layers of the target detection processing model to enhance the features of the feature network. First, the output layer of SPP is upsampled and stacked with the middle feature layer, then upsampled again and stacked with the top feature layer as the target detection head to detect large targets. Second, the stacked top feature layer is downsampled and stacked with the middle feature layer as the target detection head to detect medium targets. Finally, the stacked middle feature layer is downsampled and stacked with the output layer of SPP as the target detection head to detect small targets.

[0051] 5. The last three heads of the target detection processing model, from top to bottom, represent the detection of large, medium, and small targets, respectively. The information corresponding to large, medium, and small targets is used as the two-dimensional detection box information. Target detection is also the determination of the object category information corresponding to the two-dimensional detection boxes (large, medium, and small targets).

[0052] This completes the entire object detection process, providing corresponding semantic information, including two-dimensional detection boxes and object categories, for subsequent downstream applications.

[0053] Visual SLAM, or Simultaneous Localization and Mapping, involves locating a user's position and pose by repeatedly observing map features during movement, and then constructing a map based on that position and pose. This achieves simultaneous localization and map building. Maps generated by visual SLAM exist as point clouds (such as 3D point clouds), typically only providing a rough outline and lacking detail; they can only generate sparse or semi-dense point cloud maps.

[0054] In one feasible implementation, such as Figure 3 As shown, Figure 3 It is a visual SLAM framework diagram, based on Figure 3The following processing procedure can be used as a reference:

[0055] 1. The RGB image acquired by the RGBD camera is used as input, and the intrinsic parameters of the image are known, and the intrinsic parameters are already accurate values ​​after distortion correction; optionally, both RGB image and Depth image can be used as input.

[0056] 2. Visual odometry processing in visual SLAM mainly involves ORB feature extraction and feature matching of images to calculate the relative motion information between images, thereby inferring the camera pose.

[0057] a. First, the initial transformation matrix T cw (Note that we consider the first frame as the world coordinate system.) The solution is obtained by reverse engineering based on the homography matrix and fundamental matrix derived from the first two frames, as shown in formulas 1 and 2. Here, H represents the homography matrix and has eight degrees of freedom; generally, h is set to... 33 =1, x1, x2 represent the matching pixel coordinates in the two images, requiring four matching point pairs to solve the equation; F represents the fundamental matrix, which has 7 degrees of freedom and is generally solved using the eight-point method. Then, based on H and F, the rotation matrix R and translation matrix t, i.e., T, can be solved using the SVD decomposition method. cw It can be obtained.

[0058]

[0059]

[0060] b. Secondly, solve P w T obtained through the above solution cw Matching with image pixels allows us to solve for the final world point coordinates using SVD decomposition, as shown in Formula 3, where s is the depth value, x is the normalized planar coordinate, and T is the transformation matrix, i.e., T cw X is the world point we are solving for, i.e., P. w P can be obtained by using SVD decomposition. w .

[0061]

[0062]

[0063] C. After obtaining the initial camera pose, it is necessary to determine the camera's motion state and then estimate the camera pose.

[0064] i. First, assume the camera moves at a constant speed (pose unchanged), and its uniform motion model is shown in Equation 4, where This represents the transformation matrix from the world coordinate system to the previous frame image; This represents the transformation matrix from the image of the previous frame to the image of the current frame; This represents the transformation matrix from the world coordinate system to the current frame. Then, by projecting the world points onto the pixel coordinate system of the previous frame, matching point pairs are searched within a certain radius. If the number of matches is small, the state model is changed.

[0065]

[0066] In i, the process of calculating the reprojection is as follows: from the world coordinate system P... w To the image pixel coordinate system P p The transformation formula is shown in Formula 5 below, which includes the transformation from the world coordinate system P w To camera coordinate system P c =T cw P w From camera coordinate system P c To the image pixel coordinate system P p An identity matrix dimension transformation matrix is ​​needed in the middle; d is the scale or depth information, f x ,f y ,c x ,c y These are the camera's intrinsic parameters, the known data; the unknown parameter is P. w and T cw However, it can be obtained from the previous steps (a) and (b).

[0067] ii. If the above threshold conditions are not met, the transformation matrix is ​​directly solved based on the matching pairs, as follows: Based on the existing 3D coordinates of the world points and their corresponding 2D coordinates, the camera pose is solved using the PNP method, which is a variation of Equation 5, as shown in Equation 6. Equation 6 solves for the 12 unknowns of matrix A for the point pairs and decomposes matrix A to obtain the corresponding transformation matrix.

[0068]

[0069]

[0070] 3. Backend optimization mainly involves optimizing the preliminary camera pose estimation mentioned above. First, keyframes are selected to improve the real-time performance of the system. Second, outlier removal is performed on the map points to eliminate noise, and new map points are generated based on triangulation. The specific formula is shown in 3. Then, local BA optimization is performed on the map points and keyframes. Finally, the keyframes are verified and duplicate keyframes are removed.

[0071] 4. Loop closure detection mainly determines whether the intelligent device returns to the same position, thereby eliminating the cumulative error in the visual odometry.

[0072] 5. Map building mainly includes global BA optimization of all keyframes and map points, and then updating the map to obtain a globally consistent environment map and camera trajectory, thus completing the visual SLAM system.

[0073] S106: Determine the 3D map point cloud and camera pose in the visual SLAM, and construct an initial object semantic map based on the RGB image, the 2D detection box information, the object category information, the 3D map point cloud and the camera pose;

[0074] The initial object semantic map can be understood as a lightweight object semantic map, which does not contain depth information;

[0075] Visual SLAM contains a 3D map point cloud and camera pose. Based on the object's 2D detection box information and the object category information, an object semantic is formed. This object semantic is associated with the 3D map point cloud, and BA optimization is performed in combination with the camera pose, object pose and scale to achieve lightweight object semantic mapping and obtain an initial object semantic map.

[0076] Indicative, such as Figure 4 As shown, Figure 4 This is a schematic diagram of an initial object semantic map. By determining the 3D map point cloud and camera pose in visual SLAM, an initial semantic map can be constructed based on RGB images, 2D bounding box information, object category information, 3D map point cloud, and camera pose. Figure 4 The initial object semantic map shown.

[0077] S108: Construct a global scene point cloud based on the RGB image, the Depth image, and the camera pose of the visual SLAM; filter the global scene point cloud to obtain a dense point cloud map.

[0078] The dense point cloud map has high precision; for example, each pixel can have a corresponding three-dimensional spatial point, so no important spatial information is missed.

[0079] Indicative, such as Figure 5 As shown, Figure 5 This is a schematic diagram of a dense point cloud map. It is constructed by using camera pose based on RGB images, depth images, and visual SLAM to build a global scene point cloud. Then, the global scene point cloud is filtered, such as by Voxel filtering, to eliminate the influence of noise and control the density of the point cloud, thus obtaining a dense point cloud map.

[0080] S110: Perform map fitting on the initial object semantic map and the dense point cloud map to obtain a dense object semantic map of the scene.

[0081] The dense object semantic map contains object semantic information in addition to object geometry and appearance;

[0082] Optionally, the initial object semantic map can be overlaid onto the dense point cloud map to obtain a dense object semantic map of the scene.

[0083] Indicative, such as Figure 6 As shown, Figure 6 This is a schematic diagram of a dense object semantic map, which is obtained by overlaying an initial object semantic map and a dense point cloud map into layers, resulting in a dense object semantic map of the scene.

[0084] Furthermore, it can be applied to indoor semantic mapping of intelligent devices such as intelligent mobile robots. Once a dense object semantic map is obtained, the intelligent device can better understand its surroundings and achieve more intelligent human-machine / object-machine interaction based on semantic information, thus becoming more accurate and efficient in navigation, obstacle avoidance, and interaction. Using the method steps shown in one or more embodiments of this specification, even in scenarios without positioning signals, this solution can achieve accurate dense object semantic mapping to enable autonomous navigation and obstacle avoidance for intelligent devices.

[0085] In one or more embodiments of this specification, two-dimensional detection box information, object category information, and visual SLAM are first obtained based on RGB images. An initial object semantic map is then constructed based on the RGB images, the two-dimensional detection box information, the object category information, the three-dimensional map point cloud in the visual SLAM, and the camera pose. A global scene point cloud is then constructed based on the RGB images, depth images, and the camera pose. The global scene point cloud is filtered to obtain a dense point cloud map. Finally, map fitting is performed between the initial object semantic map and the dense point cloud map to obtain a dense object semantic map of the scene. This dense object semantic map combines density characteristics with object-level semantic characteristics, achieving high-precision dense object semantic mapping of the scene, while the RGBD image data used for mapping meets the requirements of a low-cost system.

[0086] Please see Figure 7 , Figure 7 This is a flowchart illustrating a lightweight object semantic map construction method proposed in this specification. Specifically:

[0087] S2002: Perform line detection and segmentation processing on the grayscale image corresponding to the RGB image to obtain a two-dimensional line detection result;

[0088] like Figure 8 As shown, Figure 8 This is a scene illustration of a lightweight object semantic map construction method described in this specification.

[0089] exist Figure 8 In this process, line detection is performed based on the grayscale image corresponding to the RGB image to obtain two-dimensional direct detection results such as two-dimensional lines and two-dimensional edges;

[0090] Understandably, by using the RGB image captured by the RGBD camera as input, the corresponding grayscale image can be determined. At the same time, line detection and segmentation processing is performed on the grayscale image, which can achieve fast line segment detection without parameter adjustment, thereby obtaining the two-dimensional line detection result.

[0091] To illustrate, let's take one of the optional processing methods for line detection and segmentation as an example, as follows:

[0092] First, image scaling: Gaussian downsampling is performed on the grayscale image to scale it. The purpose of scaling is to reduce or even eliminate the jagged effect in the image. For example, the sampling rate can be set to scale=0.8 to make the line detection accuracy higher.

[0093] Secondly, gradient calculation and sorting: Calculate the magnitude and direction of the gradient at all pixels (x, y) in the image, and sort them according to the gradient values. The gradient calculation method is shown in Formula 7, where g represents the gradient, i(x, y) represents the grayscale value at pixel (x, y), and the calculation is performed using a 2*2 mask template.

[0094]

[0095] The magnitude G of the image gradient and the angle θ of the row and column lines can be calculated using Equation 8;

[0096]

[0097] In gradient sorting, the regions where pixels with higher gradient magnitudes are located usually have stronger edges. Among the edges, the middle pixels generally have the highest gradient magnitudes. Subsequently, line segmentation detection is performed starting from the pixels with the highest gradient magnitudes.

[0098] Next, threshold detection: State information is set for all pixels according to the gradient threshold ρ. Pixels with gradients less than the threshold are set to UNUSED, and the rest are set to USED. This is because pixels with gradients less than the threshold correspond to smooth or slowly changing regions in the image, which can cause significant errors during quantization; therefore, these small gradient pixels need to be removed. The specific threshold calculation formula is shown in Formula 9. Optionally, q = 2, and τ is the maximum tolerance gradient direction threshold, typically set to 22.5.

[0099]

[0100] Then, region growing: Perform region growing, using the maximum gradient value as the seed point, and the gradient direction corresponding to the seed point as the initial angle of the region. Find points in the neighborhood whose initial angle is less than τ, and set the generated region as region.

[0101] Rectangle estimation: Calculate the approximate rectangle. The centroid of the rectangle is shown in Equation 10, where G represents the gradient magnitude, j represents a point in the region, x represents the x-axis pixel of this point, y represents the x-axis pixel of this point, and c represents the centroid of the approximate rectangle. The direction of the rectangle is shown in Equation 11, where the direction of the rectangle is the angle of the eigenvector associated with the smallest eigenvalue of matrix M.

[0102]

[0103]

[0104] Next, calculate the density as shown in Formula 12, where r represents the rectangle mentioned above, length is the length of the rectangle, and width is the width of the rectangle. Illustratively, if D is greater than the truncation threshold of 0.7, the line detection is considered valid.

[0105]

[0106] The above method was used to perform line detection and segmentation on the grayscale image corresponding to the RGB image, and a two-dimensional line detection result was obtained.

[0107] S2004: Based on the two-dimensional detection box information, the object category information, and the three-dimensional map point cloud, perform object association processing to obtain an object association dataset;

[0108] In one feasible implementation, object association is handled as follows:

[0109] A2: Determine the object category semantics of the object corresponding to the two-dimensional detection box information;

[0110] A4: Based on the two-dimensional detection box information, establish an object semantic mapping between the three-dimensional map point cloud and the object category semantics to obtain an object association dataset. The object association dataset consists of at least one three-dimensional map point cloud, the associated mapping data between the object category semantics corresponding to the three-dimensional map point cloud and the two-dimensional detection box information.

[0111] Indicative, such as Figure 8 As shown, by combining the information of the two-dimensional detection box (2D Box) and its corresponding object category information (Label) with the three-dimensional map point cloud (2D Points), the data is associated to obtain the object association dataset, which can then be used to determine whether to merge the objects.

[0112] S2006: Based on the two-dimensional line detection results and the object association dataset, perform object pose estimation processing to obtain the three-dimensional detection box pose information of the object;

[0113] This example illustrates how to initially estimate a 3D bounding box by combining an object association dataset and 2D line detection results. Here, it's assumed the object is parallel to the ground; therefore, only the yaw angle, translation matrix, and scale need to be estimated as the object's 3D bounding box pose information.

[0114] In one feasible implementation, the object pose estimation process is as follows:

[0115] B2: Based on the two-dimensional line detection results and the object association dataset, the isolated forest detection method is used to predict the translation matrix and object scale of the object;

[0116] This illustration demonstrates how the translation matrix *t* and object scale *s* of an object are estimated using the Iforest isolated forest detection method. The key idea behind this method is to recursively separate the data space into a series of isolated data points, then identify easily separable data points as outliers. The principle is that normal points are usually located closer together, thus requiring more steps to isolate, while outliers are typically less scattered and can be easily isolated with fewer steps.

[0117] B4: Determine the consistency evaluation score between the two-dimensional line detection result and the two-dimensional detection box information in the object association dataset; determine the target two-dimensional detection box information based on the consistency evaluation score using the minimum consistency error constraint; and determine the yaw angle of the object using the two-dimensional line segment angle in the target two-dimensional detection box information.

[0118] The yaw angle is estimated based on the angle of the line segment corresponding to the minimum consistency error constraint as the initial yaw angle. The main process calculates the consistency between the 2D detection box and the line detection results. The higher the consistency score, the closer the detection box and the line detection results are, indicating that the detection box is more accurate. We then use this 2D detection box as input to calculate the initial pose of the object's 3D detection box.

[0119] The formula for calculating the consistency evaluation score is shown in Figure 13, where N all N is the number of all line segments in the detection frame. used E(e) is the number of line segments whose edge error e between the projection of the 3D detection box onto the 2D image and the line detection is less than 0.5. E(e) is the average error, and s is the final score.

[0120]

[0121] B6: Obtain the three-dimensional bounding box pose information of the object based on the translation matrix of the object, the object scale, and the yaw angle.

[0122] The translation matrix, the object scale, and the yaw angle are used as the pose information of the object's 3D bounding box, i.e., the 3D bounding box pose information.

[0123] S2008: Based on the camera pose, perform BA optimization processing on the 3D detection box pose information of the object to obtain an initial object semantic map.

[0124] BA (Bundle Adjustment) optimization: also known as bundle adjustment optimization, BA can be regarded as an optimization model, and its essence is to minimize the error.

[0125] To illustrate, BA optimization is performed by combining camera pose and 3D bounding box pose information (i.e., yaw angle, translation matrix, and scale information of the 3D box).

[0126] Its optimization function is shown in Equation 14, where f is the optimization parameter error, and e(θ) is the optimization parameter error. y Let be the orientation angle error, e(s) be the orientation angle error, e(p) be the error between the projection of the 3D object bounding box onto the 2D image and the detection of a line segment parallel to it, and e(p) be the camera pose error of the SLAM system. Thus, we have obtained... Figure 8 The lightweight object semantic map shown is an example.

[0127] f = argmin∑(e(θ) y )+e(s))+argmin∑e(p)(Formula 14)

[0128] In one or more embodiments of this specification, a lightweight object semantic map construction method is provided. By combining the 2D detection boxes and categories obtained from the object detection in the aforementioned steps with the 3D map point cloud and camera pose in visual SLAM, a lightweight initial object semantic map is rapidly constructed. This initial object semantic map reflects the object semantics of objects in the scene.

[0129] Please see Figure 9 , Figure 9 This is a flowchart illustrating a dense point cloud map construction method proposed in this specification. Specifically:

[0130] S3002: Construct at least one frame of the first three-dimensional object point cloud in the camera coordinate system based on the RGB image and the Depth image;

[0131] The first 3D object point cloud is a 3D object point cloud constructed in the lower camera coordinate system;

[0132] Please refer to Figure 10 , Figure 10 This is a schematic diagram of a scenario involving the construction of a dense point cloud map, as described in this manual.

[0133] C2: Determine the three-channel RGB information, depth information, and feature point information based on the RGB image and the Depth image;

[0134] like Figure 10 As shown, 1. RGB images and depth images acquired by an RGBD camera are used as inputs for constructing a dense point cloud map. 2. Then, three-channel RGB features are extracted from the RGB images, depth information features are extracted from the depth images, and feature point information is determined based on the RGB and depth images.

[0135] C4: Construct at least one frame of the first three-dimensional object point cloud in the camera coordinate system based on the three-channel RGB information, the depth information, and the feature point information.

[0136] like Figure 10 As shown in Figure 3, based on the three-channel RGB information, the depth information and the feature point information, a three-dimensional object point cloud of “XYZRGB type” can be constructed for each frame. The three-dimensional object point cloud of “XYZRGB type” is a point cloud in the camera coordinate system, also known as the first three-dimensional object point cloud.

[0137] S3004: Based on the at least one frame of the first three-dimensional object point cloud and the camera pose of the visual SLAM, perform point cloud coordinate system transformation to obtain at least one frame of the second three-dimensional object point cloud in the world coordinate system, and generate a global scene point cloud based on the at least one frame of the second three-dimensional object point cloud.

[0138] Indicatively, based on the camera pose of the visual SLAM, the point cloud pose of at least one frame of the first three-dimensional object point cloud is adjusted to obtain at least one frame of the second three-dimensional object point cloud in the world coordinate system.

[0139] The first 3D object point cloud in the camera coordinate system of each frame is rotated / translated based on the camera pose obtained by visual SLAM to adjust the point cloud pose, thereby obtaining the 3D point cloud in the world coordinate system. The "3D point cloud in the world coordinate system" is also the second 3D object point cloud. All second 3D object point clouds are stored and updated to the global scene point cloud.

[0140] Furthermore, based on the camera pose, the transformation relationship between the camera coordinate system and the world coordinate system can be determined. Based on this transformation relationship, the first three-dimensional object point cloud in the camera coordinate system can be transformed to the world coordinate system in each frame, thereby obtaining at least one frame of the second three-dimensional object point cloud in the world coordinate system.

[0141] For example, the camera coordinate system at the initial moment can be used as the world coordinate system. For any point in space, its coordinate values ​​in the world coordinate system and the camera coordinate system conform to the transformation relationship, which can be characterized by the translation vector t and the rotation matrix R.

[0142] S3006: Filter the global scene point cloud to obtain a dense point cloud map.

[0143] A dense point cloud map is obtained by filtering the global scene point cloud using a preset filtering method.

[0144] Schematic illustration: The filtering process can be Voxel filtering. Voxel filtering is applied to the global point cloud to eliminate noise and control point cloud density, thus obtaining a dense point cloud map (e.g., ...). Figure 10 The Dense Map shown is shown.

[0145] In one or more embodiments of this specification, a method for constructing dense point cloud maps is provided. This method makes full use of RGB images and depth images, is easy to construct, and can ensure the correctness and accuracy of dense mapping.

[0146] In one feasible implementation, considering that the dense point cloud map or lightweight initial object semantic map may have drift due to the application of camera poses from the SLAM system, the point cloud information during the dense point cloud map construction process can be added to the aforementioned SLAM system for SLAM updates and global loop closure detection optimization. In this way, SLAM can update the 3D map point cloud and camera poses, and based on the updated 3D map point cloud and camera poses, the mapping can be corrected, reducing the error of dense mapping and improving the accuracy of the object semantic map.

[0147] Furthermore, after executing S3002: constructing at least one frame of the first 3D object point cloud in the camera coordinate system based on the RGB image and the Depth image, the executing entity also executes as follows: Figure 11 The flowchart shown illustrates the steps. Figure 11 This is a flowchart illustrating a loop optimization process, as shown below:

[0148] S4002: Use the at least one frame of the first three-dimensional object point cloud as a key frame for the visual SLAM, and trigger the back-end optimization process and loop closure detection process for the visual SLAM based on the key frame.

[0149] Here, at least one frame of the first three-dimensional object point cloud is used as a key frame in visual SLAM. The first three-dimensional object point cloud is a dense three-dimensional object point cloud, which is of better quality than the object point cloud in the SLAM system. It is introduced into visual SLAM as a key frame for key frame insertion in SLAM, thereby triggering the subsequent visual SLAM process.

[0150] S4004: In the backend optimization process, local BA optimization is performed on the map points and keyframes in the visual SLAM. In the loop closure detection process, global loop closure detection optimization is performed on the visual SLAM to obtain the target 3D map point cloud and the target camera pose.

[0151] The SLAM backend optimization process mainly optimizes camera pose estimation. It improves the real-time performance of the system based on new keyframes, removes outliers from map points in the 3D object point cloud to eliminate noise, and generates new map points based on triangulation. Then, it performs local BA optimization on the map points and keyframes. Finally, it verifies the keyframes and removes duplicate keyframes.

[0152] In the SLAM loop closure detection process, it is determined whether the device has returned to the same position. That is, by detecting whether the target vehicle has reached the previous environment, it is determined whether a loop closure has occurred. Loop closure reduces the cumulative error of the system's visual odometry and optimizes the entire visual SLAM system. The visual SLAM system module will optimize the pose of the target vehicle and the environmental map based on the loop closure information passed in when it occurs.

[0153] After completing the above process, the visual SLAM system is updated and optimized, and then the target 3D map point cloud and target camera pose can be obtained from the SLAM system.

[0154] S4006: Use the target 3D map point cloud and the target camera pose as the 3D map point cloud and the camera pose, and execute the step of determining the 3D map point cloud and camera pose in the visual SLAM and / or execute the step of constructing a global scene point cloud based on the RGB image, the Depth image and the camera pose of the visual SLAM.

[0155] Understandably, after obtaining the updated 3D map point cloud and target camera pose, on the one hand, constructing a lightweight initial object semantic map involves the 3D map point cloud and target camera pose. Based on this, the steps of determining the 3D map point cloud and camera pose in the visual SLAM can be performed to update the initial object semantic map (i.e., S106). On the other hand, the construction of a dense point cloud map involves the camera pose of the visual SLAM. Based on this, the steps of constructing a global scene point cloud based on the RGB image, the Depth image, and the camera pose of the visual SLAM can be performed, and the global scene point cloud can be filtered to obtain a dense point cloud map (i.e., S108).

[0156] In one or more embodiments of this specification, point cloud information from step S3002 is added to the dense point cloud map building process in the loop closure detection section of the SLAM system to correct the mapping and calibrate the camera pose, thereby reducing mapping errors and improving mapping accuracy. Triggering loop closure detection during dense point cloud map building ensures the correctness and accuracy of the dense mapping. The entire process can realize the construction of a complete dense object semantic map of a scene, such as an indoor scene.

[0157] The following will combine Figure 12 This specification provides a detailed description of the dense object semantic map construction apparatus provided in the embodiments. It should be noted that... Figure 12 The dense object semantic map construction apparatus shown is used to execute this specification. Figures 1 to 11 The methods shown in the embodiments are illustrated for ease of explanation, showing only the parts related to the embodiments of this specification. For specific technical details not disclosed, please refer to this specification. Figures 1 to 11 The example shown.

[0158] Please see Figure 2 This diagram illustrates the structure of a dense object semantic map construction device according to an embodiment of this specification. The dense object semantic map construction device 1 can be implemented as all or part of a user terminal through software, hardware, or a combination of both. According to some embodiments, the dense object semantic map construction device 1 includes an image acquisition module 11, an object detection module 12, and a semantic mapping module 13, specifically used for:

[0159] Image acquisition module 11 is used to acquire RGBD image data of the scene, wherein the RGBD image data includes RGB image and Depth image;

[0160] The target detection module 12 is used to perform object target detection processing based on the RGB image to obtain two-dimensional detection box information and object category information, and to perform visual SLAM processing based on the RGB image to obtain visual SLAM.

[0161] The semantic mapping module 13 is used to determine the 3D map point cloud and camera pose in the visual SLAM, and to construct an initial object semantic map based on the RGB image, the 2D detection box information, the object category information, the 3D map point cloud and the camera pose;

[0162] The semantic mapping module 13 is used to construct a global scene point cloud based on the RGB image, the Depth image and the camera pose of the visual SLAM, and to filter the global scene point cloud to obtain a dense point cloud map.

[0163] The semantic mapping module 13 is used to perform map fitting on the initial object semantic map and the dense point cloud map to obtain a dense object semantic map of the scene.

[0164] Optionally, the semantic mapping module 13 is used for:

[0165] Perform line detection and segmentation processing on the grayscale image corresponding to the RGB image to obtain a two-dimensional line detection result;

[0166] Based on the two-dimensional detection box information, the object category information, and the three-dimensional map point cloud, object association processing is performed to obtain an object association dataset.

[0167] Based on the two-dimensional line detection results and the object association dataset, object pose prediction processing is performed to obtain the three-dimensional detection box pose information of the object.

[0168] Based on the camera pose, the pose information of the 3D detection box of the object is optimized by BA to obtain an initial semantic map of the object.

[0169] Optionally, the semantic mapping module 13 is used for:

[0170] Determine the object category semantics of the object corresponding to the two-dimensional detection box information;

[0171] Based on the two-dimensional detection box information, an object semantic mapping is established between the three-dimensional map point cloud and the object category semantics to obtain an object association dataset. The object association dataset consists of at least one three-dimensional map point cloud, the associated mapping data between the object category semantics corresponding to the three-dimensional map point cloud and the two-dimensional detection box information.

[0172] Optionally, the semantic mapping module 13 is used for:

[0173] Based on the two-dimensional line detection results and the object association dataset, the isolated forest detection method is used to predict the translation matrix and object scale of the object;

[0174] Determine the consistency score between the two-dimensional line detection result and the two-dimensional detection box information in the object association dataset. Based on the consistency score, use the minimum consistency error constraint to determine the target two-dimensional detection box information. Use the two-dimensional line segment angle in the target two-dimensional detection box information to determine the yaw angle of the object.

[0175] The three-dimensional bounding box pose information of the object is obtained based on the translation matrix of the object, the object scale, and the yaw angle.

[0176] Optionally, the semantic mapping module 13 is used for:

[0177] Construct at least one frame of the first three-dimensional object point cloud in the camera coordinate system based on the RGB image and the Depth image;

[0178] Based on the point cloud of the first three-dimensional object in at least one frame and the camera pose of the visual SLAM, a point cloud coordinate system transformation is performed to obtain at least one point cloud of the second three-dimensional object in the world coordinate system.

[0179] A global scene point cloud is generated based on the at least one frame of the second three-dimensional object point cloud, and the global scene point cloud is filtered to obtain a dense point cloud map.

[0180] Optionally, the device 1 is used for:

[0181] The first three-dimensional object point cloud of at least one frame is used as a key frame for the visual SLAM, and the back-end optimization process and loop closure detection process for the visual SLAM are triggered based on the key frame.

[0182] In the backend optimization process, local BA optimization is performed on the map points and keyframes in the visual SLAM. In the loop closure detection process, global loop closure detection optimization is performed on the visual SLAM to obtain the target 3D map point cloud and the target camera pose.

[0183] The target 3D map point cloud and the target camera pose are used as the 3D map point cloud and the camera pose, and the steps of determining the 3D map point cloud and camera pose in the visual SLAM and / or the steps of constructing a global scene point cloud based on the RGB image, the Depth image and the camera pose of the visual SLAM are performed.

[0184] Optionally, the device 1 is used for:

[0185] Based on the RGB image and the Depth image, determine the three-channel RGB information, depth information, and feature point information;

[0186] Based on the three-channel RGB information, the depth information, and the feature point information, at least one frame of the first three-dimensional object point cloud in the camera coordinate system is constructed.

[0187] Optionally, the device 1 is used for:

[0188] Based on the camera pose of the visual SLAM, the point cloud pose of the at least one frame of the first three-dimensional object point cloud is adjusted to obtain at least one frame of the second three-dimensional object point cloud in the world coordinate system.

[0189] Optionally, the device 1 is used for:

[0190] The initial object semantic map is overlaid onto the dense point cloud map to obtain a dense object semantic map of the scene.

[0191] It should be noted that the dense object semantic map construction device provided in the above embodiments is only illustrated by the division of the above functional modules when executing the dense object semantic map construction method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the dense object semantic map construction device and the dense object semantic map construction method embodiments provided in the above embodiments belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.

[0192] The example numbers in this specification are for descriptive purposes only and do not represent the superiority or inferiority of the examples.

[0193] In the embodiments described in this specification, ...

[0194] This specification also provides a computer storage medium that can store multiple instructions adapted to be loaded and executed by a processor as described above. Figures 1 to 11 The dense object semantic map construction method described in the illustrated embodiment can be found in the following documentation for its specific execution process. Figures 1 to 11 The specific details of the illustrated embodiments will not be elaborated here.

[0195] This specification also provides a computer program product that stores at least one instruction, said at least one instruction being loaded and executed by the processor as described above. Figures 1 to 11 The dense object semantic map construction method described in the illustrated embodiment can be found in the following documentation for its specific execution process. Figures 1 to 11 The specific details of the illustrated embodiments will not be elaborated here.

[0196] Please refer to Figure 13 This is a structural block diagram of an electronic device provided in an embodiment of this specification. The electronic device in this specification may include one or more of the following components: a processor 110, a memory 120, an input device 130, an output device 140, and a bus 150. The processor 110, memory 120, input device 130, and output device 140 can be connected via the bus 150.

[0197] Processor 110 may include one or more processing cores. Processor 110 connects to various parts of the terminal using various interfaces and lines, and performs various functions and processes data of terminal 100 by running or executing instructions, programs, code sets, or instruction sets stored in memory 120, and by calling data stored in memory 120. Optionally, processor 110 may be implemented using at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). Processor 110 may integrate one or more of the following: central processing unit (CPU), graphics processing unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 110, but may be implemented separately using a communication chip.

[0198] The memory 120 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 120 may include non-transitory computer-readable storage medium. The memory 120 may be used to store instructions, programs, code, code sets, or instruction sets.

[0199] The input device 130 is used to receive input instructions or data, and includes, but is not limited to, a keyboard, mouse, camera, microphone, or touch device. The output device 140 is used to output instructions or data, and includes, but is not limited to, a display device and a speaker. In this embodiment, the input device 130 can be a temperature sensor to obtain the operating temperature of the terminal. The output device 140 can be a speaker to output audio signals.

[0200] In addition, those skilled in the art will understand that the structure of the terminal shown in the above figures does not constitute a limitation on the terminal. The terminal may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the terminal may also include radio frequency circuits, input units, sensors, audio circuits, wireless fidelity (WIFI) modules, power supplies, Bluetooth modules, etc., which will not be described in detail here.

[0201] In the embodiments of this specification, the executing entity for each step can be the terminal described above. Optionally, the executing entity for each step is the terminal's operating system. The operating system can be Android, iOS, or other operating systems; this specification does not limit this.

[0202] exist Figure 13 In the electronic device, the processor 110 can be used to call a program stored in the memory 120 and execute it to implement the dense object semantic map construction method as described in the various method embodiments of this specification.

[0203] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory, or random access memory, etc.

[0204] The above-disclosed embodiments are merely preferred embodiments of this specification and should not be construed as limiting the scope of this specification. Therefore, any equivalent variations made in accordance with the claims of this specification shall still fall within the scope of this specification.

Claims

1. A method for constructing a dense object semantic map, characterized in that, The method includes: Acquire RGBD image data of the scene, wherein the RGBD image data includes RGB images and depth images; Based on the RGB image, object target detection processing is performed to obtain two-dimensional detection box information and object category information; and based on the RGB image, visual SLAM processing is performed to obtain visual SLAM. The three-dimensional map point cloud and camera pose in the visual SLAM are determined. Line detection and segmentation processing is performed on the grayscale image corresponding to the RGB image to obtain two-dimensional line detection results. Object association processing is performed based on the two-dimensional detection box information, the object category information and the three-dimensional map point cloud to obtain an object association dataset. Object pose prediction processing is performed based on the two-dimensional line detection results and the object association dataset to obtain the three-dimensional detection box pose information of the object. BA optimization processing is performed on the three-dimensional detection box pose information of the object based on the camera pose to obtain an initial object semantic map. A global scene point cloud is constructed based on the RGB image, the Depth image, and the camera pose of the visual SLAM. The global scene point cloud is then filtered to obtain a dense point cloud map. Map fitting is performed on the initial object semantic map and the dense point cloud map to obtain a dense object semantic map of the scene.

2. The method according to claim 1, characterized in that, The object association processing based on the two-dimensional detection box information, the object category information, and the three-dimensional map point cloud yields an object association dataset, including: Determine the object category semantics of the object corresponding to the two-dimensional detection box information; Based on the two-dimensional detection box information, an object semantic mapping is established between the three-dimensional map point cloud and the object category semantics to obtain an object association dataset. The object association dataset consists of at least one three-dimensional map point cloud, the associated mapping data between the object category semantics corresponding to the three-dimensional map point cloud and the two-dimensional detection box information.

3. The method according to claim 1, characterized in that, The process of predicting the object's pose based on the two-dimensional line detection results and the object association dataset to obtain the object's three-dimensional bounding box pose information includes: Based on the two-dimensional line detection results and the object association dataset, the isolated forest detection method is used to predict the translation matrix and object scale of the object; Determine the consistency score between the two-dimensional line detection result and the two-dimensional detection box information in the object association dataset. Based on the consistency score, use the minimum consistency error constraint to determine the target two-dimensional detection box information. Use the two-dimensional line segment angle in the target two-dimensional detection box information to determine the yaw angle of the object. The three-dimensional bounding box pose information of the object is obtained based on the translation matrix of the object, the object scale, and the yaw angle.

4. The method according to claim 1, characterized in that, The process of constructing a global scene point cloud based on the camera pose of the RGB image, the Depth image, and the visual SLAM, and filtering the global scene point cloud to obtain a dense point cloud map includes: Construct at least one frame of the first three-dimensional object point cloud in the camera coordinate system based on the RGB image and the Depth image; Based on the point cloud of the first three-dimensional object in at least one frame and the camera pose of the visual SLAM, a point cloud coordinate system transformation is performed to obtain at least one point cloud of the second three-dimensional object in the world coordinate system. A global scene point cloud is generated based on the at least one frame of the second three-dimensional object point cloud, and the global scene point cloud is filtered to obtain a dense point cloud map.

5. The method according to claim 4, characterized in that, After constructing at least one frame of the first 3D object point cloud in the camera coordinate system based on the RGB image and the Depth image, the method further includes: The first three-dimensional object point cloud of at least one frame is used as a key frame for the visual SLAM, and the back-end optimization process and loop closure detection process for the visual SLAM are triggered based on the key frame. In the backend optimization process, local BA optimization is performed on the map points and keyframes in the visual SLAM. In the loop closure detection process, global loop closure detection optimization is performed on the visual SLAM to obtain the target 3D map point cloud and the target camera pose. The target 3D map point cloud and the target camera pose are used as the 3D map point cloud and the camera pose, and the steps of determining the 3D map point cloud and camera pose in the visual SLAM and / or the steps of constructing a global scene point cloud based on the RGB image, the Depth image and the camera pose of the visual SLAM are performed.

6. The method according to claim 4, characterized in that, The construction of at least one frame of the first 3D object point cloud in the camera coordinate system based on the RGB image and the Depth image includes: Based on the RGB image and the Depth image, determine the three-channel RGB information, depth information, and feature point information; Based on the three-channel RGB information, the depth information, and the feature point information, at least one frame of the first three-dimensional object point cloud in the camera coordinate system is constructed.

7. The method according to claim 4, characterized in that, The step of performing point cloud coordinate system transformation based on the at least one frame of the first 3D object point cloud and the camera pose of the visual SLAM to obtain at least one frame of the second 3D object point cloud in the world coordinate system includes: Based on the camera pose of the visual SLAM, the point cloud pose of the at least one frame of the first three-dimensional object point cloud is adjusted to obtain at least one frame of the second three-dimensional object point cloud in the world coordinate system.

8. The method according to claim 1, characterized in that, The step of performing map fitting on the initial object semantic map and the dense point cloud map to obtain a dense object semantic map of the scene includes: The initial object semantic map is overlaid onto the dense point cloud map to obtain a dense object semantic map of the scene.

9. A dense object semantic map construction device, characterized in that, The device includes: The image acquisition module is used to acquire RGBD image data of the scene, wherein the RGBD image data includes RGB images and depth images; The target detection module is used to perform object target detection processing based on the RGB image to obtain two-dimensional detection box information and object category information, and to perform visual SLAM processing based on the RGB image to obtain visual SLAM. The semantic mapping module is used to determine the 3D map point cloud and camera pose in the visual SLAM, perform line detection and segmentation processing on the grayscale image corresponding to the RGB image to obtain the 2D line detection result, perform object association processing based on the 2D detection box information, the object category information and the 3D map point cloud to obtain the object association dataset, perform object pose prediction processing based on the 2D line detection result and the object association dataset to obtain the 3D detection box pose information of the object, and perform BA optimization processing on the 3D detection box pose information of the object based on the camera pose to obtain the initial object semantic map. The semantic mapping module is used to construct a global scene point cloud based on the RGB image, the Depth image, and the camera pose of the visual SLAM, and to filter the global scene point cloud to obtain a dense point cloud map. The semantic mapping module is used to perform map fitting on the initial object semantic map and the dense point cloud map to obtain a dense object semantic map of the scene.

10. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions, which are adapted to be loaded by a processor and executed as method steps as claimed in any one of claims 1 to 8.

11. An electronic device, characterized in that, include: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and executed the method steps as claimed in any one of claims 1 to 8.

Images