Method, device and equipment for precise perception and positioning of a robot sweeper and medium

Abstract

The application provides a precise positioning method, device, equipment and medium for a sweeping robot. The sweeping robot rotates to obtain various scene image frames of the current position. Then, a plurality of semantic feature points of each scene image frame are obtained, and a semantic map of the sweeping robot at the current position is further constructed. Then, the spatial correlation between the semantic feature points in each scene image frame is determined, and the visual constraint condition of the sweeping robot when perceiving each similar indoor object around the current position is determined. Similarity matching is performed on adjacent scene image frames to obtain the scene coincidence degree between adjacent scene image frames, and then the visual perception parameter of the sweeping robot to the similar indoor object at the current position is determined. Further, visual repositioning is performed on the sweeping robot to obtain the positioning reliability of the sweeping robot at the current position. The scheme of the application can avoid positioning drift under the influence of similar scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of perceptual positioning, more specifically, the present application relates to a precise perceptual positioning method and device for a sweeping robot, equipment and medium. BACKGROUND

[0002] Perceptual positioning determines the position of a robot in space by acquiring environmental information, uses sensors carried by the robot to perceive the surrounding environment in real time, and combines sensor data for analysis and processing to achieve precise positioning of the robot. Perceptual positioning not only relies on sensor data, but also continuously updates and corrects to improve the accuracy of positioning. In particular, in complex or dynamic environments, obstacles can be effectively identified and avoided, and the best path can be planned. With the development of technology, perceptual positioning has been able to handle challenges in different scenarios, such as changes in lighting, changes in the environment, and repeated scenes. Through continuous optimization of algorithms and enhancement of perceptual capabilities, perceptual positioning technology has become an important foundation for robot autonomous navigation, providing reliable spatial awareness for robots to perform tasks.

[0003] In a home environment, many rooms may contain a large number of similar or repeated scene elements, such as symmetrical furniture layout, similar wall decoration and long corridors, etc. These features may cause the sweeping robot to misjudge when positioning. When the robot passes through these areas, the visual sensor may incorrectly identify similar scenes at different locations as the same place, resulting in feature matching errors. Without timely correction, positioning errors will gradually accumulate, causing positioning drift. As the task progresses, the robot may think it is still at a known location, but the actual location has shifted, resulting in inaccurate positioning of the sweeping robot. Therefore, how to avoid positioning drift under the influence of similar scenes has become a difficult problem to be solved in the industry. SUMMARY

[0004] The present application provides a precise perceptual positioning method, device, equipment and medium for a sweeping robot, which can avoid positioning drift under the influence of similar scenes.

[0005] In a first aspect, the present application provides a precise perceptual positioning method for a sweeping robot, comprising the following steps:

[0006] The sweeping robot rotates to obtain each scene image frame around the current position;

[0007] Each visual feature point in each scene image frame is semantically classified to obtain a plurality of semantic feature points for each scene image frame, and then a semantic map for positioning the sweeping robot at the current position is constructed according to all semantic feature points;

[0008] determine the spatial correlation between the semantic feature points in each scene image frame according to the feature descriptors of the semantic feature points in each scene image frame, and determine the visual constraint condition of the robotic cleaner when perceiving each similar indoor object around the current position according to the semantic map and all the spatial correlation;

[0009] perform similarity matching on adjacent scene image frames to obtain scene coincidence degrees between adjacent scene image frames, and determine the visual perception parameter of the robotic cleaner for the similar indoor object at the current position according to all the scene coincidence degrees;

[0010] perform visual repositioning on the robotic cleaner according to the visual perception parameter and the visual constraint condition of the robotic cleaner when perceiving each indoor object around the current position to obtain the position determination reliability of the robotic cleaner at the current position.

[0011] In some embodiments, the semantic classification of the visual feature points in each scene image frame includes:

[0012] determining the visual feature points in each scene image frame;

[0013] extracting the semantic labels of the visual feature points in each scene image frame from the visual feature points in each scene image frame based on a pre-trained semantic segmentation model;

[0014] performing semantic screening on the semantic labels of the visual feature points in each scene image frame to obtain the semantic feature points in each scene image frame.

[0015] In some embodiments, the construction of the semantic map of the robotic cleaner when positioning at the current position according to all the semantic feature points includes:

[0016] mapping the semantic feature points in each scene image frame to a reference coordinate system of the robotic cleaner at the current position to obtain the three-dimensional spatial coordinates of the semantic feature points in each scene image frame in the reference coordinate system;

[0017] aggregating the three-dimensional spatial coordinates of the semantic feature points in all the scene image frames to obtain different semantic entities in the reference coordinate system;

[0018] determining the semantic map of the robotic cleaner when positioning at the current position according to all the semantic entities in the reference coordinate system.

[0019] In some embodiments, the determination of the spatial correlation between the semantic feature points in each scene image frame according to the feature descriptors of the semantic feature points in each scene image frame includes:

[0020] For each scene image frame;

[0021] Determine feature descriptors of each semantic feature point in the scene image frame;

[0022] Determine feature correlation degrees between each semantic feature point in the scene image frame according to the feature descriptors of each semantic feature point in the scene image frame;

[0023] Obtain pixel coordinates of each semantic feature point in the scene image frame;

[0024] Determine spatial correlation degrees between each semantic feature point in the scene image frame according to the pixel coordinates of each semantic feature point in the scene image frame;

[0025] Determine spatial correlation relationships between each semantic feature point in the scene image frame through the feature correlation degrees and the spatial correlation degrees.

[0026] In some embodiments, the visual constraint condition of the sweeping robot perceiving each similar indoor object around the current position is determined according to the semantic map and all the spatial correlation relationships, and specifically includes:

[0027] Extract an interference influence coefficient of the sweeping robot perceiving each similar indoor object around the current position from the semantic map;

[0028] Determine a visual error of the sweeping robot perceiving each similar indoor object around the current position according to all the spatial correlation relationships;

[0029] Determine the visual constraint condition of the sweeping robot perceiving each similar indoor object around the current position through the interference influence coefficient and the visual error of the sweeping robot perceiving each similar indoor object around the current position.

[0030] In some embodiments, similarity matching is performed on adjacent scene image frames to obtain a scene coincidence degree between adjacent scene image frames, and specifically includes:

[0031] For each two adjacent scene image frames;

[0032] Determine each matched feature point pair between the two adjacent scene image frames;

[0033] Perform similarity evaluation on each matched feature point pair between the two adjacent scene image frames to obtain the scene coincidence degree between the adjacent scene image frames.

[0034] In some embodiments, the sweeping robot is visually repositioned according to the visual perception parameter and the visual constraint condition of the sweeping robot perceiving each indoor object around the current position to obtain a position determination confidence of the sweeping robot at the current position, and specifically includes:

[0035] relocating the position estimation of the robotic cleaner according to the visual perception parameter and the visual constraint condition of the robotic cleaner perceiving each indoor object around the robotic cleaner at the current position, to obtain a localization vector of the robotic cleaner at the current position;

[0036] relocating the position estimation of the robotic cleaner according to the visual perception parameter and the visual constraint condition of the robotic cleaner perceiving each indoor object around the robotic cleaner at the current position, to obtain a localization vector of the robotic cleaner at the current position;

[0037] determining a localization confidence of the robotic cleaner at the current position through the localization vector.

[0038] In a second aspect, the present application provides a precise perception and positioning device for a robotic cleaner, comprising:

[0039] an acquisition module, configured to acquire each scene image frame around the robotic cleaner in rotation;

[0040] a processing module, configured to perform semantic classification on each visual feature point in each scene image frame, to obtain a plurality of semantic feature points of each scene image frame, and further to construct a semantic map of the robotic cleaner at the current position according to all the semantic feature points;

[0041] The processing module is further configured to determine a spatial correlation relationship between each semantic feature point in each scene image frame according to a feature descriptor of each semantic feature point in the scene image frame, and to determine a visual constraint condition of the robotic cleaner perceiving each similar indoor object around the robotic cleaner at the current position according to the semantic map and all the spatial correlation relationships.

[0042] The processing module is further configured to perform similarity matching on adjacent scene image frames, to obtain a scene coincidence degree between adjacent scene image frames, and to determine a visual perception parameter of the robotic cleaner for similar indoor objects at the current position according to all the scene coincidence degrees.

[0043] an execution module, configured to perform visual relocation on the robotic cleaner according to the visual perception parameter and the visual constraint condition of the robotic cleaner perceiving each indoor object around the robotic cleaner at the current position, to obtain a localization confidence of the robotic cleaner at the current position.

[0044] In a third aspect, the present application provides a computer device, comprising a memory and a processor, the memory stores a code, and the processor is configured to acquire the code and execute the above-mentioned precise perception and positioning method for a robotic cleaner.

[0045] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above-mentioned precise perception and positioning method for a robotic cleaner.

[0046] The technical scheme provided by the embodiments disclosed in the application has the following beneficial effects:

[0047] In the method, device, equipment and medium for accurate positioning of a sweeping robot provided by the application, the sweeping robot rotates to obtain each scene image frame around the current position; each visual feature point in each scene image frame is subjected to semantic classification to obtain a plurality of semantic feature points of each scene image frame, and then a semantic map of the sweeping robot when positioning at the current position is constructed according to all the semantic feature points; the spatial correlation between each semantic feature point in each scene image frame is determined according to the feature descriptor of each semantic feature point, the visual constraint condition of the sweeping robot when perceiving each similar indoor object around the current position is determined according to the semantic map and all the spatial correlations; similarity matching is performed on adjacent scene image frames to obtain the scene coincidence degree between adjacent scene image frames, the visual perception parameter of the sweeping robot for similar indoor objects at the current position is determined according to all the scene coincidence degrees; the sweeping robot is subjected to visual repositioning according to the visual perception parameter and the visual constraint condition of the sweeping robot when perceiving each indoor object around the current position, and the positioning reliability of the sweeping robot at the current position is obtained.

[0048] Therefore, this application can perform visual relocalization of the sweeping robot based on the aforementioned visual perception parameters and the visual constraints when the sweeping robot perceives each indoor object in its surroundings, thereby obtaining the localization confidence of the sweeping robot at its current position. Specifically, firstly, semantic classification is performed on each visual feature point in each scene image frame, which can elevate traditional low-level features to semantically discriminative feature points, thus enhancing the sweeping robot's ability to recognize key objects in similar scenes. Secondly, constructing a semantic map of the sweeping robot's localization at its current position helps introduce higher-level semantic information into environmental perception, enabling the sweeping robot to identify and distinguish different categories. Compared to traditional maps based solely on geometry or texture, semantic maps, which depict indoor objects and structural features, possess stronger discriminative power. They effectively reduce the risk of mismatches in repetitive or similar indoor scenes, thereby enhancing the robot vacuum's environmental understanding and preventing positioning drift caused by scene similarity. Furthermore, by determining the spatial relationships between semantic feature points in each scene image frame, structural constraints can be established at the feature level, enhancing the spatial consistency between semantic information. This helps the robot vacuum distinguish positional differences in environments with similar visual information, avoiding incorrect matching due to local visual similarity and thus reducing the risk of positioning drift. One step involves combining the spatial relationships between semantic maps and semantic feature points to accurately identify the position and semantic features of each similar indoor object within the overall spatial structure. This visual constraint effectively limits matching ambiguities caused by scene similarity during the robot vacuum's perception process, enhancing the robot's ability to recognize subtle differences between different positions and enabling it to maintain stable and accurate positioning even when faced with repetitive or symmetrical layouts. Then, a comprehensive analysis of the overlap of all scenes effectively extracts features of the robot's perception stability in similar scenes, thereby determining visual perception parameters and enhancing the robot vacuum's ability to distinguish repetitive visual information. Finally, by... By combining the sensory parameters with the visual constraints of the robot vacuum cleaner when perceiving surrounding indoor objects, the initial position estimate can be corrected under similar scene interference, achieving more robust visual relocalization. This process can effectively identify perception errors caused by repetitive structures or similar objects in the environment, and dynamically compensate for the errors through the relocalization algorithm, thereby improving the accuracy and stability of localization. By evaluating the confidence of the localization results, the robot vacuum cleaner can determine whether the current localization is reliable. If the confidence is insufficient, further relocalization operations can be triggered to avoid continuous localization drift in similar scenes. In summary, the solution of this application can avoid localization drift under the influence of similar scenes. Attached Figure Description

[0049] Figure 1 This is an exemplary flowchart of a precise sensing and positioning method for a sweeping robot according to some embodiments of this application;

[0050] Figure 2 is a flowchart of determining semantic feature points according to some embodiments of the present application;

[0051] Figure 3 is a flowchart of determining spatial correlation between each semantic feature point according to some embodiments of the present application;

[0052] Figure 4 is a structural diagram of a precise perception and positioning device of a sweeping robot according to some embodiments of the present application;

[0053] Figure 5 is a structural diagram of a computer device for implementing a precise perception and positioning method of a sweeping robot according to some embodiments of the present application. DETAILED DESCRIPTION

[0054] In order to better understand the technical solutions of the present application, the technical solutions of the present application will be described in detail below in combination with the accompanying drawings and specific embodiments.

[0055] Reference Figure 1 The figure is an exemplary flowchart of a precise perception and positioning method of a sweeping robot according to some embodiments of the present application. The method of the present embodiment can be used for the sweeping robot to reposition after power-on in a home environment. The precise perception and positioning method 100 of the sweeping robot mainly includes the following steps:

[0056] In step 101, the sweeping robot rotates to obtain each scene image frame around the current position.

[0057] It should be noted that in the present application, the sweeping robot is powered on and starts, detects that the current position cannot be matched with the existing map, and triggers the "repositioning" mode.

[0058] In specific implementation, the sweeping robot controls the driving wheel to rotate around the vertical axis of itself at a slow speed at the current position. The rotation angle can be set to collect the scene image frame of the surrounding environment every fixed angle (such as 10° or 15°), so as to obtain each scene image frame around the current position.

[0059] In step 102, each visual feature point in each scene image frame is semantically classified to obtain a plurality of semantic feature points of each scene image frame, and then a semantic map of the sweeping robot at the current position is constructed according to all the semantic feature points.

[0060] In some embodiments, referring to Figure 2As shown, the figure is a flowchart of determining semantic feature points in some embodiments of the present application. In this embodiment, the semantic classification of each visual feature point in each scene image frame is obtained by the following steps:

[0061] First, in step 1021, determine each visual feature point in each scene image frame;

[0062] Second, in step 1022, based on the pre-trained semantic segmentation model, extract the semantic label of each visual feature point in each scene image frame from each visual feature point in each scene image frame;

[0063] Then, in step 1023, the semantic label of each visual feature point in each scene image frame is subjected to semantic screening to obtain a plurality of semantic feature points of each scene image frame.

[0064] In specific implementation, the determination of each visual feature point in each scene image frame can be realized by the following way, that is, for each scene image frame, the scene image frame is subjected to grayscale processing to improve the image contrast and edge definition, then a feature point detection algorithm (such as a scale-invariant feature transform algorithm) is used to detect the pixel points of texture, corner or edge structure from the grayscale scene image frame, and the obtained pixel points are all taken as the visual feature points of the scene image frame, and then the visual feature points in each scene image frame are obtained.

[0065] It should be noted that the visual feature point in the present application represents a pixel point with significant visual features (such as corner, edge, texture, etc.), and the visual feature point has rotation and scale invariance and can maintain stable recognition effect under different viewing angles and lighting conditions.

[0066] It should be noted that the semantic segmentation model in the present application is a kind of deep learning model (such as DeepLabv3+) for assigning each pixel in an image to a specific semantic category (such as "ground", "wall", "furniture", etc.). The deep learning model is based on a convolutional neural network, extracts multi-scale semantic features of an image through an encoder, and maps the features back to a pixel-level classification result with the same size as the input image through a decoder. The deep learning model has been trained on a large-scale indoor scene dataset (such as the ADE20K dataset), can accurately capture the boundaries and semantic structures of different indoor objects in an indoor scene, and realize high-precision semantic division of different regions in a complex indoor scene.

[0067] In a specific implementation, the semantic labels of each visual feature point in each scene image frame can be extracted from the visual feature points in the scene image frame based on a pre-trained semantic segmentation model. In this case, for each scene image frame, the pre-trained semantic segmentation model is used to perform pixel-level semantic segmentation on the visual feature points in the scene image frame to generate semantic categories of the visual feature points in the scene image frame, and the obtained semantic categories are all taken as semantic labels, thereby obtaining the semantic labels of the visual feature points in each scene image frame.

[0068] It should be noted that the semantic label in the present application refers to identification information (for example, "ground", "wall", "table", "door", etc.) describing the semantic category name of a visual feature point, and the semantic label reflects the semantic attribute of each visual feature point in a scene image frame.

[0069] In a specific implementation, the semantic labels of each visual feature point in each scene image frame can be extracted from the visual feature points in the scene image frame based on a pre-trained semantic segmentation model. In this case, for each scene image frame, the pre-trained semantic segmentation model is used to perform pixel-level semantic segmentation on the visual feature points in the scene image frame to generate semantic categories of the visual feature points in the scene image frame, and the obtained semantic categories are all taken as semantic labels, thereby obtaining the semantic labels of the visual feature points in each scene image frame.

[0070] It should be noted that the semantic feature point in the present application refers to a pixel point with a semantic label and a salient visual feature.

[0071] In some embodiments, the semantic map of the current position of the sweeping robot can be constructed according to all the semantic feature points by the following steps:

[0072] Each semantic feature point in each scene image frame is mapped to a reference coordinate system of the current position of the sweeping robot to obtain three-dimensional space coordinates of each semantic feature point in each scene image frame in the reference coordinate system.

[0073] The three-dimensional space coordinates of each semantic feature point in all scene image frames are aggregated to obtain different semantic entities in the reference coordinate system.

[0074] The semantic map of the current position of the sweeping robot is determined according to all semantic entities in the reference coordinate system.

[0075] In a specific implementation, the three-dimensional space coordinates of each semantic feature point in each scene image frame in the reference coordinate system are obtained by mapping each semantic feature point of each scene image frame to the reference coordinate system of the current position of the sweeping robot, which can be implemented in the following manner: the sweeping robot estimates a three-dimensional position of the current sweeping robot relative to the indoor environment by using a visual simultaneous localization and mapping (VSLAM) technology, and then converts each semantic feature point of each scene image frame into a three-dimensional space coordinate in the reference coordinate system of the sweeping robot by using the intrinsic and extrinsic parameters of a stereo camera carried by the sweeping robot (obtained by camera calibration) and the three-dimensional position, thereby obtaining the three-dimensional space coordinates of each semantic feature point in each scene image frame in the reference coordinate system.

[0076] In a specific implementation, the different semantic entities in the reference coordinate system are obtained by aggregating the three-dimensional space coordinates of each semantic feature point in all scene image frames, which can be implemented in the following manner: the three-dimensional space coordinates of each semantic feature point in all scene image frames are clustered by using a clustering algorithm (for example, an Euclidean clustering algorithm), thereby obtaining a plurality of data clusters, and then the semantic feature points from different scene image frames in each data cluster are spatially aligned by using a point cloud registration algorithm (for example, an ICP algorithm), thereby taking each data cluster obtained after spatial alignment as a semantic entity in the reference coordinate system, and further obtaining the different semantic entities in the reference coordinate system.

[0077] It should be noted that the semantic entity in the present application represents an indoor object having a semantic label and a plurality of three-dimensional space coordinates in the reference coordinate system of the sweeping robot.

[0078] It should be noted that the semantic entities are uniformly organized and structured into a map model capable of expressing spatial geometric relationships and high-level semantic relationships in the present application, thereby enabling the robot to have semantic cognitive ability and spatial positioning ability for the environment.

[0079] In a specific implementation, the semantic map of the sweeping robot at the current position can be determined according to all semantic entities in the reference coordinate system in the following manner. First, all semantic entities are searched by using a spatial index structure (for example, a KD-Tree), and the searched set of semantic entities is taken as the set of semantic entities in the reference coordinate system. Then, the set of semantic entities is grouped according to semantic labels, and each semantic entity is further classified by combining the geometric attributes (for example, spatial extension direction and size range) of the semantic entity, so as to eliminate redundant representations or overlapping entities caused by multiple frames of observation. On this basis, the classified semantic entities are registered in the map according to their spatial distribution by using a sparse point graph structure, and the position, category and adjacency relationship of each semantic entity in the map are recorded, so as to obtain an initial map. Finally, the position relationship between the semantic entities in the initial map is globally adjusted by using a graph optimization algorithm (for example, a G2O algorithm), and the initial map obtained after the global adjustment is taken as the semantic map of the sweeping robot at the current position.

[0080] It should be noted that the semantic map in the present application represents a map that integrates spatial information and semantic information of indoor objects.

[0081] In step 103, the spatial correlation relationship between the semantic feature points in each scene image frame is determined according to the feature descriptors of the semantic feature points in each scene image frame, and the visual constraint condition of the sweeping robot for perceiving each similar indoor object around the current position is determined according to the semantic map and all spatial correlation relationships.

[0082] In some embodiments, the reference Figure 3 As shown in the figure, which is a flowchart of determining the spatial correlation relationship between the semantic feature points in some embodiments of the present application, the spatial correlation relationship between the semantic feature points in each scene image frame can be determined according to the feature descriptors of the semantic feature points in each scene image frame in the following steps:

[0083] for each scene image frame;

[0084] determining the feature descriptors of the semantic feature points in the scene image frame;

[0085] determining the feature correlation degree between the semantic feature points in the scene image frame according to the feature descriptors of the semantic feature points in the scene image frame;

[0086] obtaining the pixel coordinates of the semantic feature points in the scene image frame;

[0087] determining the spatial correlation degree between the semantic feature points in the scene image frame according to the pixel coordinates of the semantic feature points in the scene image frame;

[0088] determine the spatial correlation relationship between each semantic feature point in the scene image frame according to the feature correlation and the spatial correlation.

[0089] In a specific implementation, the feature descriptor of each semantic feature point in the scene image frame can be determined in the following manner: for each semantic feature point in the scene image frame, a scale-invariant feature transform descriptor is used as the feature descriptor of the semantic feature point, thereby obtaining the feature descriptor of each semantic feature point in the scene image frame.

[0090] It should be noted that the feature descriptor in the present application refers to a vector that describes the local region feature of a semantic feature point.

[0091] In a specific implementation, the feature correlation between each semantic feature point in the scene image frame can be determined according to the feature descriptor of each semantic feature point in the scene image frame in the following manner: for the feature descriptors of each two semantic feature points in the scene image frame, the cosine similarity of the feature descriptors of each two semantic feature points is calculated, and the obtained cosine similarity is used as a correlation parameter between the two semantic feature points, and then the smallest correlation parameter is selected from all the correlation parameters as the feature correlation between each semantic feature point in the scene image frame, wherein the smallest correlation parameter usually means that the two semantic feature points are closest in the feature space, indicating that they have the highest visual similarity.

[0092] It should be noted that the feature correlation in the present application refers to the similarity degree of each semantic feature point in the visual information expression.

[0093] In a specific implementation, the spatial correlation between each semantic feature point in the scene image frame can be determined according to the pixel coordinates of each semantic feature point in the scene image frame in the following manner: for the pixel coordinates of each two semantic feature points in the scene image frame, the Euclidean distance of the pixel coordinates of each two semantic feature points is calculated, and then a Gaussian function is used as a weight function to assign a weight to the Euclidean distance between the pixel coordinates of each two semantic feature points, the semantic feature points with a closer distance have a higher weight, and the semantic feature points with a farther distance have a lower weight, for example, the weight function can be wherein d represents the Euclidean distance, and sigma is a pre-set parameter that can be adjusted according to experiments, and further weighted summation is performed on all the Euclidean distances, and the value obtained by the summation is used as the spatial correlation between each semantic feature point in the scene image frame.

[0094] It should be noted that the spatial correlation in the present application refers to the closeness of the distribution of each semantic feature point in the scene image frame.

[0095] In addition, it should be noted that the spatial correlation represents the feature parameters of the visual expression information formed by the spatial positions of the semantic feature points, and as a preferred embodiment, the spatial correlation between the semantic feature points in the scene image frame can be determined by the feature correlation and the spatial correlation in the following manner, that is, the quotient of the spatial correlation and the feature correlation is taken as the spatial correlation between the semantic feature points in the scene image frame, wherein the quotient of the feature correlation and the spatial correlation is taken as the spatial correlation, and in this way, the balance between visual similarity and spatial relationship can be emphasized, and the excessive influence of extreme values in one dimension on the final result can be reduced.

[0096] In some embodiments, determining the visual constraint condition of the sweeping robot in perceiving each similar indoor object around the current position according to the semantic map and all spatial correlations can be achieved in the following steps:

[0097] Extracting the interference influence coefficient of the sweeping robot in perceiving each similar indoor object around the current position from the semantic map;

[0098] Determining the visual error of the sweeping robot in perceiving each similar indoor object around the current position according to all spatial correlations;

[0099] Determining the visual constraint condition of the sweeping robot in perceiving each similar indoor object around the current position by the interference influence coefficient and the visual error of the sweeping robot in perceiving each similar indoor object around the current position.

[0100] In a specific implementation, the interference influence coefficient of the sweeping robot in perceiving each similar indoor object around the current position can be extracted from the semantic map in the following manner, that is:

[0101] It should be noted that in a real indoor environment, objects such as chairs, tables, cabinets, etc. can be highly similar in geometric appearance, and by quantifying the risk of similar indoor objects causing confusion in visual perception tasks, the positioning robustness and accuracy of the sweeping robot in similar scenes can be improved.

[0102] In a specific implementation, the interference influence coefficient of the sweeping robot perceiving each similar indoor object around the current position can be obtained by the following method: first, converting the semantic labels of all semantic entities in the semantic map into vectors by using a word embedding technology (for example, Word2Vec), and taking the obtained vectors as label vectors of the semantic entities; for each two semantic entities, calculating the cosine similarity between the label vectors of the two semantic entities; obtaining a similar indoor object threshold from the database of the sweeping robot; comparing all the cosine similarities with the similar indoor object threshold; if the cosine similarity is greater than or equal to the similar indoor object threshold, regarding the two semantic entities corresponding to the cosine similarity as similar indoor objects, thereby obtaining a plurality of similar indoor objects; then, for each similar indoor object, obtaining all three-dimensional space coordinates of the semantic entities corresponding to the similar indoor object from the semantic map, and then grouping all the three-dimensional space coordinates into a vector, and taking the obtained vector as a three-dimensional space coordinate vector of the similar indoor object; then, obtaining an orientation vector of the sweeping robot at the current position; then, calculating the cosine of the angle between the three-dimensional space coordinate vector and the orientation vector by using the cosine law; then, dividing the cosine of the angle by the distance between the sweeping robot and the similar indoor object, and taking the value obtained by the division as the interference influence coefficient of the sweeping robot perceiving the similar indoor object at the current position, thereby obtaining the interference influence coefficient of the sweeping robot perceiving each similar indoor object around the current position.

[0103] It should be noted that the interference influence coefficient in the present application represents the degree of interference of the similar indoor object on the visual perception of the sweeping robot.

[0104] In addition, it should be noted that the similar indoor object threshold in the present application can be obtained by vectorizing the names of all indoor objects, calculating the cosine similarity of all obtained vectors, clustering all the cosine similarities by using a clustering algorithm (for example, DBSCAN), obtaining a plurality of clustering clusters, selecting the smallest cosine similarity from each clustering cluster, and taking the average value of all the smallest cosine similarities as the similar indoor object threshold.

[0105] In a specific implementation, the visual error of the sweeping robot in perceiving each similar indoor object around the current position can be determined according to all the spatial correlations in the following manner: all scene image frames are obtained, for each similar indoor object, a scene image frame in which the similar indoor object appears is filtered from each scene image frame using an existing image recognition model (for example, a MobileNet model), then the spatial correlations between the semantic feature points in all the filtered scene image frames are differentially processed, all the values obtained after the differential processing are summed, and the summed value is taken as the visual error of the sweeping robot in perceiving the similar indoor object around the current position, and thus the visual error of the sweeping robot in perceiving each similar indoor object around the current position is obtained.

[0106] It should be noted that the visual error in this application represents the recognition error of the sweeping robot in visually perceiving the similar indoor object.

[0107] In addition, it should also be noted that the visual constraint condition in this application represents a parameter limited by similar visual information in the perception process of the sweeping robot for the similar indoor object.

[0108] In a specific implementation, the visual constraint condition of the sweeping robot in perceiving each similar indoor object around the current position can be determined by the interference influence coefficient and the visual error in the following manner: for each similar indoor object, the quotient of the visual error of the sweeping robot in perceiving the similar indoor object around the current position and the interference influence coefficient is taken as the visual constraint condition of the sweeping robot in perceiving the similar indoor object around the current position, and thus the visual constraint condition of the sweeping robot in perceiving each similar indoor object around the current position is obtained.

[0109] It should be noted that when the visual error is relatively large and the interference influence coefficient is small, the quotient value will be large, which indicates that although the interference of external objects is not large, the error of the robot's own visual system is large, resulting in a large uncertainty in perceiving the similar indoor object, and therefore a more stringent visual constraint condition is needed to guide the behavior of the robot, for example, more cautious positioning decisions, on the contrary, when the visual error is small and the interference influence coefficient is large, the quotient value is small, which indicates that although the interference of external objects is strong, the visual system of the robot is relatively accurate, to a certain extent, it can offset the influence of part of the interference, at this time the visual constraint condition is relatively loose, and when both are large or small, the quotient value will also reflect the comprehensive influence degree accordingly.

[0110] In step 104, similarity matching is performed on the adjacent scene image frames to obtain scene coincidence degrees between the adjacent scene image frames, and the visual perception parameters of the sweeping robot to similar indoor objects at the current position are determined according to all the scene coincidence degrees.

[0111] In some embodiments, the similarity matching on the adjacent scene image frames to obtain the scene coincidence degrees between the adjacent scene image frames can be implemented by the following steps:

[0112] for each two adjacent scene image frames;

[0113] determining each matched feature point pair between the two adjacent scene image frames;

[0114] performing similarity evaluation on each matched feature point pair between the two adjacent scene image frames to obtain the scene coincidence degrees between the adjacent scene image frames.

[0115] In a specific implementation, the determination of each matched feature point pair between the two adjacent scene image frames can be implemented by the following manner, that is, first, the visual feature points of the two adjacent scene image frames are obtained, then the fast nearest neighbor search algorithm is used to match the visual feature points of the two adjacent scene image frames, and all the similar points matched by each two scene image frames are taken as the matched feature point pair, so as to obtain each matched feature point pair between the two adjacent scene image frames.

[0116] It should be noted that the matched feature point pair in the present application represents a pixel point pair with similar local features between the two adjacent scene image frames, wherein the matched feature point pair is composed of two pixel points, and the two pixel points are in the two scene image frames.

[0117] In a specific implementation, the similarity evaluation on each matched feature point pair between the two adjacent scene image frames to obtain the scene coincidence degrees between the adjacent scene image frames can be implemented by the following manner, that is, for each matched feature point pair, one pixel point in the matched feature point pair is selected as a selected pixel point, the sum of all pixel values adjacent to the position of the selected pixel point in the scene image frame is subtracted from the sum of all pixel values adjacent to the position of the other pixel point in the scene image frame, and the value obtained by the subtraction is taken as the pixel similarity of the matched feature point pair, so as to obtain the pixel similarity of each matched feature point pair, and further, the sum of the pixel similarities of all the matched feature point pairs is taken as the scene coincidence degrees between the adjacent scene image frames.

[0118] It should be noted that the scene coincidence degrees in the present application represent the similarity of indoor scenes in the adjacent scene image frames.

[0119] In some embodiments, determining the visual perception parameter of the similar indoor object by the sweeping robot at the current position according to all the scene coincidence degrees can be implemented by the following steps:

[0120] Linearly fitting all the scene coincidence degrees to obtain a fitting curve of the scene coincidence degrees;

[0121] Determining the visual perception parameter of the similar indoor object by the sweeping robot at the current position according to the fitting curve of the scene coincidence degrees.

[0122] In a specific implementation, the linear fitting of all the scene coincidence degrees to obtain the fitting curve of the scene coincidence degrees can be implemented by the following manner, that is, using an existing linear fitting algorithm (for example, a least squares support vector machine algorithm) to linearly fit all the scene coincidence degrees, thereby obtaining the fitting curve of the scene coincidence degrees.

[0123] In a specific implementation, determining the visual perception parameter of the similar indoor object by the sweeping robot at the current position according to the fitting curve of the scene coincidence degrees can be implemented by the following manner, that is, first, calculating the slope of each position on the fitting curve, and taking the maximum slope as the visual perception parameter of the similar indoor object by the sweeping robot at the current position, wherein the greater the slope is, the faster the scene coincidence degree changes, and the more unstable the similar indoor object affects the visual perception.

[0124] It should be noted that the visual perception parameter in the present application represents a parameter relied on by the visual system of the sweeping robot to perceive the indoor environment position under the influence of the similar indoor object.

[0125] In step 105, the visual repositioning of the sweeping robot is performed according to the visual perception parameter and the visual constraint condition of the sweeping robot perceiving each indoor object around, to obtain the position determination confidence of the sweeping robot at the current position.

[0126] In some embodiments, the visual repositioning of the sweeping robot is performed according to the visual perception parameter and the visual constraint condition of the sweeping robot perceiving each indoor object around, to obtain the position determination confidence of the sweeping robot at the current position, which can be implemented by the following steps:

[0127] Initializing the position estimation of the sweeping robot;

[0128] Repositioning the position estimation of the sweeping robot according to the visual perception parameter and the visual constraint condition of the sweeping robot perceiving each indoor object around, to obtain a positioning vector of the sweeping robot at the current position;

[0129] Determining the position determination confidence of the sweeping robot at the current position through the positioning vector.

[0130] In a specific implementation, the position estimation of the robot can be achieved by using a visual simultaneous localization and mapping (VSLAM) technology to estimate a three-dimensional position of the robot at the current position, and taking the three-dimensional position as the position estimation of the robot.

[0131] It should be noted that the position estimation in the present application refers to a three-dimensional spatial coordinate estimated by the robot at the current indoor scene.

[0132] In a specific implementation, the repositioning of the position estimation of the robot according to the visual perception parameters and the visual constraint conditions of the robot when perceiving each indoor object around the robot can be achieved by the following method: first, the robot acquires environment perception data of the current position, wherein the environment perception data includes the relative position, the field of view angle and the relative distance of the robot when perceiving each indoor object around the robot, etc.; then, the position is repositioned based on a particle filter algorithm, wherein each particle contains state information such as the position estimation and orientation of the robot, and the visual perception parameters are set as the initial weight of each particle, and the visual constraint conditions are set as additional constraint conditions of the motion model in the particle filter algorithm to constrain the motion range and position update of the particle; through multiple resampling steps of the particle filter algorithm, the spatial position coordinates of the robot are recorded at each resampling step in turn; and finally, a vector composed of all the spatial position coordinates obtained after the resampling is completed is taken as the positioning vector of the robot at the current position.

[0133] It should be noted that the positioning vector in the present application refers to a vector composed of different spatial position coordinates of the robot at the current position, and the positioning vector can reflect the different spatial distribution of the robot at the current position.

[0134] In a specific implementation, the determination of the positioning confidence of the robot at the current position through the positioning vector can be achieved by the following method: acquiring the position estimation of the robot, calculating the distance between each spatial position coordinate in the positioning vector and the position estimation of the robot, and then normalizing all the distances, and further taking the mean value of all the values obtained after the normalization as the positioning confidence of the robot at the current position.

[0135] It should be noted that the positioning confidence refers to a measure of the credibility of the current spatial position estimation result of the robot.

[0136] In addition, it should be noted that the last space in the positioning vector of the sweeping robot can be taken as the positioning coordinate of the sweeping robot at the current position, and the positioning confidence is used to measure the reliability of the positioning coordinate. If the positioning confidence is lower than the preset positioning confidence threshold (which can be obtained from the positioning system of the sweeping robot), the sweeping robot continues to perform the repositioning process until the positioning confidence is greater than or equal to the preset positioning confidence threshold, so as to ensure the accuracy of the current spatial position.

[0137] In addition, another aspect of the present application, in some embodiments, the present application provides a precise perception positioning device for a sweeping robot, which is described as follows: Figure 4 The figure is a structural schematic diagram of a precise perception positioning device for a sweeping robot according to some embodiments of the present application, which includes an acquisition module 401, a processing module 402 and an execution module 403, which are described as follows:

[0138] The acquisition module 401 is mainly used to acquire each scene image frame around the current position of the sweeping robot.

[0139] The processing module 402 is used to perform semantic classification on each visual feature point in each scene image frame to obtain a plurality of semantic feature points of each scene image frame, and then construct a semantic map of the sweeping robot at the current position according to all the semantic feature points.

[0140] It should be noted that the processing module 402 is also used to determine the spatial correlation relationship between each semantic feature point in each scene image frame according to the feature descriptor of each semantic feature point in each scene image frame, and determine the visual constraint condition of the sweeping robot when perceiving each similar indoor object around the current position according to the semantic map and all the spatial correlation relationships.

[0141] In addition, the processing module 402 is also used to perform similarity matching on adjacent scene image frames to obtain the scene coincidence degree between adjacent scene image frames, and determine the visual perception parameter of the sweeping robot for similar indoor objects at the current position according to all the scene coincidence degrees.

[0142] The execution module 403 is mainly used to perform visual repositioning on the sweeping robot according to the visual perception parameter and the visual constraint condition of the sweeping robot when perceiving each indoor object around the current position to obtain the positioning confidence of the sweeping robot at the current position.

[0143] In addition, the application further provides a computer device, comprising a memory and a processor, the memory stores code, and the processor is configured to acquire the code and execute the above-mentioned accurate positioning method of the sweeping robot.

[0144] In some embodiments, with reference to Figure 5 The figure is a structural schematic diagram of a computer device for implementing the accurate positioning method of the sweeping robot according to some embodiments of the application. The accurate positioning method of the sweeping robot in the above-mentioned embodiments can be implemented by the computer device shown in the figure, which comprises at least one processor 501, a communication bus 502, a memory 503 and at least one communication interface 504. Figure 5

[0145] The processor 501 can be a general central processing unit (CPU), an application-specific integrated circuit (ASIC) or one or more circuits for controlling the execution of the accurate positioning method of the sweeping robot in the application.

[0146] The communication bus 502 can be used to transmit information between the above-mentioned components.

[0147] The memory 503 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, an optical disk storage (including a compact disk, a laser disk, an optical disk, a digital versatile disk, a Blu-ray disk, etc.), a magnetic disk or other magnetic storage device, or any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer, but is not limited to this. The memory 503 can exist independently and be connected to the processor 501 through the communication bus 502. The memory 503 can also be integrated with the processor 501.

[0148] ​The memory 503 is configured to store a program code for implementing the scheme of the present application, and the processor 501 is configured to execute the program code stored in the memory 503. The program code can include one or more software modules. The methods described in the above method embodiments can be implemented by the processor 501 and one or more software modules in the program code in the memory 503.

[0149] The communication interface 504 is configured to communicate with other devices or communication networks, such as an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc., using any transceiver-like device.

[0150] In a specific implementation, as an embodiment, the computer device can include a plurality of processors, each of which can be a single-CPU processor or a multi-CPU processor. The processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0151] The computer device described above can be a general-purpose computer device or a special-purpose computer device. In a specific implementation, the computer device can be a desktop computer, a laptop computer, a network server, a personal digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. The embodiments of the present application do not limit the type of computer device.

[0152] In addition, the present application also provides a computer-readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the above-mentioned method for precise perception and positioning of a sweeping robot.

[0153] Although the preferred embodiments of the present application have been described, those skilled in the art can make further changes and modifications to the embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including all changes and modifications falling within the scope of the present application.

[0154] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these modifications and variations.

Claims

1. A method for accurate perception positioning of a robot vacuum cleaner, characterized in that, Includes the following steps: The robot vacuum cleaner rotates to acquire various scene image frames around its current location; Semantic classification is performed on each visual feature point in each scene image frame to obtain multiple semantic feature points for each scene image frame. Then, a semantic map is constructed based on all the semantic feature points when the robot vacuum cleaner is locating at the current position. Based on the feature descriptors of each semantic feature point in each scene image frame, the spatial association between each semantic feature point in each scene image frame is determined. Based on the semantic map and all spatial associations, the visual constraints when the sweeping robot perceives each similar indoor object in the current position are determined. Similarity matching is performed on adjacent scene image frames to obtain the scene overlap between adjacent scene image frames. Based on all scene overlap, the visual perception parameters of the robot vacuum cleaner for similar indoor objects at the current position are determined. Based on the visual perception parameters and the visual constraints when the robot vacuum perceives each indoor object in its surroundings, the robot vacuum is visually relocalized to obtain the localization confidence of the robot vacuum at its current position. Specifically, determining the spatial association between semantic feature points in each scene image frame based on the feature descriptors of each semantic feature point in each scene image frame includes: For each scene image frame; Determine the feature descriptors of each semantic feature point in the scene image frame; The feature correlation degree between semantic feature points in the scene image frame is determined based on the feature descriptors of each semantic feature point in the scene image frame; Obtain the pixel coordinates of each semantic feature point within a scene image frame; The spatial correlation between semantic feature points in a scene image frame is determined based on the pixel coordinates of each semantic feature point within the scene image frame. The spatial correlation between semantic feature points in a scene image frame is determined by the feature correlation degree and the spatial correlation degree.

2. The method of claim 1, wherein, Semantic classification is performed on each visual feature point in each scene image frame to obtain multiple semantic feature points for each scene image frame, specifically including: Identify each visual feature point in each scene image frame; The semantic segmentation model based on pre-trained semantic segmentation extracts the semantic labels of each visual feature point in each scene image frame. Semantic filtering is performed on the semantic labels of each visual feature point in each scene image frame to obtain multiple semantic feature points for each scene image frame.

3. The method of claim 1, wherein, Constructing a semantic map based on all semantic feature points for the robotic vacuum cleaner to locate its current position specifically includes: Each semantic feature point in each scene image frame is mapped to the reference coordinate system of the robot vacuum cleaner at its current position, thus obtaining the three-dimensional spatial coordinates of each semantic feature point in each scene image frame under the reference coordinate system. Aggregate the three-dimensional spatial coordinates of each semantic feature point within all scene image frames to obtain different semantic entities in the reference coordinate system; The semantic map for the robot vacuum cleaner to locate its current position is determined based on all semantic entities in the reference coordinate system.

4. The method of claim 1, wherein, The visual constraint condition of the sweeping robot perceiving each similar indoor object around the current position is determined according to the semantic map and all spatial correlation relationships, and specifically includes: An interference influence coefficient of the sweeping robot perceiving each similar indoor object around the current position is extracted from the semantic map; A visual error of the sweeping robot perceiving each similar indoor object around the current position is determined according to all spatial correlation relationships; The visual constraint condition of the sweeping robot perceiving each similar indoor object around the current position is determined by the interference influence coefficient and the visual error of the sweeping robot perceiving each similar indoor object around the current position.

5. The method of claim 1, wherein, The similarity matching of adjacent scene image frames is performed to obtain the scene coincidence degree between adjacent scene image frames, and specifically includes: For each two adjacent scene image frames; Each matching feature point pair between the adjacent two scene image frames is determined; The similarity of each matching feature point pair between the adjacent two scene image frames is evaluated to obtain the scene coincidence degree between adjacent scene image frames.

6. The method of claim 1, wherein, The sweeping robot is visually repositioned according to the visual perception parameter and the visual constraint condition of the sweeping robot perceiving each indoor object around the current position to obtain the positioning confidence of the sweeping robot at the current position, and specifically includes: The position estimation of the sweeping robot is initialized; The position estimation of the sweeping robot is repositioned according to the visual perception parameter and the visual constraint condition of the sweeping robot perceiving each indoor object around the current position to obtain the positioning vector of the sweeping robot at the current position; The positioning confidence of the sweeping robot at the current position is determined by the positioning vector.

7. A device for accurate positioning of a robot vacuum cleaner, the device being configured to perform the method according to any one of claims 1 to 6 for accurate positioning of a robot vacuum cleaner. The device specifically includes: An acquisition module is configured to acquire each scene image frame around the current position by rotating the sweeping robot; A processing module is configured to perform semantic classification on each visual feature point in each scene image frame to obtain a plurality of semantic feature points of each scene image frame, and further construct a semantic map of the sweeping robot at the current position according to all semantic feature points; The processing module is further configured to determine spatial correlation relationships between each semantic feature point in each scene image frame according to feature descriptors of the semantic feature points, and determine the visual constraint condition of the sweeping robot perceiving each similar indoor object around the current position according to the semantic map and all spatial correlation relationships; The processing module is further configured to perform similarity matching on adjacent scene image frames to obtain the scene coincidence degree between adjacent scene image frames, and determine the visual perception parameter of the sweeping robot for similar indoor objects at the current position according to all scene coincidence degrees; An execution module is configured to visually reposition the sweeping robot according to the visual perception parameter and the visual constraint condition of the sweeping robot perceiving each indoor object around the current position to obtain the positioning confidence of the sweeping robot at the current position.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the sweeping robot accurate perception positioning method in any one of claims 1 to 6.

9. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by a processor to implement the precise perception positioning method of the floor cleaning robot as claimed in any one of claims 1 to 6.

Images