Dynamic environment-based robot vacuum positioning method, device, apparatus, and medium

Abstract

The application provides a dynamic environment-based robot positioning method, device, equipment and medium. A positioning system of a robot obtains a three-dimensional map point cloud and an environment image through the robot, then performs semantic segmentation on the three-dimensional map point cloud according to semantic information contained in the environment image to obtain a moving object point cloud. The positioning system determines whether an object corresponding to the moving object point cloud is a moving object. If the object corresponding to the moving object point cloud is a moving object, the object motion speed and object pose of the moving object are analyzed according to the position of the moving object in the environment image of a continuous frame. The positioning system uses the motion speed and object pose to perform re-projection matching in the environment image of the continuous frame to obtain the robot pose of the robot. In the application, the positioning system fully utilizes the information contained in the moving object to position the robot, thereby effectively improving the information utilization rate and the positioning accuracy of the robot.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of floor sweeping machines, and particularly relates to a floor sweeping machine positioning method and device based on a dynamic environment, equipment and a medium. BACKGROUND

[0002] Currently, floor sweeping machines based on laser or visual positioning in the market all regard dynamic objects as interference when dealing with dynamic environments, and try to identify and delete dynamic objects during positioning and mapping to reduce the negative impact of dynamic objects on positioning and mapping. However, because the moving objects also contain environmental information, deleting these objects will also cause information loss, and in a dynamic object environment, such processing is easy to cause the algorithm to fail. SUMMARY

[0003] The main purpose of the present application is to provide a floor sweeping machine positioning method and device based on a dynamic environment, and equipment and a medium, which aims to solve the problem that the existing floor sweeping robot ignores dynamic object information during positioning, resulting in loss of environmental information and failure of the positioning algorithm.

[0004] To achieve the above purpose, the present application provides a floor sweeping machine positioning method based on a dynamic environment, comprising:

[0005] obtaining a three-dimensional map point cloud and an environment image;

[0006] performing semantic segmentation on the three-dimensional map point cloud according to semantic information contained in the environment image to obtain a moving object point cloud;

[0007] judging whether the object corresponding to the moving object point cloud is a moving object;

[0008] if the object corresponding to the moving object point cloud is a moving object, then analyzing the position of the moving object in the environment image of the continuous frame to obtain the object motion speed and object pose of the moving object;

[0009] using the motion speed and the object pose to perform re-projection matching in the environment image of the continuous frame to obtain the floor sweeping machine pose of the floor sweeping machine.

[0010] The present application also provides a floor sweeping machine positioning device based on a dynamic environment, comprising:

[0011] an acquisition module for acquiring a three-dimensional map point cloud and an environment image;

[0012] a segmentation module for performing semantic segmentation on the three-dimensional map point cloud according to semantic information contained in the environment image to obtain a moving object point cloud;

[0013] a judgment module for judging whether the object corresponding to the moving object point cloud is a moving object;

[0014] the parsing module is configured to parse the object motion speed and the object pose of the moving object according to the position of the moving object in the environment images of the continuous frames if the object corresponding to the moving object point cloud is a moving object;

[0015] the positioning module is configured to perform re-projection matching in the environment images of the continuous frames using the motion speed and the object pose, and solve the robot pose of the robot.

[0016] The application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method according to any one of the preceding embodiments when executing the computer program.

[0017] The application also provides a computer readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the steps of the method according to any one of the preceding embodiments.

[0018] The application provides a robot positioning method and device based on a dynamic environment, a positioning system of a robot, which obtains a three-dimensional map point cloud and environment images through the robot, performs semantic segmentation on the three-dimensional map point cloud according to semantic information contained in the environment images, and obtains a moving object point cloud. The positioning system determines whether the object corresponding to the moving object point cloud is a moving object, and if the object corresponding to the moving object point cloud is a moving object, parses the object motion speed and the object pose of the moving object according to the position of the moving object in the environment images of the continuous frames. The positioning system performs re-projection matching in the environment images of the continuous frames using the motion speed and the object pose, and solves the robot pose of the robot. In the application, the positioning system fully utilizes information contained in the moving object to position the robot, thereby effectively improving information utilization and positioning accuracy of the robot. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a step schematic diagram of the robot positioning method based on a dynamic environment in an embodiment of the application;

[0020] Figure 2 is a whole structure block diagram of the robot positioning device based on a dynamic environment in an embodiment of the application;

[0021] Figure 3 is a structure schematic block diagram of the computer device in an embodiment of the application.

[0022] The implementation, functional features and advantages of the application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0023] In order to make the purposes, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0024] With reference to Figure 1 An embodiment of the present application provides a robot positioning method based on a dynamic environment, comprising:

[0025] S1: acquiring a three-dimensional map point cloud and an environment image;

[0026] Specifically, a three-dimensional sensor and a vision sensor are deployed on the robot, and during movement or mapping, a positioning system (hereinafter referred to as a positioning system) of the robot continuously acquires three-dimensional point cloud data of the surrounding environment through the three-dimensional sensor, and splices the three-dimensional point cloud data to obtain a three-dimensional map point cloud; at the same time, a two-dimensional image of the surrounding environment is acquired through the vision sensor (such as a camera) to obtain an environment image of the surrounding environment.

[0027] S2: performing semantic segmentation on the three-dimensional map point cloud according to semantic information contained in the environment image to obtain a mobile object point cloud;

[0028] Specifically, the positioning system first extracts features from the environment image to obtain object feature points corresponding to each object (such as feature points corresponding to a table and feature points corresponding to a pet dog) contained in the environment image, and then performs object type recognition on the object feature points corresponding to each object through a pre-trained object recognition model, so as to obtain a first object type corresponding to each object, and screen out mobile object feature points corresponding to one or more mobile objects (such as a person and a pet dog) having a movable attribute. The positioning system takes the identified mobile object as semantic information, clusters and segments the three-dimensional point cloud corresponding to the mobile object feature points in the three-dimensional map point cloud according to the three-dimensional information corresponding to the mobile object feature points, to obtain a mobile object point cloud corresponding to the mobile object in the three-dimensional map point cloud.

[0029] S3: judging whether the object corresponding to the mobile object point cloud is a mobile object;

[0030] Specifically, the positioning system is pre-constructed with an object shape template, which is trained with each object and the corresponding shape as training data; in application, the positioning system compares the moving object point cloud with the object shape template, so as to obtain the second object type corresponding to the moving object point cloud, and then according to the second object type, identify whether the object corresponding to the moving object point cloud has a movable attribute (that is, on the basis of object type identification of object feature points, the object type identification of point cloud data is performed again to ensure that the movable attribute of the object is identified multiple times to improve the identification accuracy). If the object corresponding to the moving object point cloud has a movable attribute (such as a cat or a dog, whether the object has a movable attribute is marked by a person in the training stage of the template), it is determined that the object corresponding to the moving object point cloud is a moving object.

[0031] S4: If the object corresponding to the moving object point cloud is a moving object, the object motion speed and the object pose of the moving object are analyzed according to the position of the moving object in the continuous frames of the environment image.

[0032] Specifically, the positioning system determines whether the moving object is in a motion state, and if the moving object is in a motion state, the motion speed of the sweeper when moving at a constant speed is called, and the object motion speed and the object pose of the moving object are calculated according to the motion speed of the sweeper and the position change of the moving object in the continuous frames (more than two frames, i.e. at least three frames) of the environment image.

[0033] S5: Using the motion speed and the object pose, re-projection matching is performed in the continuous frames of the environment image to solve the sweeper pose of the sweeper.

[0034] Specifically, the positioning system uses the motion speed and the object pose of the moving object to perform re-projection matching in the continuous frames of the environment image, so as to solve the sweeper pose of the sweeper in the current environment scene.

[0035] In this embodiment, the positioning system makes full use of the information contained in the moving object to position the sweeper, thereby effectively improving the information utilization rate and the positioning accuracy of the sweeper.

[0036] Further, the step of performing semantic segmentation on the three-dimensional map point cloud according to the semantic information contained in the environment image to obtain the moving object point cloud comprises:

[0037] S201: performing feature extraction on the environment image to obtain object feature points corresponding to each object contained in the environment image;

[0038] Specifically, the positioning system performs feature extraction on the environment image to obtain a series of feature points, and then screens out object feature points corresponding to each object included in the environment image from the feature points.

[0039] S202: performing object type identification on each of the object feature points to screen out mobile object feature points corresponding to a mobile object;

[0040] Specifically, the positioning system performs object type identification on each of the object feature points corresponding to each object by using a pre-trained object recognition model, thereby obtaining the object type of each object, and then screens out a mobile object having a movable attribute according to the object type of each object and a pre-labeled object attribute (the object recognition model uses the feature points of each object as training data and the object attribute of each object is labeled by a human being during training), thereby obtaining mobile object feature points corresponding to the mobile object.

[0041] S203: clustering and segmenting three-dimensional points corresponding to the mobile object feature points in the three-dimensional map point cloud according to three-dimensional information corresponding to the mobile object feature points, to obtain a mobile object point cloud.

[0042] Specifically, the three-dimensional map point cloud is composed of a series of three-dimensional points (x, y, z), where z represents the distance from the point to the three-dimensional sensor (equivalent to the sweeping robot), i.e., the depth. The feature points are obtained by feature extraction of the image, and can be matched with the depth (in theory, each point on the image corresponds to a real point in the three-dimensional world). Accordingly, the sweeping robot associates the mobile object feature points with the corresponding three-dimensional points in the three-dimensional map point cloud, and then clusters and segments the three-dimensional points corresponding to the mobile object feature points, to obtain a mobile object point cloud corresponding to the mobile object.

[0043] Further, the step of clustering and segmenting the three-dimensional points corresponding to the mobile object feature points in the three-dimensional map point cloud according to the three-dimensional information corresponding to the mobile object feature points to obtain the mobile object point cloud comprises:

[0044] S2031: obtaining an object shape corresponding to the mobile object feature points according to the object type of the mobile object feature points, the object shape comprising volume information;

[0045] Specifically, the positioning system obtains an object shape corresponding to the mobile object feature points according to the object type of the mobile object feature points, wherein the object shape corresponding to the object type has been pre-set, and the object shape comprises the volume of the object. For example, the object type is a pet dog, and the object shape of the pet dog comprises the contour of the pet dog and the volume data composed of length, width and height.

[0046] S2032: Cluster and segment the three-dimensional points corresponding to the mobile object feature points according to the object shape, to obtain the mobile object point cloud.

[0047] Specifically, the positioning system clusters the three-dimensional points corresponding to the mobile object feature points according to the object shape, and segments all the clustered three-dimensional points to obtain the mobile object point cloud corresponding to the mobile object.

[0048] Further, the sweeping machine moves at a uniform speed, and the step of analyzing the object motion speed and the object pose of the mobile object according to the position of the mobile object in the environment images of consecutive frames comprises:

[0049] S401: Determine whether the mobile object is in a motion state;

[0050] Specifically, the positioning system determines whether the mobile object is in a motion state by identifying whether the position of the mobile object in the environment images of consecutive frames changes.

[0051] S402: If the mobile object is in a motion state, retrieve the sweeping machine motion speed of the sweeping machine, and calculate the object motion speed and the object pose of the mobile object according to the sweeping machine motion speed and the position change of the mobile object in the environment images of consecutive frames.

[0052] Specifically, if the position of the mobile object in the environment images of consecutive frames changes, it is determined that the mobile object is in a motion state. The positioning system retrieves the sweeping machine motion speed when the sweeping machine moves at a uniform speed, and analyzes the position change of the sweeping machine by the sweeping machine motion speed and the time length of the collection of adjacent two environment images. Then, according to the relative position change of the mobile object in the adjacent two environment images, combined with the position change of the sweeping machine, the absolute position change of the mobile object is analyzed, and then the object motion speed of the mobile object is calculated according to the absolute position change of the mobile object and the corresponding time length. Moreover, according to the pose of the mobile object in the environment images of consecutive frames, the pose of the mobile object relative to the sweeping machine (the mobile object is photographed by the vision sensor on the sweeping machine) can be analyzed, and then the object pose of the mobile object in the entire scene map can be obtained through coordinate system conversion.

[0053] Further, the step of determining whether the mobile object is in a motion state comprises:

[0054] S4011: Obtain the object image position of the mobile object in the environment images of consecutive frames, the object image position being the position coordinates of the mobile object in a world coordinate system, and the world coordinate system being a coordinate system established with the scene where the sweeping machine is located as a reference;

[0055] Specifically, the positioning system obtains object image positions of the mobile object in the environment images of the continuous frames, taking the position of an object in the environment images having an immovable attribute (such as a wall or an object such as a table or a chair that does not have the ability to move by itself) as a reference, that is, taking the scene where the sweeping machine is located as a reference to establish a world coordinate system, and the object image position is a position coordinate of the mobile object in the world coordinate system rather than a position coordinate in the sweeping machine coordinate system.

[0056] S4012: determining whether the object image positions are the same;

[0057] Specifically, the positioning system determines whether the object image positions of the mobile object in the environment images of different frames change according to the object image positions obtained above.

[0058] S4013: if the object image positions are different, determining that the mobile object is in a motion state.

[0059] Specifically, if the object image positions of the mobile object in the environment images of different frames change (that is, the object image positions are different), it indicates that the mobile object has a displacement, and the positioning system determines that the mobile object is currently in a motion state.

[0060] Further, the sweeping machine is provided with a three-dimensional sensor, and the step of obtaining the three-dimensional map point cloud comprises:

[0061] S101: continuously acquiring three-dimensional point cloud data through the three-dimensional sensor during the motion of the sweeping machine;

[0062] Specifically, the sweeping machine is provided with a three-dimensional sensor such as a 3D-tof sensor or a 3D structured light sensor; during the motion of the sweeping machine, the positioning system continuously acquires three-dimensional point cloud data of the surrounding environment through the three-dimensional sensor.

[0063] S102: splicing the three-dimensional point cloud data to obtain the three-dimensional map point cloud.

[0064] Specifically, the positioning system splices the three-dimensional point cloud data using a matching technology such as PNP (perspective-n-point) to obtain the three-dimensional map point cloud.

[0065] Further, the step of using the motion speed and the object pose to perform re-projection matching in the environment images of the continuous frames to obtain the sweeping machine pose of the sweeping machine comprises:

[0066] S501: obtaining a position change of the mobile object in the environment images of the continuous frames based on the object poses of the mobile object at continuous time points, wherein the continuous time points correspond to the shooting time points of the environment images of the continuous frames.

[0067] Specifically, the positioning system screens the respective object poses of the moving object at continuous time points (i.e., different photographing time points) based on photographing time points of the environment images of the continuous frames, and further obtains the position change of the moving object in the environment images of the continuous frames according to the changes of the respective object poses at the continuous time points.

[0068] S502: obtaining an initial pixel difference according to the position change of the moving object in the environment images of the continuous frames, and obtaining a pixel displacement of the moving object in the environment images of the continuous frames according to the motion speed analysis;

[0069] Specifically, the positioning system first obtains an initial pixel difference according to the position change of the moving object in the environment images of the continuous frames (the initial pixel difference represents the change of the pixel coordinates of the moving object in different frames of environment images caused by the motion of the moving object and the motion of the sweeper), and obtains a pixel displacement of the moving object in the environment images of the continuous frames according to the motion speed analysis (the pixel displacement represents the change of the pixel coordinates caused by the motion of the moving object itself).

[0070] S503: obtaining an absolute pixel difference by subtracting the pixel displacement from the initial pixel difference;

[0071] Specifically, the positioning system obtains an absolute pixel difference by subtracting the pixel displacement from the initial pixel difference, which is the pixel position change caused by the motion of the sweeper and obtained by analyzing the moving object as a fixed object.

[0072] S504: obtaining the sweeper pose by bundle adjustment method according to the absolute pixel difference.

[0073] Specifically, the positioning system obtains the sweeper pose in the scene map by bundle adjustment method, 5-point method or 8-point method, etc., to realize the positioning of the sweeper.

[0074] Referring to Figure 2 , an embodiment of the present application further provides a sweeper positioning device based on dynamic environment, comprising:

[0075] The acquisition module 1 is configured to acquire a three-dimensional map point cloud and an environment image.

[0076] The segmentation module 2 is configured to perform semantic segmentation on the three-dimensional map point cloud according to semantic information contained in the environment image to obtain a moving object point cloud.

[0077] The judgment module 3 is configured to judge whether the object corresponding to the moving object point cloud is a moving object.

[0078] The analysis module 4 is configured to, if the object corresponding to the mobile object point cloud is a mobile object, analyze the object motion speed and the object pose of the mobile object according to the position of the mobile object in the environment images of the continuous frames.

[0079] The positioning module 5 is configured to use the motion speed and the object pose to perform re-projection matching in the environment images of the continuous frames, and solve the robot pose of the robot.

[0080] Further, the segmentation module 2 comprises:

[0081] The extraction unit is configured to perform feature extraction on the environment images to obtain object feature points corresponding to each object contained in the environment images.

[0082] The screening unit is configured to perform object type identification on each object feature point to screen mobile object feature points corresponding to mobile objects.

[0083] The segmentation unit is configured to cluster and segment three-dimensional points corresponding to the mobile object feature points in the three-dimensional map point cloud according to three-dimensional information corresponding to the mobile object feature points, to obtain the mobile object point cloud.

[0084] Further, the segmentation unit comprises:

[0085] The first acquisition subunit is configured to acquire an object shape corresponding to the mobile object feature points according to the object type of the mobile object feature points, the object shape comprising volume information.

[0086] The segmentation subunit is configured to cluster and segment the three-dimensional points corresponding to the mobile object feature points according to the object shape, to obtain the mobile object point cloud.

[0087] Further, the robot moves at a constant speed, and the analysis module 4 comprises:

[0088] The judgment unit is configured to judge whether the mobile object is in a motion state.

[0089] The first calculation unit is configured to, if the mobile object is in the motion state, acquire a robot motion speed of the robot, and calculate the object motion speed and the object pose of the mobile object according to the robot motion speed and a position change of the mobile object in the environment images of the continuous frames.

[0090] Further, the judgment unit comprises:

[0091] A second acquisition subunit is configured to acquire object image positions of the moving object in the environment images of the continuous frames, the object image positions being position coordinates of the moving object in a world coordinate system, and the world coordinate system being a coordinate system established based on a scene in which the sweeping machine is located;

[0092] A judgment subunit is configured to judge whether the object image positions are the same.

[0093] A determination subunit is configured to determine that the moving object is in a motion state if the object image positions are different.

[0094] Further, the sweeping machine is provided with a three-dimensional sensor, and the acquisition module 1 comprises:

[0095] An acquisition unit is configured to continuously acquire three-dimensional point cloud data through the three-dimensional sensor during the motion of the sweeping machine.

[0096] A splicing unit is configured to splice the three-dimensional point cloud data to obtain the three-dimensional map point cloud.

[0097] Further, the positioning module 5 comprises:

[0098] A first analysis unit is configured to obtain a position change of the moving object in the environment images of the continuous frames based on the object poses of the moving object at continuous time points, wherein the continuous time points correspond to shooting time points of the environment images of the continuous frames.

[0099] A second analysis unit is configured to obtain an initial pixel difference according to the position change of the moving object in the environment images of the continuous frames, and to obtain a pixel displacement of the moving object in the environment images of the continuous frames according to the motion speed.

[0100] A second calculation unit is configured to subtract the pixel displacement from the initial pixel difference to obtain an absolute pixel difference.

[0101] A third calculation unit is configured to obtain the sweeping machine pose by a bundle adjustment method according to the absolute pixel difference.

[0102] In this embodiment, each module, unit and subunit in the sweeping machine positioning device based on a dynamic environment is configured to perform each step in the sweeping machine positioning method based on a dynamic environment, and the specific implementation process is not described here.

[0103] The embodiment provides a dynamic environment-based positioning device of a sweeping machine. A positioning system of the sweeping machine acquires a three-dimensional map point cloud and an environment image through the sweeping machine, then performs semantic segmentation on the three-dimensional map point cloud according to semantic information contained in the environment image, and obtains a mobile object point cloud. The positioning system judges whether an object corresponding to the mobile object point cloud is a mobile object. If the object corresponding to the mobile object point cloud is a mobile object, the object motion speed and the object pose of the mobile object are analyzed according to the position of the mobile object in the environment image of a continuous frame. The positioning system uses the motion speed and the object pose to perform re-projection matching in the environment image of the continuous frame, and solves the sweeping machine pose of the sweeping machine. In the application, the positioning system fully utilizes the information contained in the mobile object to position the sweeping machine, thereby effectively improving the information utilization rate and the positioning accuracy of the sweeping machine.

[0104] Reference Figure 3 In the embodiment of the application, a computer device is also provided, which can be a server, and the internal structure thereof can be as shown in Figure 3 The computer device includes a processor, a memory, a network interface and a database connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is used to store environment images and other data. The network interface of the computer device is used to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement a dynamic environment-based positioning method of a sweeping machine.

[0105] The processor executes the steps of the dynamic environment-based positioning method of the sweeping machine.

[0106] S1: acquiring a three-dimensional map point cloud and an environment image;

[0107] S2: performing semantic segmentation on the three-dimensional map point cloud according to semantic information contained in the environment image, to obtain a mobile object point cloud;

[0108] S3: judging whether an object corresponding to the mobile object point cloud is a mobile object;

[0109] S4: if the object corresponding to the mobile object point cloud is a mobile object, analyzing the object motion speed and the object pose of the mobile object according to the position of the mobile object in the environment image of a continuous frame;

[0110] S5: using the motion speed and the object pose to perform re-projection matching in the environment image of the continuous frame, and solving the sweeping machine pose of the sweeping machine.

[0111] Further, the step of performing semantic segmentation on the three-dimensional point cloud according to semantic information contained in the environment image to obtain a mobile object point cloud comprises:

[0112] S201: performing feature extraction on the environment image to obtain object feature points corresponding to each object contained in the environment image;

[0113] S202: performing object type identification on each object feature point to obtain mobile object feature points corresponding to a mobile object through screening;

[0114] S203: clustering and segmenting three-dimensional points corresponding to the mobile object feature points in the three-dimensional point cloud according to three-dimensional information corresponding to the mobile object feature points to obtain the mobile object point cloud.

[0115] Further, the step of clustering and segmenting three-dimensional points corresponding to the mobile object feature points in the three-dimensional point cloud according to three-dimensional information corresponding to the mobile object feature points to obtain the mobile object point cloud comprises:

[0116] S2031: obtaining an object shape corresponding to the mobile object feature points according to the object type of the mobile object feature points, wherein the object shape comprises volume information;

[0117] S2032: clustering and segmenting three-dimensional points corresponding to the mobile object feature points according to the object shape to obtain the mobile object point cloud.

[0118] Further, the sweeper moves at a constant speed, and the step of analyzing object motion speed and object pose of the mobile object according to positions of the mobile object in environment images of consecutive frames comprises:

[0119] S401: determining whether the mobile object is in a motion state;

[0120] S402: if the mobile object is in a motion state, calling a sweeper motion speed of the sweeper, and calculating the object motion speed and the object pose of the mobile object according to the sweeper motion speed and position changes of the mobile object in environment images of consecutive frames.

[0121] Further, the step of determining whether the mobile object is in a motion state comprises:

[0122] S4011: obtaining an object image position of the mobile object in environment images of consecutive frames, wherein the object image position is a position coordinate of the mobile object in a world coordinate system, and the world coordinate system is a coordinate system established with a scene in which the sweeper is located as a reference.

[0123] S4012: judging whether positions of each object image are same or not;

[0124] S4013: if the positions of each object image are different, determining that the moving object is in a motion state.

[0125] Further, the sweeping robot is provided with a three-dimensional sensor, and the step of obtaining the three-dimensional map point cloud comprises:

[0126] S101: continuously collecting three-dimensional point cloud data through the three-dimensional sensor in the motion process of the sweeping robot;

[0127] S102: splicing the three-dimensional point cloud data to obtain the three-dimensional map point cloud.

[0128] Further, the step of using the motion speed and the object pose to perform re-projection matching in the continuous frame environment image to obtain the sweeping robot pose comprises:

[0129] S501: obtaining a position change of the moving object in the continuous frame environment image based on the object pose of the moving object at continuous time points, wherein the continuous time points correspond to shooting time points of the continuous frame environment image;

[0130] S502: obtaining an initial pixel difference according to the position change of the moving object in the continuous frame environment image, and obtaining a pixel displacement of the moving object in the continuous frame environment image according to the motion speed;

[0131] S503: using the initial pixel difference to subtract the pixel displacement to obtain an absolute pixel difference;

[0132] S504: obtaining the sweeping robot pose by a bundle adjustment method according to the absolute pixel difference.

[0133] An embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement a sweeping robot positioning method based on a dynamic environment, and the sweeping robot positioning method based on the dynamic environment specifically comprises:

[0134] S1: obtaining a three-dimensional map point cloud and an environment image;

[0135] S2: performing semantic segmentation on the three-dimensional map point cloud according to semantic information contained in the environment image to obtain a moving object point cloud;

[0136] S3: judging whether an object corresponding to the moving object point cloud is a moving object or not;

[0137] S4: If the object corresponding to the mobile object point cloud is a mobile object, then according to the position of the mobile object in the environment images of consecutive frames, the object motion speed and object pose of the mobile object are obtained by analysis;

[0138] S5: Using the motion speed and the object pose, re-projection matching is performed in the environment images of consecutive frames to obtain the robot pose of the robot.

[0139] Further, the step of performing semantic segmentation on the three-dimensional map point cloud according to the semantic information contained in the environment image to obtain a mobile object point cloud comprises:

[0140] S201: Feature extraction is performed on the environment image to obtain object feature points corresponding to each object contained in the environment image;

[0141] S202: Object type recognition is performed on each object feature point to obtain mobile object feature points corresponding to mobile objects;

[0142] S203: According to the three-dimensional information corresponding to the mobile object feature points, the three-dimensional points corresponding to the mobile object feature points in the three-dimensional map point cloud are clustered and segmented to obtain the mobile object point cloud.

[0143] Further, the step of performing semantic segmentation on the three-dimensional map point cloud according to the semantic information contained in the environment image to obtain a mobile object point cloud comprises:

[0144] S2031: According to the object type of the mobile object feature point, the corresponding object shape is obtained, and the object shape comprises volume information;

[0145] S2032: According to the object shape, the three-dimensional points corresponding to the mobile object feature points are clustered and segmented to obtain the mobile object point cloud.

[0146] Further, the robot moves at a constant speed, and the step of obtaining the object motion speed and object pose of the mobile object according to the position of the mobile object in the environment images of consecutive frames comprises:

[0147] S4011: Obtain the object image position of the mobile object in the environment images of consecutive frames, wherein the object image position is the position coordinates of the mobile object in a world coordinate system, and the world coordinate system is a coordinate system established based on the scene where the robot is located;

[0148] S4012: Determine whether each object image position is the same;

[0149] S4013: If the object image positions are different, it is determined that the moving object is in a motion state.

[0150] Further, the sweeping machine is provided with a three-dimensional sensor, and the step of obtaining the three-dimensional map point cloud comprises:

[0151] S101: In the motion process of the sweeping machine, three-dimensional point cloud data is continuously collected by the three-dimensional sensor;

[0152] S102: The three-dimensional point cloud data is spliced to obtain the three-dimensional map point cloud.

[0153] Further, the step of using the motion speed and the object pose to perform re-projection matching in the environment images of continuous frames to obtain the sweeping machine pose comprises:

[0154] S501: Based on the object poses of the moving object at continuous time points, the position changes of the moving object in the environment images of continuous frames are obtained, wherein the continuous time points correspond to the shooting time points of the environment images of the continuous frames;

[0155] S502: An initial pixel difference is obtained according to the position changes of the moving object in the environment images of continuous frames, and a pixel displacement of the moving object in the environment images of continuous frames is obtained according to the motion speed;

[0156] S503: The absolute pixel difference is obtained by subtracting the pixel displacement from the initial pixel difference;

[0157] S504: The sweeping machine pose is obtained by a bundle adjustment method according to the absolute pixel difference.

[0158] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiment methods can be included. Any reference to memory, storage, databases, or other media in this application and in examples provided herein, unless specifically stated otherwise, can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0159] It should be noted that in this document, the terms "comprising", "including", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, an apparatus, a first object or a method that comprises a list of elements does not only include those elements, but can also include other elements not expressly listed or inherent to such process, apparatus, first object or method. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, apparatus, first object or method that includes the element.

[0160] The above description is only the preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, which is made by using the content of the specification and drawings of the present application, is also included in the patent protection scope of the present application.

Claims

1. A sweeper positioning method based on a dynamic environment, characterized in that: The method comprises the following steps: acquiring a three-dimensional map point cloud and an environment image; performing semantic segmentation on the three-dimensional map point cloud according to semantic information contained in the environment image to obtain a mobile object point cloud; judging whether an object corresponding to the mobile object point cloud is a mobile object; if the object corresponding to the mobile object point cloud is a mobile object, then analyzing an object motion speed and an object pose of the mobile object according to a position of the mobile object in environment images of consecutive frames; performing re-projection matching in the environment images of the consecutive frames using the motion speed and the object pose to obtain a robot vacuum cleaner pose of the robot vacuum cleaner; the step of performing re-projection matching in the environment images of the consecutive frames using the motion speed and the object pose to obtain the robot vacuum cleaner pose of the robot vacuum cleaner comprises the following steps: obtaining a position change of the mobile object in the environment images of the consecutive frames based on the object pose of the mobile object at consecutive time points, wherein the consecutive time points correspond to photographing time points of the environment images of the consecutive frames; obtaining an initial pixel difference according to the position change of the mobile object in the environment images of the consecutive frames, and analyzing a pixel displacement of the mobile object in the environment images of the consecutive frames according to the motion speed; subtracting the pixel displacement from the initial pixel difference to obtain an absolute pixel difference; obtaining the robot vacuum cleaner pose by bundle adjustment method according to the absolute pixel difference; the robot vacuum cleaner is provided with a three-dimensional sensor, and the step of acquiring the three-dimensional map point cloud comprises the following steps: continuously collecting three-dimensional point cloud data by the three-dimensional sensor during movement of the robot vacuum cleaner; splicing the three-dimensional point cloud data to obtain the three-dimensional map point cloud. 2.The dynamic environment-based robot vacuum positioning method of claim 1, wherein, the step of performing semantic segmentation on the three-dimensional map point cloud according to the semantic information contained in the environment image to obtain the mobile object point cloud comprises the following steps: performing feature extraction on the environment image to obtain object feature points corresponding to each object contained in the environment image; performing object type identification on each object feature point to screen mobile object feature points corresponding to mobile objects; performing clustering and segmentation on three-dimensional points corresponding to the mobile object feature points in the three-dimensional map point cloud according to three-dimensional information corresponding to the mobile object feature points to obtain the mobile object point cloud. 3.The dynamic environment based robot vacuum positioning method of claim 2, wherein, the step of performing clustering and segmentation on three-dimensional points corresponding to the mobile object feature points in the three-dimensional map point cloud according to three-dimensional information corresponding to the mobile object feature points to obtain the mobile object point cloud comprises the following steps: obtaining an object shape corresponding to the mobile object feature points according to the object type of the mobile object feature points, wherein the object shape comprises volume information; performing clustering and segmentation on the three-dimensional points corresponding to the mobile object feature points according to the object shape to obtain the mobile object point cloud.

4. The dynamic environment-based robot localization method of claim 1, wherein, the robot vacuum cleaner moves at a constant speed, and the step of analyzing the object motion speed and the object pose of the mobile object according to the position of the mobile object in the environment images of the consecutive frames comprises the following steps: judging whether the mobile object is in a motion state; If the moving object is in a moving state, a sweeping machine moving speed of the sweeping machine is called, and the object moving speed and the object pose of the moving object are calculated according to the sweeping machine moving speed and a position change of the moving object in the environment images of the continuous frames.

5. The dynamic environment-based robot localization method of claim 4, wherein, The step of judging whether the moving object is in a moving state comprises: An object image position of the moving object in the environment images of the continuous frames is acquired, the object image position being a position coordinate of the moving object in a world coordinate system, and the world coordinate system being a coordinate system established with the scene where the sweeping machine is located as a reference; It is judged whether each object image position is the same; If each object image position is different, it is determined that the moving object is in a moving state.

6. A dynamic environment based robot positioning device for performing the method of any one of claims 1-5, characterized by, Comprise: An acquisition module is configured to acquire a three-dimensional map point cloud and an environment image; A segmentation module is configured to perform semantic segmentation on the three-dimensional map point cloud according to semantic information contained in the environment image to obtain a moving object point cloud; A judgment module is configured to judge whether an object corresponding to the moving object point cloud is a moving object; An analysis module is configured to, if the object corresponding to the moving object point cloud is a moving object, analyze an object moving speed and an object pose of the moving object according to a position of the moving object in the environment images of the continuous frames; A positioning module is configured to use the moving speed and the object pose to perform re-projection matching in the environment images of the continuous frames to obtain a sweeping machine pose of the sweeping machine. 7.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-6. The processor executes the computer program to implement the steps of the method in any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 5.

Images