Path planning method, device and system

Abstract

The embodiments of the present invention disclose a path planning method, device and system for circumventing and avoiding obstacles when it is determined that an obstacle in a space to be cleaned is of a preset obstacle type. The method includes: obtaining the obstacle type in the space to be cleaned through laser radar data; if it is determined that the obstacle type is the preset obstacle type, obtaining a deep reinforcement learning model corresponding to the obstacle type; determining a predicted execution action of a sweeping machine based on the deep reinforcement learning model; controlling the sweeping machine to execute a corresponding action according to the predicted execution action to reach a current target point corresponding to the predicted execution action in the space to be cleaned; if it is determined that the current obstacle type is of the preset obstacle type based on the current laser radar data corresponding to the current target point, returning to the step of determining the predicted execution action of the sweeping machine based on the deep reinforcement learning model until the current obstacle type is not the preset obstacle type.

Description

Technical Field

[0001] The present invention relates to the field of robotics technology, and in particular to a path planning method, device and system. Background Art

[0002] With the development of technology, sweepers (i.e. cleaning robots) have been widely used in household cleaning. When sweepers are cleaning the rooms to be cleaned at home, they often avoid obstacles by colliding with them. This makes the sweepers inflexible and takes too much time to avoid obstacles in scenes with many obstacles. At the same time, when faced with obstacles such as "C"-shaped obstacles, they are prone to fall into local optimality and cannot escape. Summary of the invention

[0003] The embodiments of the present invention provide a path planning method, device and system, which can effectively utilize the exploration capability of reinforcement learning to enable the model to explore different routes, thereby achieving rapid obstacle avoidance of preset obstacle types.

[0004] In a first aspect, an embodiment of the present invention provides a path planning method, which is applied to a sweeping machine. The path planning method includes:

[0005] Obtain the type of obstacles in the space to be cleaned through LiDAR data;

[0006] If it is determined that the obstacle type is a preset obstacle type, obtaining a deep reinforcement learning model corresponding to the obstacle type;

[0007] Determining a predicted execution action of the sweeper based on the deep reinforcement learning model;

[0008] Controlling the sweeping machine to perform a corresponding action according to the predicted execution action, so as to reach a current target point corresponding to the predicted execution action in the space to be cleaned;

[0009] If the current obstacle type is determined to be the preset obstacle type based on the current lidar data corresponding to the current target point, return to the step of determining the predicted execution action of the sweeper based on the deep reinforcement learning model until the current obstacle type is not the preset obstacle type.

[0010] In a second aspect, an embodiment of the present invention further provides a path planning method, which is applied to a path planning system, wherein the path planning system includes a sweeping machine and a cloud, and the sweeping machine is communicatively connected to the cloud; the path planning method includes:

[0011] The cloud sends a pre-trained model to the sweeping robot;

[0012] The sweeping robot receives the pre-trained model sent by the cloud, and performs model training on the pre-trained model based on the acquired current home environment map data to obtain a deep reinforcement learning model;

[0013] The sweeper obtains the type of obstacles in the space to be cleaned through laser radar data;

[0014] If the sweeping machine determines that the obstacle type is a preset obstacle type, then acquiring the deep reinforcement learning model corresponding to the obstacle type;

[0015] The sweeping machine determines a predicted execution action of the sweeping machine based on the deep reinforcement learning model;

[0016] The sweeping machine performs a corresponding action according to the predicted execution action to reach a current target point corresponding to the predicted execution action in the space to be cleaned;

[0017] If the sweeping robot determines that the current obstacle type is the preset obstacle type based on the current lidar data corresponding to the current target point, it returns to execute the step of determining the predicted execution action of the sweeping robot based on the deep reinforcement learning model until the current obstacle type is not the preset obstacle type.

[0018] In a third aspect, an embodiment of the present invention further provides a path planning device, which includes a unit for executing the path planning method in the first aspect.

[0019] In a fourth aspect, an embodiment of the present invention further provides a path planning system, which includes a sweeping machine and a cloud, wherein the sweeping machine is communicatively connected to the cloud; the path planning system is used to execute the path planning method in the above-mentioned second aspect.

[0020] In a fifth aspect, an embodiment of the present invention further provides a computer device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the sweeping robot is equipped with an image acquisition device, and when the processor executes the computer program, the path planning method in the first aspect or the second aspect mentioned above is implemented.

[0021] In a sixth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a computer program, wherein the computer program includes program instructions, and wherein the program instructions, when executed by a processor, can implement the path planning method in the first aspect or the second aspect described above.

[0022] Compared with the prior art, the embodiments of the present invention have the following advantages:

[0023] When an obstacle of a preset obstacle type is detected in the space to be cleaned, the sweeping robot needs to continuously call the deep reinforcement learning model to determine the next predicted execution action and control the sweeping robot to move so as to avoid obstacles of the preset obstacle type until the sweeping robot has completed the detour around the obstacles of the preset obstacle type. The exploration capability of reinforcement learning is effectively utilized to enable the model to explore different routes, thereby achieving rapid obstacle avoidance of obstacles of the preset obstacle type. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0025] Figure 1 A schematic diagram of a flow chart of a path planning method provided by an embodiment of the present invention;

[0026] Figure 2 A schematic diagram of a sub-process of a path planning method provided by an embodiment of the present invention;

[0027] Figure 3 A schematic diagram of a sweeping machine moving from a current position to a next position in a room to be cleaned in a path planning method provided by an embodiment of the present invention;

[0028] Figure 4 Another schematic diagram of a flow chart of a path planning method provided by an embodiment of the present invention;

[0029] Figure 5 A schematic diagram of a flow chart of a path planning method provided by another embodiment of the present invention;

[0030] Figure 6 Another schematic diagram of a flow chart of a path planning method provided by another embodiment of the present invention;

[0031] Figure 7 A schematic block diagram of a path planning device provided in an embodiment of the present invention;

[0032] Figure 8 A schematic block diagram of a path planning system provided by an embodiment of the present invention;

[0033] Fig. 9 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0036] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0037] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0038] The embodiments of the present invention provide a path planning method, device, system, sweeper and end-cloud system. The execution subject of the path planning method may be the path planning device provided in the embodiments of the present invention, or a sweeper integrated with the path planning device, wherein the path planning device may be implemented in hardware or software. The execution subject of the path planning method may also be the path planning system provided in the embodiments of the present invention, wherein the path planning system includes a sweeper and a cloud, the sweeper is communicatively connected to the cloud, and the path planning system may be implemented in hardware or software.

[0039] In the present invention, the sweeping machine includes the components of a conventional sweeping machine by default, such as a housing, rollers, a sweeping structure assembly, a laser radar, a mopping structure assembly, and a control module, etc. The sweeping machine continuously transmits and receives laser signals through the laser radar, and can obtain information about the surrounding environment in real time, providing the sweeping machine with accurate navigation and obstacle avoidance capabilities.

[0040] The specific process of the embodiment of the present invention is described below, taking a sweeping machine as an example for illustration.

[0041] Figure 1 FIG. 1 is a flow chart of a path planning method provided by an embodiment of the present invention. Figure 1As shown, the method includes the following steps S110-S150.

[0042] S110, obtaining the type of obstacles in the space to be cleaned through laser radar data.

[0043] In this embodiment, when the sweeping machine scans through the laser radar and finds that there are obstacles in the space to be cleaned, and determines the obstacle type corresponding to the obstacle in the space to be cleaned based on the laser radar data, it can determine whether to call the pre-trained deep reinforcement learning model for path planning by judging whether the obstacle type is a preset obstacle type. When obtaining the obstacle type corresponding to the obstacle in the space to be cleaned through the laser radar data, the laser radar continuously emits and receives laser signals, and obtains the outer contour shape of the obstacle in the space to be cleaned in real time to determine the obstacle type.

[0044] S120: If it is determined that the obstacle type is a preset obstacle type, obtaining a deep reinforcement learning model corresponding to the obstacle type.

[0045] In this embodiment, if the preset obstacle type stored in the sweeping robot is a "C"-shaped obstacle (in specific implementation, it is not limited to "C"-shaped obstacles, but can also be other types of obstacles set based on actual needs), then when the obstacle type in the space to be cleaned is determined based on the lidar data, it is also a "C"-shaped obstacle, indicating that the sweeping robot needs to call a pre-trained deep reinforcement learning model for path planning.

[0046] S130. Determine a predicted execution action of the sweeping robot based on the deep reinforcement learning model.

[0047] In this embodiment, after the sweeping machine calls the deep reinforcement learning model, the current motion state data of the sweeping machine can be used as the input of the deep reinforcement learning model, so as to calculate the predicted execution action of the sweeping machine at the next moment. Specifically, the motion state data acquired by the sweeping machine includes at least the rotation angle, angular velocity and linear velocity during implementation. The input vector composed of the above multiple parameters included in the motion state data is input into the deep reinforcement learning model to obtain the predicted execution action calculated by the deep reinforcement learning model. Moreover, the parameter composition of the motion state data is also referenced in the predicted execution action, which also includes at least the rotation angle, angular velocity and linear velocity.

[0048] In one embodiment, if Figure 2 As shown, step S130 includes:

[0049] S131, obtaining a first model in the deep reinforcement learning model, inputting the obtained motion state data of the sweeping robot into the first model, and obtaining a model reasoning action;

[0050] S132: Obtain a second model in the deep reinforcement learning model, and randomly generate a random execution action based on the second model;

[0051] S133: Fusing the model inference action and the random execution action based on a preset weighted fusion strategy to obtain the predicted execution action.

[0052] In this embodiment, the deep reinforcement learning model used in this application is a multi-model structure, which at least includes a first model for performing action reasoning output based on motion state data and a second model for randomly generating a random execution action. Figure 3 As shown, if the sweeping robot has moved to the current position P in the space to be cleaned now For example, the sweeper is at the current position P now Based on the laser radar data, the obstacle type in the space to be cleaned is determined to be a preset obstacle type (specifically, Figure 3 The "C" shaped obstacle shown in the figure is present, and at the current position P now The motion state data of the sweeper can be determined. At this time, the parameters in the motion state data including the rotation angle, angular velocity and linear velocity of the sweeper are composed into an input parameter sequence (such as forming an input vector in the order of rotation angle, angular velocity and linear velocity) and input into the first model, and the model reasoning action output by the first model can be obtained. Moreover, when the second model outputs a random execution action, the parameter composition of the reference motion state data is also used in the random execution action, which also includes at least rotation angle, angular velocity and linear velocity; wherein, the input parameters of the second model have no fixed requirements, and can be the current system time, predicted action generation instructions, etc. The essence of the second model is a random function, and can output a random execution action including at least rotation angle, angular velocity and linear velocity.

[0053] Specifically, there is no clear order between step S131 and step S132, and both steps can be performed simultaneously, as long as the output model reasoning action and random execution action can be obtained. now After the corresponding model inference action and random execution action, the model inference action and random execution action are fused based on the preset weighted fusion strategy to obtain the current position P nowThe corresponding predicted execution action, and then the sweeping machine controls the sweeping machine to execute the corresponding action according to the predicted execution action, so as to reach the next position. In the process of predicting the predicted execution action based on the deep reinforcement learning model, it will not only use the known effective strategy to generate a model reasoning action to explore the unknown area information, but also generate a random execution action. According to the weighted fusion of the random execution action and the model reasoning action, it will speed up the learning of the action strategy for escaping in the space to be cleaned with the preset obstacle type.

[0054] In one embodiment, step S133 includes:

[0055] Obtaining a first weight coefficient corresponding to the model reasoning action in the weighted fusion strategy, and obtaining a second weight coefficient corresponding to the random execution action in the weighted fusion strategy; wherein the sum of the first weight coefficient and the second weight coefficient is equal to a first preset value;

[0056] Obtain a first multiplication result corresponding to the multiplication of the model reasoning action and the first weight coefficient, obtain a second multiplication result corresponding to the multiplication of the random execution action and the second weight coefficient, and sum the first multiplication result and the second multiplication result to obtain the predicted execution action.

[0057] In this embodiment, the weighted fusion strategy corresponds to the following formula (1) during specific implementation:

[0058] Action pred = (1-a)*Action net +a*Action random (1)

[0059] In formula (1), Action pred Indicates the predicted execution action, Action net Represents the model reasoning action, Action random represents a random execution action, 1-a represents a first weight coefficient, and a represents a second weight coefficient; specifically, the first preset value is set to 1, and the first weight coefficient and the second weight coefficient in the weighted fusion strategy can be flexibly adjusted according to actual user needs.

[0060] After weighted fusion of random execution actions and model reasoning actions, it is possible to quickly determine the control of the sweeper from the current position P now Move to the next position, denoted as P next (See Figure 3 The P shown next ) of the predicted execution action.

[0061] S140: Control the sweeping robot to perform a corresponding action according to the predicted execution action, so as to reach a current target point corresponding to the predicted execution action in the space to be cleaned.

[0062] In this embodiment, after the sweeping machine determines the predicted execution action based on the deep reinforcement learning model, the sweeping machine is controlled to execute the corresponding action to move from the current position P now Move to the next position P next , and the next position P next It can be regarded as the current target point of the sweeper in the space to be cleaned after the predicted execution action is completed. At this time, in order to determine whether the sweeper needs to bypass obstacles of the preset obstacle type in the space to be cleaned after reaching the current target point, the sweeper needs to further obtain the current lidar data and determine whether the sweeper is still facing obstacles of the preset obstacle type in the space to be cleaned based on the current lidar data.

[0063] If it is determined based on the current laser radar data that the sweeping machine is still facing an obstacle of the preset obstacle type in the space to be cleaned, that is, it is determined based on the current laser radar data corresponding to the current target point that the current obstacle type is the preset obstacle type, the sweeping machine needs to continue to call the deep reinforcement learning model to determine the next predicted execution action and control the sweeping machine to move so as to avoid the obstacle of the preset obstacle type. If it is determined based on the current laser radar data that the sweeping machine is not facing an obstacle of the preset obstacle type in the space to be cleaned, that is, it is determined based on the current laser radar data corresponding to the current target point that the current obstacle type is not the preset obstacle type, it means that the sweeping machine has completed the detour around the above-mentioned obstacle of the preset obstacle type, and it is no longer necessary to call the deep reinforcement learning model.

[0064] In one embodiment, in order to determine and label the obstacle avoidance behavior reward and penalty value for each predicted execution action outputted in the deep reinforcement learning model, after step S110, the following is further included:

[0065] Acquire a current starting point of the sweeping machine in the space to be cleaned, and acquire first motion information corresponding to the current starting point;

[0066] And after step S140, the method further includes:

[0067] Acquire second motion information corresponding to the current target point;

[0068] Determining a predicted action labeling value corresponding to the predicted execution action based on a preset sports reward and punishment value determination strategy, the first sports information, and the second sports information;

[0069] A mapping relationship is established between the predicted action label value and the predicted execution action, and the relationship is saved in a model training set, and the model training set is updated; wherein the model training set is used to perform model training on the deep reinforcement learning model when preset model training conditions are met.

[0070] In this embodiment, when determining the first operation information and the second motion information of the sweeping machine, reference is made to Figure 3 , specifically:

[0071] 1) When the sweeper is at the current starting point and recorded as Figure 3 P now When the first motion information includes at least a first distance and a first angle, the first distance is the distance between the current starting point and the final target point preset in the space to be cleaned (which is Figure 3 P end denoted by ), if the line between the current starting point and the final target point is recorded as the first reference line, the first angle is the angle between the positive direction line of the sweeping machine at the current starting point and the first reference line; Figure 3 In the figure, the first spacing is represented by d3, and the first angle is represented by β;

[0072] 2) When the sweeper is at the current target point and recorded as Figure 3 P next When , the second motion information includes at least a second spacing and a second angle, the second spacing is the spacing between the current target point and the final target point preset in the space to be cleaned, if the line between the current target point and the final target point is recorded as the second reference line, the second angle is the angle between the forward direction line of the sweeping machine at the current target point and the second reference line; Figure 3 In the figure, the second distance is represented by d2 and the first angle is represented by α.

[0073] Among them, Figure 3 In the example, the third distance between the current target point and the current starting point may be represented by d1. When determining the predicted action label value corresponding to the predicted execution action based on the preset motion reward and penalty value determination strategy, the first motion information, and the second motion information, the specific process is as follows:

[0074] 11) Get P next With P now The third spacing d1 between them, get P now With P end The first spacing d3 between them, get P next With P end A second spacing d2 between them;

[0075] 12) Determine a fourth distance d4 according to d2-d1;

[0076] 13) determining a fifth distance d5 according to the absolute value of the fourth distance d4, that is, d5=abs(d4);

[0077] 14) Get the cosine value cosβ of the first angle β, get the absolute value abs(α-β) of the angle difference between the second angle α and the first angle β and record it as the current angle difference θ diff ;

[0078] 15) If it is determined that the second interval d2≠0, the fifth interval d5 is normalized, that is, d5 is updated by d5 / d1; if it is determined that the second interval d2=0, d5 is updated by 0;

[0079] 16) If confirmed , the fifth interval d5 updated in step 15) is updated again, that is, d5 is updated by d5*(-6); if it is determined , the fifth distance d5 updated in step 15) is updated again, that is, d5 is updated by d5*6;

[0080] 17) If it is determined that α-β>0, then the current angle difference θ in step 14) diff Update the process, that is, use θ diff * (-3) Update θ diff ; If it is determined that α-β≤0, then the current angle difference θ in step 14) diff Update the process, that is, use θ diff *2 Update θ diff ;

[0081] 18) In the sweeper from P now Move to P next If no collision occurs during the process, the fifth distance d5 obtained by the latest update in step 16) and the current angle difference θ obtained by the latest update in step 17) are obtained. diff , with d5+cosβ+θ diff As the predicted action label value corresponding to the predicted execution action;

[0082] 19) In the sweeper from P now Move to P next If a collision occurs during the process, a preset negative value (which is a large negative number, such as -10000, -100000, etc.) is obtained and the preset negative value is used as the predicted action label value corresponding to the predicted execution action.

[0083] Still refer to Figure 3 , if the sweeper's P next With P end is the same point, that is, the sweeper can be judged to have reached the final target point P when it reaches the current target pointend , a preset positive value (which is a large negative number, such as 10000, 100000, etc.) is obtained and used as the predicted action label value corresponding to the predicted execution action. It can be seen that based on the above-mentioned motion reward and penalty value determination strategy, combined with the first motion information of the sweeper at the current starting point and the second operation information at the current target point, the predicted action label value corresponding to the predicted execution action can be quickly determined.

[0084] Moreover, in the process of avoiding obstacles of preset obstacle types in the space to be cleaned by the sweeper, each predicted execution action output by the deep reinforcement learning model can be called, and a corresponding predicted action label value can be determined by referring to the processing process of the above steps 11)-19). After the mapping relationship is established between the two, a predicted execution action with a label value can be obtained, and it can be further used as training data for the deep reinforcement learning model. That is, each predicted execution action with a label value is saved to the model training set, and the model training set is updated. After completing the cleaning work of the space to be cleaned this time, it can be regarded as a situation that meets the preset model training conditions. At this time, the deep reinforcement learning model can be trained through the model training set, and the deep reinforcement learning model with the latest state can be updated and saved after the training is completed. Of course, other situations that meet the preset model training conditions can also be that the time interval between the current system time and the previous model training is equal to the preset model training cycle, etc. The specific implementation is not limited to the several situations of the above examples, and can be flexibly set according to the actual needs of the user.

[0085] S150. If the current obstacle type is determined to be the preset obstacle type based on the current lidar data corresponding to the current target point, return to the step of determining the predicted execution action of the sweeper based on the deep reinforcement learning model until the current obstacle type is not the preset obstacle type.

[0086] In this embodiment, if the current obstacle type is determined to be the preset obstacle type based on the current laser radar data corresponding to the current target point, it means that the sweeper is still facing an obstacle of the preset obstacle type. At this time, it is still necessary to continue to call the deep reinforcement learning model to determine the predicted execution action of the sweeper, and the termination condition of the above loop process is that the current obstacle type is not the preset obstacle type based on the current laser radar data corresponding to the current target point. After the above loop process is terminated, it means that the sweeper has successfully bypassed the above obstacle of the preset obstacle type, and it is no longer necessary to call the deep reinforcement learning model to predict the execution action.

[0087] In one embodiment, after step S150, the method further includes:

[0088] When the current obstacle type is not the preset obstacle type, a current planned path is generated based on a preset path planning algorithm to control the sweeping machine to perform sweeping and cleaning work according to the current planned path.

[0089] In this embodiment, after the sweeping machine successfully bypasses the obstacles of the preset obstacle type based on several calls to the deep reinforcement learning model for execution action prediction, in order to complete the cleaning task of the space to be cleaned more quickly and reach the final target point, the current planned path can be generated by calling the local preset path planning algorithm to control the sweeping machine to perform the sweeping cleaning work according to the current planned path. Moreover, the starting point of the current planned path is the end point after the sweeping machine successfully bypasses the obstacles of the preset obstacle type, and the end point of the current planned path is the final target point in the space to be cleaned.

[0090] In one embodiment, if Figure 4 As shown, before step S110, it also includes:

[0091] S101, obtaining current home environment map data, and adding the current home environment map data to a locally constructed historical environment map data set;

[0092] S102, uploading the current home environment map data to the cloud;

[0093] S103, obtaining the pre-trained model sent by the cloud;

[0094] S104: Perform model training on the pre-trained model based on the current home environment map data to obtain the deep reinforcement learning model.

[0095] The historical environment map data set is stored in a local data storage area; the pre-trained model is obtained by training an initial deep reinforcement learning model on the cloud based on the current home environment map data and other home environment map data uploaded by other sweepers;

[0096] In this embodiment, when the sweeping machine is deployed in the home corresponding to the space to be cleaned in the above example, the current home environment map data can be first obtained through the walking and scanning of the sweeping machine, and the current home environment map data is added to the historical environment map data set constructed locally (because the model training also needs to be performed locally on the sweeping machine, it is necessary to add the obtained current home environment map data to the historical environment map data set), thereby realizing the collection and local storage of the current home environment map data. When there are multiple sweeping machines as in the above example that have obtained multiple different current home environment map data, they are also added to the historical environment map data set constructed locally by the sweeping machine with reference to the above process.

[0097] Then, the sweeping machine uploads the current home environment map data to the cloud. If multiple different sweeping machines upload the current home environment map data they have collected to the cloud, a training set can be composed based on multiple current home environment map data in the cloud and the initial deep reinforcement learning model can be trained to obtain a pre-trained model. After the model training is completed in the cloud and the pre-trained model is obtained, it can be sent to each sweeping machine. The pre-trained model obtained in the cloud is generated by iteratively training the initial deep reinforcement learning model based on multiple different current home environment map data, which can be regarded as the initial training process of the model, which can ensure its strong generalization and robustness.

[0098] Finally, if we still take the sweeping machine in the above example as an example, after the sweeping machine receives the pre-trained model sent from the cloud, it can also train the pre-trained model based on the current home environment map data stored in the local historical environment map data set to obtain the deep reinforcement learning model. The deep reinforcement learning model obtained locally on the sweeping machine based on the current home environment map data and the pre-trained model can be regarded as a secondary training process of the model, which can further improve the model's adaptability to its environment.

[0099] To summarize, when the path planning method provided by the embodiment of the present invention is used to control the operation of a sweeping machine, if an obstacle of a preset obstacle type is detected in the space to be cleaned, the sweeping machine needs to continuously call the deep reinforcement learning model to determine the next predicted execution action and control the sweeping machine to move so as to avoid obstacles of the preset obstacle type until the sweeping machine has completed the detour around the obstacles of the preset obstacle type, thereby effectively utilizing the exploration capability of reinforcement learning to enable the model to explore different routes, thereby achieving rapid obstacle avoidance of obstacles of the preset obstacle type.

[0100] Figure 5 1 is a flow chart of a path planning method provided by another embodiment of the present invention. A path planning method provided by another embodiment of the present invention is applied to a path planning system, wherein the path planning system includes a sweeping machine and a cloud, wherein the sweeping machine is in communication connection with the cloud. Figure 5 As shown, the path planning method of this embodiment includes steps S210-S270.

[0101] S210, the cloud sends a pre-trained model to the sweeping robot;

[0102] S220, the sweeping robot receives the pre-trained model sent by the cloud, and performs model training on the pre-trained model based on the acquired current home environment map data to obtain a deep reinforcement learning model;

[0103] S230, the sweeping machine obtains the type of obstacles in the space to be cleaned through laser radar data;

[0104] S240: If the sweeping robot determines that the obstacle type is a preset obstacle type, then obtaining the deep reinforcement learning model corresponding to the obstacle type;

[0105] S250, the sweeping machine determines a predicted execution action of the sweeping machine based on the deep reinforcement learning model;

[0106] S260, the sweeping machine performs a corresponding action according to the predicted execution action to reach a current target point corresponding to the predicted execution action in the space to be cleaned;

[0107] S270: If the sweeping robot determines that the current obstacle type is the preset obstacle type based on the current lidar data corresponding to the current target point, it returns to the step of determining the predicted execution action of the sweeping robot based on the deep reinforcement learning model until the current obstacle type is not the preset obstacle type.

[0108] In this embodiment, what is different from the embodiment in which the aforementioned path planning method is applied to a sweeping robot is that before the sweeping robot bypasses obstacles of preset obstacle types in the cleaning space, a pre-trained model is sent from the cloud to the sweeping robot, and the sweeping robot trains the pre-trained model based on the pre-trained model received from the cloud and the current home environment map data obtained, to obtain the specific process of the deep reinforcement learning model.

[0109] After that, when the robot vacuum cleaner bypasses obstacles of the preset obstacle type in the cleaning space, it continuously calls the deep reinforcement learning model to determine the robot vacuum cleaner's predicted execution action, and the termination condition of the above loop process is to determine that the current obstacle type is not the preset obstacle type based on the current lidar data corresponding to the current target point. After the above loop process is terminated, it means that the robot vacuum cleaner has successfully bypassed the obstacles of the preset obstacle type, and it is no longer necessary to call the deep reinforcement learning model to predict the execution action.

[0110] In one embodiment, if Figure 6 As shown, before step S210, it also includes:

[0111] S201, the cloud receives the current home environment map data sent by the sweeping robot;

[0112] S202, the cloud obtains other home environment map data uploaded by other sweepers and forms a training set with the previous home environment map data;

[0113] S203. The cloud performs sweeping robot modeling and map modeling on the sweeping robot and the training set respectively according to the current simulation training request, and allocates corresponding current system computing resources to perform model iterative simulation training on the initial deep reinforcement learning model to obtain the pre-trained model.

[0114] In this embodiment, after obtaining the home environment map data sent by multiple sweepers in the cloud, a training set can be formed. Afterwards, if the current simulation training request is detected in the cloud, the specific training process is as follows:

[0115] 21) Modeling the sweeping robot for the sweeping robot, and modeling the map data of each home environment in the training set; more specifically, using but not limited to ROS (i.e., Robot Operating System), RViz (RViz is a 3D visualization tool in the robot operating system framework) and Gazebo (a physical environment simulation platform) to model the sweeping robot and map model the home environment map data;

[0116] 22) The cloud allocates the current system computing resources corresponding to the current simulation training request, and performs model training on the initial deep reinforcement learning model to obtain model iterative simulation training to obtain a pre-trained model; more specifically, the model training process is recorded and analyzed through the cloud-based log management module, and the trained pre-trained model is saved and managed through the cloud-based model management module.

[0117] The pre-trained model obtained in the cloud is generated by iteratively training the initial deep reinforcement learning model based on multiple different current home environment map data, which can be regarded as the initial training process of the model, which can ensure its strong generalization and robustness. Afterwards, after the sweeping machine receives the pre-trained model sent from the cloud, it can also train the pre-trained model based on the current home environment map data stored in the local historical environment map data set to obtain the deep reinforcement learning model. The deep reinforcement learning model obtained locally on the sweeping machine based on the current home environment map data and the pre-trained model can be regarded as a secondary training process of the model, which can further improve the model's adaptability to its environment.

[0118] Figure 7 is a schematic block diagram of a path planning device 100 provided in an embodiment of the present invention. Figure 7 As shown, corresponding to the above path planning method, the present invention also provides a path planning device 100. The path planning device 100 includes a unit for executing the above path planning method, and the device is configured in a sweeping machine. Figure 7The path planning device 100 includes a first obstacle type determination unit 110, a model acquisition unit 120, a predicted execution action acquisition unit 130, an execution action control unit 140 and a second obstacle type determination unit 150.

[0119] A first obstacle type determination unit 110 is used to obtain the type of obstacles in the space to be cleaned through laser radar data;

[0120] A model acquisition unit 120, configured to acquire a deep reinforcement learning model corresponding to the obstacle type if it is determined that the obstacle type is a preset obstacle type;

[0121] A predicted execution action acquisition unit 130, configured to determine a predicted execution action of the sweeper based on the deep reinforcement learning model;

[0122] An execution action control unit 140, configured to control the sweeping machine to execute a corresponding action according to the predicted execution action, so as to reach a current target point corresponding to the predicted execution action in the space to be cleaned;

[0123] The second obstacle type determination unit 150 is used to return to the step of determining the predicted execution action of the sweeper based on the deep reinforcement learning model if the current obstacle type is determined to be the preset obstacle type based on the current laser radar data corresponding to the current target point, until the current obstacle type is not the preset obstacle type.

[0124] In one embodiment, the predicted execution action acquisition unit 130 is specifically used for:

[0125] Obtaining a first model in the deep reinforcement learning model, inputting the acquired motion state data of the sweeping robot into the first model, and obtaining a model reasoning action;

[0126] Obtaining a second model in the deep reinforcement learning model, and randomly generating a random execution action based on the second model;

[0127] The model inference action and the random execution action are fused based on a preset weighted fusion strategy to obtain the predicted execution action.

[0128] In one embodiment, when the predicted execution action acquisition unit 130 performs the step of fusing the model inference action and the random execution action based on a preset weighted fusion strategy to obtain the predicted execution action, it is specifically used to:

[0129] Obtaining a first weight coefficient corresponding to the model reasoning action in the weighted fusion strategy, and obtaining a second weight coefficient corresponding to the random execution action in the weighted fusion strategy; wherein the sum of the first weight coefficient and the second weight coefficient is equal to a first preset value;

[0130] Obtain a first multiplication result corresponding to the multiplication of the model reasoning action and the first weight coefficient, obtain a second multiplication result corresponding to the multiplication of the random execution action and the second weight coefficient, and sum the first multiplication result and the second multiplication result to obtain the predicted execution action.

[0131] In one embodiment, the path planning device 100 further includes:

[0132] A first motion information acquisition unit, used to acquire a current starting point of the sweeping machine in the space to be cleaned, and acquire first motion information corresponding to the current starting point;

[0133] A second motion information acquisition unit, used to acquire second motion information corresponding to the current target point;

[0134] A predicted action label value acquisition unit, configured to determine a predicted action label value corresponding to the predicted execution action based on a preset sports reward and punishment value determination strategy, the first sports information, and the second sports information;

[0135] A model training set updating unit is used to establish a mapping relationship between the predicted action label value and the predicted execution action, save it to the model training set, and update the model training set; wherein, the model training set is used to perform model training on the deep reinforcement learning model when the preset model training conditions are met.

[0136] In one embodiment, the path planning device 100 further includes:

[0137] The path planning unit is used to generate a current planned path based on a preset path planning algorithm when the current obstacle type is not the preset obstacle type, so as to control the sweeping machine to perform sweeping and cleaning work according to the current planned path.

[0138] In one embodiment, the path planning device 100 further includes:

[0139] An environment map data acquisition unit, used to acquire current home environment map data, and add the current home environment map data to a locally constructed historical environment map data set; wherein the historical environment map data set is stored in a local data storage area;

[0140] An environment map data uploading unit, used to upload the current home environment map data to the cloud;

[0141] A pre-trained model acquisition unit, used to acquire the pre-trained model sent by the cloud; wherein the pre-trained model is obtained by training an initial deep reinforcement learning model on the cloud based on the current home environment map data and other home environment map data uploaded by other sweepers;

[0142] A model local training unit is used to perform model training on the pre-trained model based on the current home environment map data to obtain the deep reinforcement learning model.

[0143] To summarize, when the path planning device provided by the embodiment of the present invention is used to control the operation of a sweeping machine, if an obstacle of a preset obstacle type is detected in the space to be cleaned, the sweeping machine needs to continuously call the deep reinforcement learning model to determine the next predicted execution action and control the sweeping machine to move so as to avoid obstacles of the preset obstacle type until the sweeping machine has completed the detour around the obstacles of the preset obstacle type, thereby effectively utilizing the exploration capability of reinforcement learning to enable the model to explore different routes, thereby achieving rapid obstacle avoidance of obstacles of the preset obstacle type.

[0144] Figure 8 is a schematic block diagram of a path planning system 200 provided in an embodiment of the present invention. Figure 8 As shown, corresponding to the above path planning method, the present invention also provides a path planning system 200. Specifically, please refer to Figure 8 The path planning system 200 includes a sweeping machine 210 and a cloud 220 , and the sweeping machine 210 is communicatively connected with the cloud 220 .

[0145] The cloud 220 is used to send the pre-trained model to the sweeping robot;

[0146] The sweeping machine 210 is used to receive the pre-trained model sent by the cloud, and perform model training on the pre-trained model based on the acquired current home environment map data to obtain a deep reinforcement learning model; obtain the obstacle type in the space to be cleaned through the laser radar data; if it is determined that the obstacle type is a preset obstacle type, obtain the deep reinforcement learning model corresponding to the obstacle type; determine the predicted execution action of the sweeping machine based on the deep reinforcement learning model; execute corresponding actions according to the predicted execution action to reach the current target point corresponding to the predicted execution action in the space to be cleaned; if it is determined that the current obstacle type is the preset obstacle type based on the current laser radar data corresponding to the current target point, return to execute the step of determining the predicted execution action of the sweeping machine based on the deep reinforcement learning model until the current obstacle type is not the preset obstacle type.

[0147] The above-mentioned path planning device or path planning system can be implemented in the form of a computer program. The computer program can be used in Fig. 9 The computer device 400 shown is running, and the computer device is a sweeping machine or a terminal cloud system.

[0148] See also Fig. 9 The computer device 400 includes a processor 402 , a memory and a network interface 405 connected via a system bus 401 , wherein the memory may include a storage medium 403 and an internal memory 404 .

[0149] The storage medium 403 may store an operating system 4031 and a computer program 4032. The computer program 4032 includes program instructions, and when the program instructions are executed, the processor 402 may execute the above-mentioned path planning method.

[0150] The processor 402 is used to provide computing and control capabilities to support the operation of the entire computer device.

[0151] The internal memory 404 provides an environment for the operation of the computer program 4032 in the storage medium 403. When the computer program 4032 is executed by the processor 402, the processor 402 can execute the above-mentioned path planning method.

[0152] The network interface 405 is used to communicate with other devices over the network. Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0153] The processor 402 is used to run a computer program 4032 stored in the memory to implement the path planning method as described above.

[0154] It should be understood that in the embodiment of the present invention, the processor 402 may be a central processing unit (CPU), and the processor 402 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0155] It can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0156] Therefore, the present invention also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the path planning method as described above.

[0157] The computer-readable storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which are computer-readable storage media that can store program codes.

[0158] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0159] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0160] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0161] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0162] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or substitutions within the technical scope disclosed by the present invention, and these modifications or substitutions should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A path planning method, applied to a sweeping machine, characterized in that: include: Obtain the type of obstacles in the space to be cleaned through LiDAR data; If it is determined that the obstacle type is a preset obstacle type, obtaining a deep reinforcement learning model corresponding to the obstacle type; Determining a predicted execution action of the sweeper based on the deep reinforcement learning model; Controlling the sweeping machine to perform a corresponding action according to the predicted execution action, so as to reach a current target point corresponding to the predicted execution action in the space to be cleaned; If it is determined based on the current laser radar data corresponding to the current target point that the current obstacle type is the preset obstacle type, then return to the step of determining the predicted execution action of the sweeper based on the deep reinforcement learning model until the current obstacle type is not the preset obstacle type; After the step of controlling the sweeping machine to perform a corresponding action according to the predicted execution action to reach a current target point corresponding to the predicted execution action in the space to be cleaned, the method further includes: Acquire second motion information corresponding to the current target point; Determine a predicted action label value corresponding to the predicted execution action based on a preset motion reward and penalty value determination strategy, first motion information, and second motion information; wherein the first motion information is motion information corresponding to the current starting point of the sweeping machine in the space to be cleaned; A mapping relationship is established between the predicted action label value and the predicted execution action, and the relationship is saved in a model training set, and the model training set is updated; wherein the model training set is used to perform model training on the deep reinforcement learning model when preset model training conditions are met.

2. The path planning method according to claim 1, characterized in that: The step of determining the predicted execution action of the sweeper based on the deep reinforcement learning model includes: Obtaining a first model in the deep reinforcement learning model, inputting the acquired motion state data of the sweeping robot into the first model, and obtaining a model reasoning action; Obtaining a second model in the deep reinforcement learning model, and randomly generating a random execution action based on the second model; The model inference action and the random execution action are fused based on a preset weighted fusion strategy to obtain the predicted execution action.

3. The path planning method according to claim 2, characterized in that: The fusing the model inference action and the random execution action based on a preset weighted fusion strategy to obtain the predicted execution action includes: Obtaining a first weight coefficient corresponding to the model reasoning action in the weighted fusion strategy, and obtaining a second weight coefficient corresponding to the random execution action in the weighted fusion strategy; wherein the sum of the first weight coefficient and the second weight coefficient is equal to a first preset value; Obtain a first multiplication result corresponding to the multiplication of the model reasoning action and the first weight coefficient, obtain a second multiplication result corresponding to the multiplication of the random execution action and the second weight coefficient, and sum the first multiplication result and the second multiplication result to obtain the predicted execution action.

4. The path planning method according to claim 1, characterized in that: After the step of obtaining the obstacle type in the space to be cleaned by using the laser radar data, the method further includes: The current starting point of the sweeping machine in the space to be cleaned is obtained, and first motion information corresponding to the current starting point is obtained.

5. The path planning method according to claim 1, characterized in that: When the current obstacle type is not the preset obstacle type, a current planned path is generated based on a preset path planning algorithm to control the sweeping machine to perform sweeping and cleaning work according to the current planned path.

6. The path planning method according to claim 1, characterized in that: Before the step of obtaining the obstacle type in the space to be cleaned by using laser radar data, the method further includes: Acquire current home environment map data, and add the current home environment map data to a locally constructed historical environment map data set; wherein the historical environment map data set is stored in a local data storage area; Uploading the current home environment map data to the cloud; Acquire the pre-trained model sent by the cloud; wherein the pre-trained model is obtained by training an initial deep reinforcement learning model on the cloud based on the current home environment map data and other home environment map data uploaded by other sweepers; The pre-trained model is trained based on the current home environment map data to obtain the deep reinforcement learning model.

7. A path planning method, applied to a path planning system, characterized in that: The path planning system includes a sweeping machine and a cloud, and the sweeping machine is communicatively connected with the cloud; the path planning method includes: The cloud sends a pre-trained model to the sweeping robot; The sweeping robot receives the pre-trained model sent by the cloud, and performs model training on the pre-trained model based on the acquired current home environment map data to obtain a deep reinforcement learning model; The sweeper obtains the type of obstacles in the space to be cleaned through laser radar data; If the sweeping machine determines that the obstacle type is a preset obstacle type, then acquiring the deep reinforcement learning model corresponding to the obstacle type; The sweeping machine determines a predicted execution action of the sweeping machine based on the deep reinforcement learning model; The sweeping machine performs a corresponding action according to the predicted execution action to reach a current target point corresponding to the predicted execution action in the space to be cleaned; If the sweeping machine determines that the current obstacle type is the preset obstacle type based on the current laser radar data corresponding to the current target point, the sweeping machine returns to the step of determining the predicted execution action of the sweeping machine based on the deep reinforcement learning model until the current obstacle type is not the preset obstacle type; After the sweeping machine performs a corresponding action according to the predicted execution action to reach a current target point corresponding to the predicted execution action in the to-be-cleaned space, the sweeping machine is further configured to: Acquire second motion information corresponding to the current target point; Determine a predicted action label value corresponding to the predicted execution action based on a preset motion reward and penalty value determination strategy, first motion information, and second motion information; wherein the first motion information is motion information corresponding to the current starting point of the sweeping machine in the space to be cleaned; A mapping relationship is established between the predicted action label value and the predicted execution action, and the relationship is saved in a model training set, and the model training set is updated; wherein the model training set is used to perform model training on the deep reinforcement learning model when preset model training conditions are met.

8. The path planning method according to claim 7, characterized in that: Before the step of sending the pre-trained model to the sweeping robot in the cloud, the method further includes: The cloud receives the current home environment map data sent by the sweeping robot; The cloud obtains other home environment map data uploaded by other sweepers and forms a training set with the previous home environment map data; The cloud performs sweeping robot modeling and map modeling on the sweeping robot and the training set respectively according to the current simulation training request, and allocates corresponding current system computing resources to perform model iterative simulation training on the initial deep reinforcement learning model to obtain the pre-trained model.

9. A path planning device, characterized in that: The path planning device is deployed in the sweeping machine, and the path planning device includes: A first obstacle type determination unit, used to obtain the type of obstacles in the space to be cleaned through laser radar data; A model acquisition unit, configured to acquire a deep reinforcement learning model corresponding to the obstacle type if it is determined that the obstacle type is a preset obstacle type; A predicted execution action acquisition unit, used to determine the predicted execution action of the sweeper based on the deep reinforcement learning model; An execution action control unit, used for controlling the sweeping machine to execute a corresponding action according to the predicted execution action, so as to reach a current target point corresponding to the predicted execution action in the space to be cleaned; A second obstacle type determination unit is used for returning to the step of determining the predicted execution action of the sweeper based on the deep reinforcement learning model if the current obstacle type is determined to be the preset obstacle type based on the current laser radar data corresponding to the current target point, until the current obstacle type is not the preset obstacle type; The path planning device also includes: A second motion information acquisition unit, used to acquire second motion information corresponding to the current target point; A predicted action label value acquisition unit, used to determine a predicted action label value corresponding to the predicted execution action based on a preset motion reward and penalty value determination strategy, first motion information, and second motion information; wherein the first motion information is motion information corresponding to the current starting point of the sweeping machine in the space to be cleaned; A model training set updating unit is used to establish a mapping relationship between the predicted action label value and the predicted execution action, save it to the model training set, and update the model training set; wherein, the model training set is used to perform model training on the deep reinforcement learning model when the preset model training conditions are met.

10. A path planning system, characterized in that: It includes a sweeping machine and a cloud, wherein the sweeping machine is communicatively connected with the cloud; The cloud is used to send the pre-trained model to the sweeping robot; The sweeper is used to receive the pre-trained model sent by the cloud, and perform model training on the pre-trained model based on the acquired current home environment map data to obtain a deep reinforcement learning model; and obtain the obstacle type in the space to be cleaned through the laser radar data; If it is determined that the obstacle type is a preset obstacle type, the deep reinforcement learning model corresponding to the obstacle type is obtained; the predicted execution action of the sweeper is determined based on the deep reinforcement learning model; the corresponding action is executed according to the predicted execution action to reach the current target point corresponding to the predicted execution action in the space to be cleaned; if it is determined that the current obstacle type is the preset obstacle type based on the current laser radar data corresponding to the current target point, the step of determining the predicted execution action of the sweeper based on the deep reinforcement learning model is returned to execute until the current obstacle type is not the preset obstacle type; After performing a corresponding action according to the predicted execution action to reach a current target point corresponding to the predicted execution action in the to-be-cleaned space, the sweeping machine is further used for: Acquire second motion information corresponding to the current target point; Determine a predicted action label value corresponding to the predicted execution action based on a preset motion reward and penalty value determination strategy, first motion information, and second motion information; wherein the first motion information is motion information corresponding to the current starting point of the sweeping machine in the space to be cleaned; A mapping relationship is established between the predicted action label value and the predicted execution action, and the relationship is saved in a model training set, and the model training set is updated; wherein the model training set is used to perform model training on the deep reinforcement learning model when preset model training conditions are met.

Images