Path planning method and device for water cleaning robot coordinated by UAV

Abstract

The present invention provides a method and device for path planning of a water cleaning robot in cooperation with a drone, including: the drone first cruises over the river, records the moving path, uses the U‑Net semantic segmentation network to realize the division of the water surface and the road surface, and marks them; uses a convolutional neural network to identify garbage pollutants and then transmits their position information to the water cleaning robot, the water cleaning robot obtains the position information, uses the A* algorithm to calculate the distance between target points, establishes a cost matrix, and uses dynamic planning to design the path to achieve the goal of the shortest movement path and recycling the corresponding garbage; the water cleaning robot combines the image recognition system to perform dynamic obstacle avoidance design. The present invention effectively solves the problems of limited cleaning range, low cleaning efficiency, and high energy consumption in the cleaning process of intelligent water cleaning robots during autonomous navigation.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of mobile robots, and in particular to a method and device for path planning of an aquatic cleaning robot coordinated with an unmanned aerial vehicle. Background Art

[0002] In the field of mobile robots, path planning is an important link. For intelligent water cleaning robots, the requirements for path planning should be: able to avoid water obstacles and ensure the safety and stability of the cleaning robot during driving; able to efficiently and thoroughly perform water cleaning in the area. Due to the higher complexity of water surface cleaning, the path planning of water surface cleaning robots often has problems such as insufficient cleaning efficiency, low cleaning range, and insufficient intelligence. Therefore, the development of a path planning method and equipment for water cleaning robots coordinated with drones can effectively overcome the defects in the above-mentioned related technologies, which has become a technical problem that needs to be solved urgently in the industry. Summary of the invention

[0003] In view of the above-mentioned problems existing in the prior art, an embodiment of the present invention provides a method and device for path planning of an aquatic cleaning robot in cooperation with a drone.

[0004] In the first aspect, an embodiment of the present invention provides a method for path planning of a water cleaning robot in collaboration with a drone, comprising: an image recognition system based on a convolutional neural network for the drone, a path planning system based on a genetic algorithm for the drone, a vision system for the water cleaning robot, a communication system between the drone and the water cleaning robot, and a path planning system based on an A* algorithm for the water cleaning robot; the drone first cruises over the river, records the moving path, and uses a U-Net semantic segmentation network to divide the water surface and the road surface, and marks them out; after using a convolutional neural network to identify garbage pollutants, its position information is transmitted to the water cleaning robot, the water cleaning robot obtains the position information, uses an A* algorithm to calculate the distance between target points, establishes a cost matrix, and uses dynamic planning to design a path to achieve the goal of shortest motion path and recycling corresponding garbage; the water cleaning robot combines the image recognition system for dynamic obstacle avoidance design.

[0005] Based on the content of the above method embodiment, the UAV-coordinated water cleaning robot path planning method provided in the embodiment of the present invention uses the U-Net semantic segmentation network to realize the division of water surface and road surface, and mark them out, including: Step 1: Five preliminary effective network feature layers can be obtained by using the backbone extraction network, and the characteristics of water surface and road surface targets are obtained; Step 2: U-Net enters the stage of enhanced feature extraction, and uses the upsampling function of the network to obtain the total effective feature layer after feature fusion; Step 3: After feature recognition in steps 1 and 2, the network will use the results of feature recognition for classification, which is equivalent to classifying each pixel point; Step 4: Build a U-Net network, program it according to its composition principle, and use Opencv to implement image input and detection input.

[0006] Based on the contents of the above-mentioned method embodiments, a method for path planning of a water cleaning robot in collaboration with a drone is provided in an embodiment of the present invention. After using a convolutional neural network to identify garbage pollutants, the location information thereof is transmitted to the water cleaning robot, and the water cleaning robot obtains the location information. The method includes: using a convolutional neural network to create a target recognition framework based on the acquired research object, and the main framework uses a YoloV4 target detection network to design a recognition algorithm based on the DarkNet network main body to further obtain the scene information of the target water area.

[0007] Based on the contents of the above method embodiments, a method for path planning of a water cleaning robot coordinated by a drone is provided in an embodiment of the present invention, wherein the A* algorithm is used to calculate the distance between target points, a cost matrix is ​​established, and a path design using dynamic programming is used to achieve the goal of the shortest motion path and recycling the corresponding garbage, including: Step 1: The distances between the pre-processed garbage target points identified by the drone, and the distances between the starting point and each target point are calculated using the A* algorithm; Step 2: Several target points closest to the water surface cleaning robot in a straight line are selected as the objects of this cleaning, and then a cost matrix is ​​established to plan the path and a dynamic programming function is established; Step 3: The water surface cleaning robot performs dynamic obstacle avoidance design in combination with an image recognition system.

[0008] Based on the content of the above method embodiment, the path planning method for the water cleaning robot coordinated by a drone provided in the embodiment of the present invention, wherein the establishment of the dynamic planning function comprises:

[0009] d(i,V')=min{cik+d(k,V-{k})}(k∈V')

[0010] d(k,{})=cki(k≠i)

[0011] Where d represents the distance from the starting point i, passing through each target point once, only once, and finally returning to i; V is the set of all points; V' represents the set of passed points; cik is the distance from point i to point k; min is the minimum value; d(k,{})=cki is the last step back to the origin.

[0012] In the second aspect, an embodiment of the present invention provides a path planning device for a water cleaning robot coordinated by a drone, comprising: a first main module, an image recognition system based on a convolutional neural network for the drone, a path planning system based on a genetic algorithm for the drone, a visual system for the water cleaning robot, a communication system between the drone and the water cleaning robot, and a path planning system based on an A* algorithm for the water cleaning robot; a second main module, for the drone to cruise over the river first, record the moving path, and use a U-Net semantic segmentation network to divide the water surface and the road surface, and mark them out; a third main module, for using a convolutional neural network to identify garbage pollutants and then transmit their position information to the water cleaning robot, the water cleaning robot obtains the position information, uses the A* algorithm to calculate the distance between target points, establishes a cost matrix, and uses dynamic planning to design a path to achieve the goal of shortest motion path and recycling corresponding garbage; a fourth main module, for the water cleaning robot to perform dynamic obstacle avoidance design in combination with an image recognition system.

[0013] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0014] at least one processor; and

[0015] at least one memory in communication with the processor, wherein:

[0016] The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the path planning method for the water cleaning robot coordinated by a drone provided in any of the various implementation methods of the first aspect.

[0017] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions enable a computer to execute a drone-coordinated water cleaning robot path planning method provided in any one of the various implementation methods of the first aspect.

[0018] The UAV-coordinated water cleaning robot path planning method and device provided in the embodiments of the present invention records the moving path and uses the U-Net semantic segmentation network to realize the division of the water surface and the road surface. The water cleaning robot obtains the position information and uses the dynamic planning path design to achieve the goal of shortest movement path and recycling corresponding garbage, which effectively solves the problems of limited cleaning range, low cleaning efficiency and high energy consumption of the intelligent water cleaning robot during autonomous navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A flow chart of a method for planning a path for a water cleaning robot in collaboration with a drone provided in an embodiment of the present invention;

[0021] Figure 2 A schematic diagram of the structure of a path planning device for a water cleaning robot coordinated by a drone provided in an embodiment of the present invention;

[0022] Figure 3 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention;

[0023] Figure 4 A schematic diagram of the image prediction result provided by an embodiment of the present invention;

[0024] Figure 5 A flow chart of visual SLAM modeling provided by an embodiment of the present invention;

[0025] Figure 6 A schematic diagram of the simulation effect of three-dimensional path planning using a genetic algorithm provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention can be arbitrarily combined with each other to form a feasible technical solution. This combination is not subject to the constraints of the sequence of steps and / or the structural composition mode, but must be based on the ability of ordinary technicians in this field to achieve. When the combination of technical solutions is contradictory or cannot be achieved, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0027] The embodiment of the present invention provides a method for planning a path for a water cleaning robot in cooperation with a drone, see Figure 1 The method includes: an image recognition system based on a convolutional neural network for the UAV, a path planning system based on a genetic algorithm for the UAV, a visual system for a water cleaning robot, a communication system between the UAV and the water cleaning robot, and a path planning system based on an A* algorithm for the water cleaning robot; the UAV first cruises over the river, records the moving path, and uses a U-Net semantic segmentation network to divide the water surface and the road surface and mark them; the convolutional neural network is used to identify garbage pollutants and then transmit their position information to the water cleaning robot. The water cleaning robot obtains the position information, uses the A* algorithm to calculate the distance between target points, establishes a cost matrix, and uses dynamic planning to design a path to achieve the goal of shortest motion path and recycling corresponding garbage; the water cleaning robot is designed to avoid obstacles dynamically in combination with the image recognition system.

[0028] Based on the content of the above method embodiment, as an optional embodiment, the drone-coordinated water cleaning robot path planning method provided in the embodiment of the present invention, which uses the U-Net semantic segmentation network to realize the division of the water surface and the road surface, and mark them out, includes: Step 1: Five preliminary effective network feature layers can be obtained by using the backbone extraction network, and the characteristics of the water surface and road surface targets are obtained; Step 2: U-Net enters the stage of enhanced feature extraction, and uses the upsampling function of the network to obtain the total effective feature layer after feature fusion; Step 3: After the feature recognition of steps 1 and 2, the network will use the results of feature recognition for classification, which is equivalent to classifying each pixel point; Step 4: Build a U-Net network, program it according to its composition principle, and use Opencv to implement image input and detection input.

[0029] Based on the content of the above method embodiment, as an optional embodiment, the drone-coordinated water cleaning robot path planning method provided in the embodiment of the present invention, which uses a convolutional neural network to identify garbage pollutants and then transmits their location information to the water cleaning robot, and the water cleaning robot obtains the location information, includes: using a convolutional neural network to create a target recognition framework based on the acquired research object, the main framework uses the YoloV4 target detection network, and designs a recognition algorithm based on the DarkNet network body to further obtain the scene information of the target water area.

[0030] Based on the content of the above method embodiment, as an optional embodiment, the drone-coordinated water cleaning robot path planning method provided in the embodiment of the present invention uses the A* algorithm to calculate the distance between target points, establishes a cost matrix, and uses dynamic programming to design the path to achieve the goal of shortest motion path and recycling corresponding garbage, including: Step 1: Calculate the distance between the pre-processed garbage target points identified by the drone, the distance between the starting point and each target point using the A* algorithm; Step 2: Select several target points that are closest to the water surface cleaning robot in a straight line as the objects of this cleaning, and then establish a cost matrix, plan the path, and establish a dynamic programming function; Step 3: The water surface cleaning robot performs dynamic obstacle avoidance design in combination with the image recognition system.

[0031] Based on the content of the above method embodiment, as an optional embodiment, the path planning method for the water cleaning robot coordinated by a drone provided in the embodiment of the present invention, wherein the establishment of the dynamic planning function includes:

[0032] d(i,V')=min{cik+d(k,V-{k})}(k∈V') (1)

[0033] d(k,{})=cki(k≠i) (2)

[0034] Where d represents the distance from the starting point i, passing through each target point once, only once, and finally returning to i; V is the set of all points; V' represents the set of passed points; cik is the distance from point i to point k; min is the minimum value; d(k,{})=cki is the last step back to the origin.

[0035] The method for planning a path for a water cleaning robot in collaboration with a drone provided in an embodiment of the present invention records the moving path and uses a U-Net semantic segmentation network to divide the water surface and the road surface. The water cleaning robot obtains position information and uses dynamic planning to design a path to achieve the goal of shortest movement path and recycling corresponding garbage. This method effectively solves the problems of limited cleaning range, low cleaning efficiency, and high energy consumption of the intelligent water cleaning robot during autonomous navigation.

[0036] In another embodiment, it is a genetic algorithm drone path planning process, and the specific process is as follows:

[0037] (1) Establishing a theoretical model

[0038] The drone uses vision to mark coordinate information such as road signs and updates the map status at any time based on the GPS data it obtains and the sensor's perception of the surrounding environment. The path planning needs to be iteratively updated in a dynamic scene, and a three-dimensional space path solution based on a genetic algorithm is used to solve the path planning problem in a dynamic environment.

[0039] (2) Design algorithm

[0040] The genetic algorithm represents the possible solutions of the optimization function as an individual. Each individual forms a gene in a certain coding method. With the help of genetic operators, selection, crossover, and mutation operations, the population is evolved to select the population that is more adaptable to the environment. In path planning, each path is planned as an individual. Each population has n individuals, that is, there are n paths. At the same time, each individual has m chromosomes, that is, the number of intermediate transition points. Each point (chromosome) has two dimensions (x, y). In the code, genx and geny are used to represent a population. Through the evolution of each generation, the genetic operator operation is performed on the population to select the appropriate individual (optimal path). Genetic algorithm three-dimensional path planning simulation is as follows: Figure 6 shown.

[0041] In another embodiment, when the drone is identifying pollutants, the visual part mainly uses vision to cyclically implement image segmentation and image recognition in the target area, that is, using visual algorithms to identify and process the external environment under a high-definition camera equipped with a gimbal.

[0042] (1) U-Net image segmentation

[0043] Based on the demand for image segmentation training on the water surface, the U-Net semantic segmentation network is used to divide the water surface and the road surface and mark them out. U-Net is mainly divided into three parts. The first part is the backbone feature extraction network, which is mainly composed of the superposition of convolutional layers and pooling layers. Five preliminary effective network feature layers can be obtained using the backbone extraction network. After obtaining the features of the water surface and road targets, U-Net enters the second stage, which is the stage of enhanced feature extraction. The main function of this stage is to use the upsampling function of the network to obtain the total effective feature layer after feature fusion. The third stage is the prediction part. After the previous feature recognition, the network will use the results of feature recognition for classification, which is equivalent to classifying each pixel.

[0044] The target features are firstly produced using the surface navigation dataset. The video is first divided into 3,000 frames. Labels are then used to annotate and label them. After completion, the U-Net network is built. The construction of the U-Net network is mainly programmed according to its composition principle, and Opencv is used to realize image input, detection input, and image display.

[0045] (2) Convolutional Neural Network Image Recognition

[0046] Based on the acquisition of the research object, the target recognition framework is created. The relevant main framework uses the YoloV4 target detection network. YoloV4 is mainly composed of three parts: backbone extraction network, feature pyramid, and classification regression layer. Its Backbone is composed of CSPDarknet53 backbone network, and the feature pyramid part includes SPP and PANet networks. The Yolohead part is mainly used for prediction

[0047] In order to further obtain the scene information of the target water area, a recognition algorithm based on the DarkNet network was designed. The drone uses a high-altitude camera to identify the target water area and obtain the environmental status in the field of view by marking the target object. The algorithm mainly identifies garbage, algal blooms and natural buildings.

[0048] The pollutant data set in the waterway is formed by combining the data obtained after preprocessing some low-quality images with the data in the original images. The training input image size is 608*608, and there are 3510 valid frames. Before training, the ratio of the training set to the test set is set to 9:1, and the batch size of each round of iteration is set to 2. Each round needs to complete 1580 batches, and the total number of iterations is 50. The training of the target is mainly divided into two parts, the training preparation stage and the training process.

[0049] During the training process, the training of the convolutional neural network mainly involves iterative operations and updates of its convolutional layer, pooling layer, and fully connected layer. The prediction results are as follows: Figure 4 .

[0050] In another embodiment, after the drone transmits the signal to the cleaning robot, the robot is automatically driven through 3D reconstruction. The ORB-SLAM2 mapping algorithm is used in the algorithm, and the position and depth perception of the surrounding scenery is realized through the binocular camera. In terms of the algorithm, the robot mapping needs to be sufficiently accurate because obstacles encountered in the path need to be considered. The project team intends to use radar-assisted mapping and obtain surrounding point cloud data through laser radar. The process is as follows Figure 5 shown.

[0051] In another embodiment, the water surface cleaning robot needs to plan its own movement path after obtaining map data and garbage location to achieve the purpose of cleaning garbage. In view of the characteristics of the water surface cleaning robot that needs to clean garbage in multiple locations and return to the starting point, this project team analyzed and used the A* algorithm to calculate the distance between target points, established a cost matrix, and used dynamic programming to design the path to achieve the goal of the shortest movement path and recover the corresponding garbage.

[0052] Building a theoretical model

[0053] The solution starts from the path planning problem based on assumptions. The drone uses the vision to mark the coordinate information such as road signs. In unknown terrain, the map status is updated at any time based on the GPS data obtained by the drone and the sensor's perception data of the surrounding environment. The robot goes to the target point to clean the garbage according to the planned movement path. As the time complexity of the dynamic planning method increases with the number of target points, and considering that the number of garbage recovered by the surface cleaning robot in a single cleaning is also limited, the number of single cleaning target points is set to be at most n, so as to reduce the time complexity of the algorithm and improve the cleaning efficiency.

[0054] Algorithm Design

[0055] First, the distances between the pre-processed garbage target points identified by the drone, and the distances between the starting point and each target point are calculated using the A* algorithm. Assuming that the distance between any two points AB is calculated on a grid map, the A* algorithm needs to be used for valuation using the valuation formula

[0056] F=G+H

[0057] G is the cost of moving from the starting point A to the specified square, along the path generated to reach that square. We agree that the cost of a straight move is 10, and the cost of a diagonal move is 14. (The actual diagonal move distance is the square root of 2, or approximately 1.414 times the horizontal or vertical move cost). H is the estimated cost of moving from the specified square to the end point B. Calculate the number of squares you need to move horizontally or vertically from the current square to the target square, ignoring diagonal moves, and then multiply the total by 10.

[0058] Mark FGH at the upper left, lower left and lower right positions in a square so that you can observe the results of each valuation.

[0059] The agreed order of coordinate access and parent node search is: right, upper right, upper, upper left, left, lower left, lower, lower right. The direction of increase along the X axis is right, and the direction of increase along the Y axis is up. There may be multiple parent nodes. If obstacles are inaccessible, the one with the lowest cost and the last search is selected as the parent node.

[0060] After calculating all the distance data, select at most n target points that are closest to the water surface cleaning robot in a straight line as the objects of this cleaning. Then establish the cost matrix, plan the path, and establish the dynamic programming function as shown in equations (1) and (2).

[0061] The implementation basis of each embodiment of the present invention is to implement programmed processing through a device with a processor function. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present invention can be encapsulated into various modules. Based on this reality, on the basis of the above embodiments, an embodiment of the present invention provides a drone-coordinated water cleaning robot path planning device, which is used to execute the drone-coordinated water cleaning robot path planning method in the above method embodiment. Figure 2 The device includes: a first main module, which is used for an image recognition system based on a convolutional neural network for the UAV, a path planning system based on a genetic algorithm for the UAV, a visual system for a water cleaning robot, a communication system between the UAV and the water cleaning robot, and a path planning system based on an A* algorithm for the water cleaning robot; a second main module, which is used for the UAV to cruise over the river first, record the moving path, and use a U-Net semantic segmentation network to divide the water surface and the road surface, and mark them out; a third main module, which is used to use a convolutional neural network to identify garbage pollutants and then transmit their position information to the water cleaning robot. The water cleaning robot obtains the position information, uses the A* algorithm to calculate the distance between target points, establishes a cost matrix, and uses dynamic planning to design a path to achieve the goal of the shortest motion path and recovering the corresponding garbage; a fourth main module, which is used for the water cleaning robot to perform dynamic obstacle avoidance design in combination with an image recognition system.

[0062] The path planning device for a water cleaning robot coordinated by a drone provided in an embodiment of the present invention adopts Figure 2 Several modules in the system record the moving path and use the U-Net semantic segmentation network to divide the water surface and the road surface. The water cleaning robot obtains the position information and uses dynamic planning to design the path to achieve the shortest movement path and recycle the corresponding garbage. This effectively solves the problems of limited cleaning range, low cleaning efficiency and high energy consumption of the intelligent water cleaning robot during autonomous navigation.

[0063] It should be noted that the device in the device embodiment provided by the present invention can be used to implement the method in the above method embodiment as well as the method in other method embodiments provided by the present invention. The only difference is that the corresponding functional modules are set, and the principle is basically the same as the principle of the above device embodiment provided by the present invention. As long as the technical personnel in the field refer to the specific technical solutions in other method embodiments on the basis of the above device embodiment, obtain the corresponding technical means and the technical solutions composed of these technical means by combining technical features, the device in the above device embodiment can be improved on the premise of ensuring the practicality of the technical solution, thereby obtaining the corresponding device class embodiment, which is used to implement the methods in other method class embodiments. For example:

[0064] Based on the content of the above-mentioned device embodiment, as an optional embodiment, the drone-coordinated water cleaning robot path planning device provided in the embodiment of the present invention also includes: a first sub-module, used to realize the division of water surface and road surface by using the U-Net semantic segmentation network, and mark them out, including: step one: five preliminary effective network feature layers can be obtained by using the backbone extraction network, and the characteristics of water surface and road surface targets are obtained; step two: U-Net enters the stage of enhanced feature extraction, and uses the upsampling function of the network to obtain the total effective feature layer after feature fusion; step three: after the feature recognition in steps one and two, the network will use the results of feature recognition for classification, which is equivalent to classifying each pixel point; step four: build a U-Net network, program it according to its composition principle, and use Opencv to implement image input and detection input.

[0065] Based on the content of the above-mentioned device embodiment, as an optional embodiment, the drone-coordinated water cleaning robot path planning device provided in the embodiment of the present invention also includes: a second sub-module, used to implement the use of a convolutional neural network to identify garbage pollutants and then transmit their location information to the water cleaning robot, and the water cleaning robot obtains the location information, including: using a convolutional neural network to create a target recognition framework based on the acquired research object, the main framework uses the YoloV4 target detection network, and a recognition algorithm based on the DarkNet network body is designed to further obtain the scene information of the target water area.

[0066] Based on the contents of the above-mentioned device embodiments, as an optional embodiment, the drone-coordinated water cleaning robot path planning device provided in the embodiments of the present invention also includes: a third sub-module, which is used to implement the use of the A* algorithm to calculate the distance between target points, establish a cost matrix, and use dynamic programming to design a path to achieve the shortest motion path and recycle the corresponding garbage, including: Step 1: Calculate the distance between the pre-processed garbage target points identified by the drone, the distance between the starting point and each target point using the A* algorithm; Step 2: Select several target points that are closest to the water surface cleaning robot in a straight line as the objects of this cleaning, and then establish a cost matrix, plan the path, and establish a dynamic programming function; Step 3: The water surface cleaning robot performs dynamic obstacle avoidance design in combination with the image recognition system.

[0067] Based on the content of the above device embodiment, as an optional embodiment, the drone-coordinated water cleaning robot path planning device provided in the embodiment of the present invention further includes: a fourth submodule, which is used to implement the establishment of the dynamic planning function, including:

[0068] d(i,V')=min{cik+d(k,V-{k})}(k∈V')

[0069] d(k,{})=cki(k≠i)

[0070] Where d represents the distance from the starting point i, passing through each target point once, only once, and finally returning to i; V is the set of all points; V' represents the set of passed points; cik is the distance from point i to point k; min is the minimum value; d(k,{})=cki is the last step back to the origin.

[0071] The method of the embodiment of the present invention is implemented by relying on electronic devices, so it is necessary to introduce the relevant electronic devices. Based on this purpose, the embodiment of the present invention provides an electronic device, such as Figure 3 As shown, the electronic device includes: at least one processor, a communication interface, at least one memory, and a communication bus, wherein at least one processor, the communication interface, and at least one memory communicate with each other through the communication bus. At least one processor can call the logic instructions in at least one memory to execute all or part of the steps of the method provided by the aforementioned various method embodiments.

[0072] In addition, the logic instructions in the at least one memory mentioned above can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each method embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0073] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0074] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0075] The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. Based on this understanding, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and sometimes in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0076] In this patent, the terms "include", "comprises" or any other variation thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "includes..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for path planning of a water cleaning robot in cooperation with a drone, characterized in that: include: The UAV has an image recognition system based on convolutional neural networks, a path planning system based on genetic algorithms, a water cleaning robot vision system, a communication system between the UAV and the water cleaning robot, and a path planning system based on the A* algorithm for the water cleaning robot. The UAV first cruises over the river, records the moving path, and uses the U-Net semantic segmentation network to divide the water surface and the road surface and mark them. The convolutional neural network is used to identify garbage pollutants and then transmit their location information to the water cleaning robot. The water cleaning robot obtains the location information, uses the A* algorithm to calculate the distance between the target points, establishes a cost matrix, and uses dynamic programming to design the path to achieve the shortest movement path and recycle the corresponding garbage. The water cleaning robot is designed to avoid obstacles dynamically in combination with the image recognition system. The method adopts the A* algorithm to calculate the distance between target points, establishes a cost matrix, and uses dynamic programming to design a path to achieve the goal of the shortest motion path and recycling the corresponding garbage, including: step 1: using the A* algorithm to calculate the distance between the pre-processed garbage target points identified by the drone, and the distance between the starting point and each target point; step 2: selecting several target points closest to the water surface cleaning robot in a straight line as the objects of this cleaning, and then establishing a cost matrix, planning the path, and establishing a dynamic programming function; step 3: the water surface cleaning robot combines the image recognition system to perform dynamic obstacle avoidance design; The step of establishing a dynamic programming function comprises: d(i,V' )=min{cik+d(k,V-{k})} k∈V' d(k,{})=cki k≠i Among them, d represents the distance from the starting point i, passing through each target point once, only once, and finally returning to i; V is the set of all points; V' represents the set of passing points; cik is the distance from point i to point k; min is the minimum value; d(k,{})=cki is the last step back to the origin.

2. The method for path planning of a water cleaning robot coordinated by a drone according to claim 1, characterized in that: The method of using the U-Net semantic segmentation network to divide the water surface and the road surface and mark them includes: step 1: using the backbone extraction network to obtain five preliminary effective network feature layers, after obtaining the characteristics of the water surface and the road surface targets; step 2: U-Net enters the stage of enhanced feature extraction, and uses the upsampling function of the network to obtain the total effective feature layer after feature fusion; step 3: after feature recognition in steps 1 and 2, the network will use the results of feature recognition for classification, which is equivalent to classifying each pixel point; step 4: build a U-Net network, program it according to its composition principle, and use Opencv to implement image input and detection input.

3. The method for path planning of a water cleaning robot coordinated by a drone according to claim 2, characterized in that: The method adopts a convolutional neural network to identify garbage pollutants and transmits their position information to an aquatic cleaning robot, and the aquatic cleaning robot obtains the position information, including: adopting a convolutional neural network, creating a target recognition framework based on the acquired research object, the main framework adopts the YoloV4 target detection network, and designing a recognition algorithm based on the DarkNet network main body to further obtain the scene information of the target water area.

4. A drone-coordinated water cleaning robot path planning device, the drone-coordinated water cleaning robot path planning device is used to implement the drone-coordinated water cleaning robot path planning method of claim 1, characterized in that: include: The first main module is used for the UAV's image recognition system based on convolutional neural networks, the UAV's path planning system based on genetic algorithms, the water cleaning robot's visual system, the UAV and water cleaning robot's communication system, and the water cleaning robot's path planning system based on the A* algorithm; the second main module is used for the UAV to cruise over the river first, record the moving path, and use the U-Net semantic segmentation network to divide the water surface and the road surface and mark them out; the third main module is used to use convolutional neural networks to identify garbage pollutants and then pass their location information to the water cleaning robot. The water cleaning robot obtains the location information, uses the A* algorithm to calculate the distance between target points, establishes a cost matrix, and uses dynamic planning to design a path to achieve the goal of shortest motion path and recovering the corresponding garbage; the fourth main module is used for the water cleaning robot to combine the image recognition system for dynamic obstacle avoidance design.

5. An electronic device, characterized in that: include: At least one processor, at least one memory and a communication interface; wherein, The processor, memory and communication interface communicate with each other; The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the method according to any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, which cause the computer to execute the method of any one of claims 1 to 3.

Images