Industrial inspection transfer robot navigation method and device based on multi-sensor fusion

Abstract

The invention discloses an industrial inspection transfer robot navigation method and device based on multi-sensor fusion, and relates to the technical field of industrial inspection transfer. The method comprises the following steps: acquiring multivariate sensing data; unifying the multivariate perception data to a common framework to carry out consistency characterization and processing, and constructing a multi-level map; performing semantic segmentation on a semantic layer in the multi-level map based on a semantic segmentation technology of deep learning to obtain a semantic segmentation result; and according to a semantic segmentation result, an intelligent motion planning technology based on reinforcement learning is combined with internally excited reward function learning, a collision-free shortest path in a complex dynamic environment is planned, and long-term autonomous navigation is realized. According to the invention, the adaptive capacity of the inspection transfer robot for large-range complex and diversified industrial factory scenes can be improved, reliable three-dimensional scene modeling and reconstruction, long-term navigation and autonomous scene understanding are realized, and finally, full-autonomous operation of the industrial inspection transfer robot is realized.

Description

Technical Field

[0001] The present application relates to the technical field of industrial inspection and handling, and in particular to a navigation method and device for an industrial inspection and handling robot based on multi-sensor fusion. Background Art

[0002] With the continuous advancement of industrial automation and intelligence, the traditional manual inspection and handling mode can no longer meet the needs of modern industrial production. Manual inspection and handling has many drawbacks such as low efficiency, high cost, potential safety hazards, and difficult data management. There is an urgent need for an automated, efficient, accurate, and safe solution.

[0003] Industrial inspection and handling robots are powerful tools to overcome the drawbacks of the traditional manual mode. They have advantages such as automation, high efficiency, accuracy, and safety, and can effectively improve production efficiency and reduce operating costs. However, as the application environment of the robot system expands from a quasi-structured artificial scenario to a large-scale outdoor unstructured scenario, and the working time develops from a limited working cycle to long-term autonomous operation, it faces complex scenario adaptability problems and long-term autonomy problems.

[0004] Currently, when industrial inspection and handling robots face complex and diverse industrial plant scenarios, they lack reliable three-dimensional scene modeling and reconstruction capabilities, are difficult to achieve long-term navigation and autonomous scene understanding, cannot get rid of the dependence on remote operation, and cannot achieve fully autonomous operation. Therefore, there is an urgent need for new technical means to improve the long-term navigation and autonomous scene understanding capabilities of inspection and handling robots and achieve fully autonomous operation of industrial inspection and handling robots. Summary of the Invention

[0005] The purpose of the present application is to provide a navigation method and device for an industrial inspection and handling robot based on multi-sensor fusion, which can improve the adaptability of the inspection and handling robot to large-scale complex and diverse industrial plant scenarios, as well as the long-term navigation and autonomous scene understanding capabilities, and ultimately achieve fully autonomous operation of the industrial inspection and handling robot.

[0006] To achieve the above purpose, the present application provides the following solutions:

[0007] In the first aspect, the present application provides a navigation method for an industrial inspection and handling robot based on multi-sensor fusion, and the navigation method for the industrial inspection and handling robot based on multi-sensor fusion includes:

[0008] Obtain multi-source perception data.

[0009] Unify the multi-source perception data into a common framework for consistent representation and processing, and construct a multi-level map; the common framework is a framework obtained based on the consistent description of the multi-source perception data at different levels.

[0010] Using semantic segmentation technology based on deep learning, perform semantic segmentation on the semantic layer in the multi-level map to obtain a semantic segmentation result; the semantic segmentation result is used to achieve the understanding of complex dynamic scenes.

[0011] According to the semantic segmentation result, using intelligent motion planning technology based on reinforcement learning, combined with the learning of the reward function of intrinsic motivation, plan a collision-free shortest path in a complex dynamic environment to achieve long-term autonomous navigation; the intelligent motion planning technology based on reinforcement learning refers to planning a collision-free shortest path to the target position through the self-observation of the inspection and handling robot in a complex dynamic environment; the learning of the reward function of intrinsic motivation is used to encourage the inspection and handling robot to explore unknown areas.

[0012] In a second aspect, the present application provides an industrial inspection and handling robot navigation device based on multi-sensor fusion, and the industrial inspection and handling robot navigation device based on multi-sensor fusion includes:

[0013] A data acquisition unit for acquiring multi-source perception data.

[0014] A consistency representation and processing unit for unifying the multi-source perception data into a common framework for consistency representation and processing to construct a multi-level map; the common framework is a framework obtained based on the consistency description of the multi-source perception data at different levels.

[0015] A semantic segmentation unit for performing semantic segmentation on the semantic layer in the multi-level map using semantic segmentation technology based on deep learning to obtain a semantic segmentation result; the semantic segmentation result is used to achieve the understanding of complex dynamic scenes.

[0016] A path planning unit for planning a collision-free shortest path in a complex dynamic environment according to the semantic segmentation result, using intelligent motion planning technology based on reinforcement learning, combined with the learning of the reward function of intrinsic motivation, to achieve long-term autonomous navigation; the intelligent motion planning technology based on reinforcement learning refers to planning a collision-free shortest path to the target position through the self-observation of the inspection and handling robot in a complex dynamic environment; the learning of the reward function of intrinsic motivation is used to encourage the inspection and handling robot to explore unknown areas.

[0017] In a third aspect, the present application provides a computer device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the computer program to implement the industrial inspection and handling robot navigation method based on multi-sensor fusion described in any one of the above.

[0018] Fourthly, the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the navigation method for an industrial inspection and handling robot based on multi-sensor fusion described in any one of the above.

[0019] Fifthly, the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the navigation method for an industrial inspection and handling robot based on multi-sensor fusion described in any one of the above.

[0020] According to the specific embodiments provided by the present application, the following technical effects are disclosed in the present application:

[0021] The present application provides a navigation method and device for an industrial inspection and handling robot based on multi-sensor fusion. The method includes: acquiring multi-source perception data; unifying the multi-source perception data into a common framework for consistency representation and processing, and constructing a multi-level map; based on the semantic segmentation technology of deep learning, performing semantic segmentation on the semantic layer in the multi-level map to obtain a semantic segmentation result; according to the semantic segmentation result, based on the intelligent motion planning technology of reinforcement learning, combined with the learning of the reward function of intrinsic motivation, planning a collision-free shortest path in a complex dynamic environment to achieve long-term autonomous navigation. Through the multi-level map construction technology, the present application can quickly switch between different map levels and perform data association according to the specific application needs of the robot, realizing long-term navigation and autonomous scene understanding, and improving the adaptability of the inspection and handling robot to large-scale, complex and diverse industrial plant scenarios; at the same time, combined with the semantic segmentation technology of deep learning, the intelligent motion planning technology of reinforcement learning and the learning of the reward function of intrinsic motivation, the understanding ability of complex dynamic scenes is further improved, and the long-term autonomous navigation of the inspection and handling robot is realized. Description of the Drawings

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required to be used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0023] Figure 1 It is an application environment diagram of a navigation method for an industrial inspection and handling robot based on multi-sensor fusion in an embodiment of the present application.

[0024] Figure 2 It is a flowchart of a navigation method for an industrial inspection and handling robot based on multi-sensor fusion provided by an embodiment of the present application.

[0025] Figure 3Schematic diagram of the system framework of a multi-level environmental map taking 3D point cloud data as an example and the association relationship between maps at each level provided by an embodiment of the present application.

[0026] Figure 4 Schematic diagram of relocalization within the topological node sub-map provided by an embodiment of the present application.

[0027] Figure 5 Schematic diagram of learning the reward function based on intrinsic motivation provided by an embodiment of the present application.

[0028] Figure 6 Schematic diagram of the functional modules of an industrial inspection and handling robot navigation device based on multi-sensor fusion provided by an embodiment of the present application.

[0029] Figure 7 Schematic diagram of the structure of a computer device provided by an embodiment of the present application. Detailed implementation manners

[0030] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0031] To make the above objects, features, and advantages of the present application more obvious and understandable, the present application will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0032] The prior art has the following disadvantages:

[0033] 1. Existing industrial inspection and handling robots lack reliable 3D scene modeling and reconstruction capabilities and are difficult to adapt to the large-scale, complex, and diverse industrial plant environments.

[0034] 2. The prior art cannot effectively implement the long-term autonomous navigation and autonomous scene understanding functions of industrial inspection and handling robots.

[0035] 3. Existing industrial inspection and handling robots cannot get rid of the dependence on remote control operations and cannot achieve fully autonomous operation.

[0036] 4. Existing industrial inspection and handling robots lack effective planning algorithms in complex dynamic environments and are difficult to achieve collision-free shortest path navigation.

[0037] 5. Existing industrial inspection and handling robots have limited sensing capabilities and lack the ability to deeply understand complex dynamic scenes.

[0038] 6. In the prior art, it is difficult to achieve reliable three-dimensional scene modeling and reconstruction, long-term autonomous navigation, and autonomous scene understanding of industrial inspection and handling robots in a large-scale, complex, and diverse industrial plant environment, and it is impossible to achieve fully autonomous operation.

[0039] Therefore, to solve the above problems, the present application provides an industrial inspection and handling robot navigation method and device based on multi-sensor fusion.

[0040] The industrial inspection and handling robot navigation method based on multi-sensor fusion provided by the embodiments of the present application can be applied to, for example Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, placed on the cloud or other servers. The terminal 102 can send the obtained multi-sensory data to the server 104, and the multi-sensory data includes: 3D laser ranging data, visible light vision data, and infrared vision data; after receiving the multi-sensory data, for the multi-sensory data, the server 104 unifies the multi-sensory data under a common framework for consistent characterization and processing, and constructs a multi-level map; the common framework is a framework obtained based on the consistent description of the multi-sensory data at different levels; the multi-level map includes: the data layer, feature layer, and semantic layer of the node scene; the data layer of the node scene is the environmental perception data obtained based on the model of point cloud plus spatial octree grid; the feature layer is the pose estimation and target recognition obtained based on the geometric representation model extracted from the multi-source fusion data features; the semantic layer is the representation form abstracted based on the semantic map representation model on the basis of the 3D scene semantic segmentation result; the consistency at different levels includes: the data observation space and time consistency, modeling scale consistency, scene feature consistency, and scene semantic relationship consistency of the inspection and handling robot; based on the semantic segmentation technology of deep learning, semantic segmentation is performed on the semantic layer in the multi-level map to obtain a semantic segmentation result; the semantic segmentation technology includes: a semantic segmentation model based on a fully convolutional neural network and a semantic segmentation model based on a fully connected conditional random field of a convolutional neural network; the semantic segmentation result is used to realize the understanding of complex dynamic scenes; according to the semantic segmentation result, based on the intelligent motion planning technology of reinforcement learning, combined with the learning of the reward function of intrinsic motivation, plan a collision-free shortest path in a complex dynamic environment to achieve long-term autonomous navigation; the intelligent motion planning technology based on reinforcement learning refers to planning a collision-free shortest path to a target position through the self-observation of the inspection and handling robot in a complex dynamic environment; the learning of the reward function of intrinsic motivation is used to encourage the inspection and handling robot to explore unknown areas. The server 104 can feedback the obtained navigation path to the terminal 102. In addition, in some embodiments, the industrial inspection and handling robot navigation method based on multi-sensor fusion can also be implemented separately by the server 104 or the terminal 102. For example, the terminal 102 can directly perform navigation on the obtained multi-sensory data, or the server 104 can obtain multi-sensory data from the data storage system and perform navigation on the multi-sensory data.

[0041] Among them, the terminal 102 can be, but is not limited to, various desktop computers, laptop computers, smart phones, and tablet computers. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers, and can also be a cloud server.

[0042] In an exemplary embodiment, as Figure 2 shown, a navigation method for an industrial inspection and handling robot based on multi-sensor fusion is provided. This method is executed by a computer device, specifically, it can be executed alone by a computer device such as a terminal or a server, or jointly executed by a terminal and a server. In the embodiments of the present application, taking this method applied to Figure 1 the server 104 therein as an example for illustration, it includes the following steps S1 to S4. Among them:

[0043] S1: Obtain multi-source perception data; the multi-source perception data includes: three-dimensional laser ranging data, visible light vision data, and infrared vision data.

[0044] S2: Unify the multi-source perception data under a common framework for consistent representation and processing, and construct a multi-level map; the common framework is a framework obtained based on the consistent description of multi-source perception data at different levels; the multi-level map includes: a data layer, a feature layer, and a semantic layer of the node scene; the data layer of the node scene is environmental perception data obtained based on a model of point cloud plus spatial octree grid; the feature layer is pose estimation and target recognition obtained based on a geometric representation model extracted from multi-source fusion data features; the semantic layer is a representation form abstracted based on the semantic map representation model on the basis of the three-dimensional scene semantic segmentation result; the consistency at different levels includes: data observation space and time consistency, modeling scale consistency, scene feature consistency, and scene semantic relationship consistency of the inspection and handling robot.

[0045] S3: Based on the semantic segmentation technology of deep learning, perform semantic segmentation on the semantic layer in the multi-level map to obtain a semantic segmentation result; the semantic segmentation technology includes: a semantic segmentation model based on a fully convolutional neural network and a semantic segmentation model based on a fully connected conditional random field of a convolutional neural network; the semantic segmentation result is used to realize the understanding of complex dynamic scenes.

[0046] S4: According to the semantic segmentation result, based on the intelligent motion planning technology of reinforcement learning, combined with the learning of the reward function of intrinsic motivation, plan a collision-free shortest path in a complex dynamic environment to achieve long-term autonomous navigation; the intelligent motion planning technology based on reinforcement learning refers to planning a collision-free shortest path to a target position through the self-observation of the inspection and handling robot in a complex dynamic environment; the learning of the reward function of intrinsic motivation is used to encourage the inspection and handling robot to explore unknown areas.

[0047] Implementing the above steps S1 to S4 improves the adaptability of the inspection and handling robot to large-scale, complex and diverse industrial plant scenarios, and realizes reliable 3D scene modeling and reconstruction; through the multi-level map construction technology, long-term navigation and autonomous scene understanding are realized, overcoming the limitation that the inspection and handling robot cannot work autonomously for a long time; based on the semantic segmentation technology of deep learning, the understanding ability of complex dynamic scenes is improved; based on the intelligent motion planning technology of reinforcement learning, combined with the learning of the reward function of intrinsic motivation, the collision-free shortest path planning in complex dynamic environments is realized; it promotes the inspection and handling robot working in complex dynamic environments to get rid of the dependence on remote control operations and gradually develop towards the direction of full autonomy.

[0048] As an optional implementation manner, in step S1, it specifically includes:

[0049] S11: Use a 3D laser sensor to measure distances and obtain 3D laser ranging data.

[0050] S12: Use a visible light vision sensor to capture the visible light vision data around the inspection and handling robot; the visible light vision data includes: the visual features of the environment and the shape and color information of objects.

[0051] S13: Use an infrared vision sensor for industrial inspection to obtain infrared vision data.

[0052] Since the perception data obtained by the inspection and handling robot working in a large outdoor scene comes from different locations and different perspectives, and there are also obvious differences in the observation dimensions, scales and other characteristics of heterogeneous sensor data, how to unify the multi-source heterogeneous perception data into a common framework for consistent representation and processing is the premise and foundation of this research project. In order to achieve the consistent representation and description of multi-source environmental perception data, based on multi-sensor joint calibration and data association, through the comparative analysis of different types of environmental feature representation methods, the framework design is carried out from technical perspectives such as the spatial and temporal consistency of the inspection and handling robot data observation, the modeling scale (parameters) consistency, the scene feature consistency, and the scene semantic relationship consistency, so as to finally give the consistent description algorithm framework of multi-source environmental information at different levels. Among them, the spatial and temporal consistency of data observation belongs to the underlying data acquisition layer, the environmental model construction scale (parameters) and scene feature consistency belong to the intermediate knowledge generation layer, and the scene semantic relationship consistency belongs to the top-level scene understanding layer.

[0053] Specifically, the steps of using multiple heterogeneous sensors such as 3D laser ranging, visible light vision, and infrared vision to obtain multi-source perception data are as follows:

[0054] A1: Rapid distance measurement is carried out using a 3D laser sensor. Based on laser technology, it measures the distance by emitting laser and receiving the signal reflected back. It consists of a laser, a laser detector, and a measurement circuit. The laser sensor can achieve non-contact long-distance measurement, and has the advantages of high speed, high precision, large measurement range, and strong resistance to light and electrical interference.

[0055] A2: A visible light vision sensor is used to capture visual image data around the robot. The visible light vision sensor obtains the visual characteristics of the environment and the shape and color information of objects by capturing the images generated by the reflection and refraction of light, and is used to provide the robot with vision-based environmental perception ability, and can identify visual markers of road signs and obstacles.

[0056] A3: An infrared vision sensor is used for industrial inspection. It is a technology that uses infrared radiation for imaging and detection, and can obtain information such as the heat distribution and temperature change of the target object. In infrared vision technology, near-infrared and mid- and far-infrared are two common wavelength ranges, corresponding to different application scenarios and characteristics respectively. Mid- and far-infrared images can be used for defect detection in the industrial field, equipment heat distribution detection, etc., to help engineers discover problems in time and carry out repairs.

[0057] As an optional implementation manner, in step S2, it specifically includes:

[0058] S21: Based on the consistency of data observation space and time, modeling scale, scene features, and scene semantic relationship of the inspection and handling robot, a common framework is established.

[0059] S22: The multi-source perception data is preliminarily integrated using timestamp alignment technology to obtain the integrated data.

[0060] S23: The integrated data is jointly calibrated and paralleled to obtain the calibrated and paralleled data.

[0061] S24: Regarding the consistency of data observation space and time of the inspection and handling robot as the underlying data acquisition layer, the consistency of modeling scale and scene features as the intermediate knowledge generation layer, and the consistency of scene semantic relationship as the top-level scene understanding layer, the common framework is optimized to obtain the optimized framework.

[0062] S25: Based on the calibrated and paralleled data and the optimized framework, a multi-level map is constructed.

[0063] Specifically, unifying the multi-source perception data obtained in step S1 under the common framework for consistency representation and processing, and constructing a multi-level map, including the following content:

[0064] B1: Initially integrate multi-source data from A1, A2, and A3 to establish a time-synchronized dataset, providing a basis for subsequent in-depth analysis and data fusion. Specifically, first ensure that the data from each source reflects the state at the same time point through timestamp alignment technology, solving the problem of time inconsistency caused by differences in sensor acquisition frequencies or processing delays; information can be accurately recorded and aligned on a unified timeline, laying a solid foundation for subsequent advanced data processing and fusion analysis.

[0065] B2: In order to unify heterogeneous multi-sensor data under a common framework for consistent representation and processing. First, jointly calibrate and correlate the multi-sensor data.

[0066] B3: Take the spatial and temporal consistency of data observations as the underlying data acquisition layer.

[0067] B4: Take the consistency between the environmental model construction scale (parameters) and scene features as the intermediate knowledge generation layer.

[0068] B5: Take the consistency of scene semantic relationships as the top-level scene understanding layer.

[0069] B6: Through comparative analysis of different types of environmental feature representation methods, design a framework from the perspectives of the spatial and temporal consistency of inspection and transportation robot data observations, modeling scale (parameters) consistency, scene feature consistency, scene semantic relationship consistency, etc., and give an algorithm framework for the consistency description of multi-source environmental information at different levels.

[0070] This application intends to use a topological map model to complete the global environmental representation of a large-scale scene. The topological nodes in the global topological map correspond to specific local scenes. Therefore, these topological node scenes need to be further expanded into a multi-level map to support the autonomous navigation and task planning of inspection and transportation robots. The specific technical route to be taken is as follows:

[0071] 1) The data layer representation of the node scene uses a model of point cloud plus spatial octree grid. Although it occupies more storage space, its rich environmental perception data is conducive to the implementation of autonomous obstacle avoidance and path planning algorithms for inspection and transportation robots.

[0072] 2) The feature layer representation is a geometric representation model based on the feature extraction of multi-source fusion data, used for the pose estimation and target recognition of the robot.

[0073] 3) The semantic layer adopts a semantic map representation model, which is an abstract representation form based on the semantic segmentation results of a three-dimensional scene and is the key to enabling the inspection and handling robot to understand the environment similar to humans. In summary, the multi-level environmental map representation framework composed of multiple map models has information complementarity and can meet the different functional requirements of the inspection and handling robot during autonomous navigation in a large-scale complex environment.

[0074] Another key technology for constructing a multi-level map is how to achieve data interconnection between different levels of the multi-level map in the above system framework design. Since the environmental perception information of the inspection and handling robot is mainly three-dimensional laser point clouds, Figure 3 shows a system framework of a multi-level environmental map taking three-dimensional point cloud data as an example, as well as the data flow and association relationships between different levels. As Figure 3 shown in the framework, different types and multi-level environmental map systems are integrated together, and rapid switching and data association can be performed between different map levels according to the specific application needs of the robot.

[0075] As an optional implementation manner, in step S3, it specifically includes:

[0076] S31: Using a projection transformation algorithm, convert the three-dimensional point cloud of the semantic layer in the multi-level map into an image to obtain a scene description with balanced pixel distribution.

[0077] S32: Use a semantic segmentation model based on a fully convolutional neural network to perform preliminary feature extraction on the scene description with balanced pixel distribution to obtain a preliminary extraction result.

[0078] S33: Use a semantic segmentation model based on a fully connected conditional random field of a convolutional neural network to optimize the preliminary extraction result to obtain a semantic segmentation result.

[0079] Considering that the sampling intervals, observation perspectives, and observation ranges of different sensors are different, which will lead to unbalanced point cloud distributions after reconstruction. Therefore, it is necessary to use a projection transformation algorithm to convert the three-dimensional point cloud into an image for representation, so that the originally unbalanced three-dimensional point cloud can obtain a scene description with relatively balanced pixel distribution on the image model. For the problem of multi-category scene recognition in outdoor scenes, two models in deep learning theory are proposed, namely a fully convolutional neural network and a fully connected conditional random field based on a convolutional neural network. Both of these models use images as data sources and use operations such as convolutional layers, pooling layers, activation layers, and deconvolution layers as computational units. The fully convolutional neural network uses the result of fusing multiple deconvolution layers and intermediate convolutional layers to ensure the effective transmission of residuals during model training, and uses a SoftMax layer as the final classification output layer.

[0080] The design of the fully connected conditional random field model of the convolutional neural network has a relatively deep convolutional neural network model structure, which can mine deep global image features and use the fully connected conditional random field as the final classification output layer. The advantage of the fully connected conditional random field is that it considers the influence of the semantic relationship of adjacent pixels in the global image, and designs the corresponding penalty term to be added to the classifier model. Finally, the residual is calculated according to the labeled data, and the conversion result is passed to the convolutional neural network model.

[0081] Semantic segmentation mainly classifies pixels, that is, classifies and labels each frame of the picture to distinguish semantic information such as background, people, objects, vehicles, etc.

[0082] The fully convolutional neural network uses the result of fusing multiple deconvolution layers and intermediate convolutional layers to ensure the effective transmission of residuals during model training, and uses the SoftMax layer as the final classification output layer. The fully convolutional neural network can achieve good results in semantic segmentation, but there are still problems with rough segmentation edges. Therefore, the fully connected conditional random field based on the convolutional neural network is introduced. First, the fully convolutional neural network is used for preliminary feature extraction, and then the fully connected conditional random field based on the convolutional neural network is used for optimized output to solve the problem of rough segmentation edges.

[0083] Specifically, the semantic segmentation technology based on deep learning performs semantic segmentation on the semantic layer in the multi-level map to achieve the understanding of complex dynamic scenes, specifically including:

[0084] C1: Using the projection transformation algorithm, the three-dimensional point cloud is converted into an image for representation, so that the originally unevenly distributed three-dimensional point cloud obtains a scene description with relatively balanced pixel distribution on the image model.

[0085] C2: For the recognition of multi-category scenes in outdoor scenes, the fully convolutional neural network and the fully connected conditional random field based on the convolutional neural network in the deep learning theory are used. The image is used as the data source, and operations such as convolutional layer, pooling layer, activation layer, deconvolution layer, etc. are used as calculation units. The fully convolutional neural network uses the result of fusing multiple deconvolution layers and intermediate convolutional layers to ensure the effective transmission of residuals during model training, and uses the SoftMax layer as the final classification output layer.

[0086] C3: Using the fully connected conditional random field model of the convolutional neural network to design a relatively deep convolutional neural network model structure to mine deep global image features, and using the fully connected conditional random field as the final classification output layer. Considering the influence of the semantic relationship of adjacent pixels in the global image, design the corresponding penalty term to be added to the classifier model. Finally, calculate the residual according to the labeled data, and pass the conversion result to the convolutional neural network model.

[0087] As an alternative implementation, in step S4, it specifically includes:

[0088] S41: Use several consecutive frame RGB images captured by the inspection and handling robot as the input of reinforcement learning, and use the target position as the output of reinforcement learning to optimize the learning of the navigation strategy in an end-to-end manner to obtain an optimized strategy; the target position is specified by the user and is represented by the RGB image captured by the inspection and handling robot from the first perspective; the end-to-end learning of the navigation strategy is used to reflect the relative position relationship between the current position of the robot and the target position.

[0089] S42: Based on the reward function learning method of intrinsic motivation, store the image features of the environmental observation during the operation of the inspection and handling robot in a memory buffer, and judge the similarity between the current scene and the scene image features in the memory through a deep neural network.

[0090] S43: Determine the reward and punishment result according to the similarity; the reward and punishment result is that if the current scene is a new scene, the inspection and handling robot is rewarded, otherwise it is punished.

[0091] S44: Plan a collision-free shortest path in a complex dynamic environment according to the semantic segmentation result, the optimized strategy and the reward and punishment result to achieve long-term autonomous navigation.

[0092] As an optional implementation manner, the navigation method of the industrial inspection and handling robot based on multi-sensor fusion further includes:

[0093] S5: Effectively fuse the multi-source perception data by using the pose and view point information of the inspection and handling robot to obtain the fused data;

[0094] S6: Perform scene registration based on the fused data;

[0095] S7: Remove the dynamic obstacle nodes in the scene and use the 3D point cloud registration technology to solve the optimal pose of the inspection and handling robot;

[0096] S8: Optimize the result of scene registration according to the optimal pose of the inspection and handling robot and the matching degree of the effective background information in the semantic segmentation result.

[0097] The long-term robust relocalization of the inspection and handling robot is divided into global localization at the topological node level and local precise localization within the topological node sub-map. In an environment with weak GPS, the rough localization at the topological node level mainly relies on a robust scene recognition method. During the long-term autonomous operation of the inspection and handling robot, it is difficult to avoid the situation where the localization of a certain node fails due to the similarity of the environment of individual nodes or the change of the scene. Therefore, the topological structure of the topological map should be fully utilized to achieve robust global relocalization at the topological node level based on the node sequence.

[0098] The key to precise positioning inside a topological node is to determine the optimal pose of the inspection and transportation robot within this topological sub-node. Since the 3D laser scanning has a wide range, the well-structured and highly distinguishable background information in the environment such as buildings and trees can be fully utilized for accurate scene registration. As Figure 4 shown, the pose and viewpoint information of the inspection and transportation robot are used to effectively fuse the immediately obtained multi-sensory data to detect dynamic obstacles such as pedestrians and vehicles, as well as background information such as trees, buildings, and prominent road signs. During the scene registration process, the uncertainty of the dynamic obstacle positions affects the accuracy of scene registration. Therefore, unstable factors such as pedestrians and vehicles should be removed, and only relatively stable background information should be retained. At this time, 3D point cloud registration technologies such as NDT can be used to solve the optimal pose of the robot within the sub-map of this node. Finally, the matching degree of the effective background information in the semantic segmentation result is used to optimize the result of scene registration, thereby ensuring the accuracy of the robot's positioning within the topological node.

[0099] The research goal of intelligent motion planning based on reinforcement learning is to plan a collision-free shortest path to the target position through the robot's own observations in a complex dynamic environment. This project intends to adopt an end-to-end approach to learn the navigation strategy, which can effectively reflect the relative position relationship between the robot's current position and the target position. The continuous n-frame RGB images captured by the robot are used as the input for reinforcement learning, and the target position is represented by the RGB image taken from the first-person perspective of the robot. The user can flexibly specify a new target position. Aiming at the sparse reward problem in reinforcement learning-based path planning, a reward function learning method based on intrinsic motivation is proposed, and the robot's intrinsic motivation is used to further drive the robot to explore the environment. To obtain the reward function of intrinsic motivation, the image features of the robot's environmental observations during operation are stored in a memory buffer. The similarity between the current scene and the scene image features in the memory is measured by a deep neural network. If the current scene is a new scene, the robot is rewarded, otherwise it is punished. As Figure 5 shown, the reward function learning based on intrinsic motivation includes two networks: the scene embedding network E and the scene comparison network C. The scene comparison network determines whether the current scene and the memory scene have a proximity relationship in terms of spatial position. If the two scenes are adjacent scenes, the reward function estimation module will output a lower reward value, thereby encouraging the robot to explore unknown areas and avoid lingering in the areas that have been traversed.

[0100] Specifically, according to the semantic segmentation result, the intelligent motion planning technology based on reinforcement learning, combined with the reward function learning of intrinsic motivation, plans a collision-free shortest path in a complex dynamic environment to achieve long-term autonomous navigation, specifically including:

[0101] D1: Make full use of the topological structure of the topological map to achieve robust global relocalization at the topological node level based on the node sequence. In an environment with weak GPS, the rough localization at the topological node level relies on a robust scene recognition method to avoid the situation where the localization of a certain node fails due to similar environments of individual nodes or scene changes.

[0102] D2: For precise internal localization of topological nodes, three-dimensional laser scanning is used to determine the optimal pose of the inspection and handling robot within the topological sub-node. Use the well-structured and highly distinguishable background information in the environment such as buildings and trees for accurate scene registration. Effectively fuse the immediately obtained multi-sensory data using the pose and viewing point information of the inspection and handling robot to detect dynamic obstacles such as pedestrians and vehicles and background information such as trees, buildings, and significant road signs.

[0103] D3: During the scene registration process, the uncertainty of the dynamic obstacle position affects the accuracy of scene registration. Remove unstable factors such as pedestrians and vehicles and only retain relatively stable background information. Use 3D point cloud registration technologies such as NDT to solve the optimal pose of the robot within the node sub-map.

[0104] D4: Optimize the result of scene registration using the matching degree of the effective background information in the semantic segmentation result to ensure the accuracy of the robot's localization within the topological node.

[0105] D5: To effectively reflect the relative position relationship between the current position and the target position of the robot, an end-to-end method is adopted for intelligent motion planning based on reinforcement learning. Use the continuous n-frame RGB images captured by the robot as the input of reinforcement learning, and the target position is represented by the RGB image captured from the first perspective of the robot, and a new target position can be flexibly specified.

[0106] Furthermore, in step D5, it specifically includes:

[0107] D51: For the sparse reward problem in the reinforcement learning-based path planning, use a reward function learning method based on intrinsic motivation to further drive the robot to explore the environment using the robot's intrinsic motivation.

[0108] D52: To obtain the reward function of intrinsic motivation, store the image features of the robot's environmental observations during operation in a memory buffer, measure the similarity between the current scene and the scene image features in the memory through a deep neural network, and reward the robot if the current scene is a new scene, otherwise punish it.

[0109] The present application also provides an application scenario, which applies the above-mentioned navigation method for an industrial inspection and handling robot based on multi-sensor fusion. Specifically: The navigation method for an industrial inspection and handling robot based on multi-sensor fusion provided in this embodiment can be applied in an industrial inspection and handling scenario. The industrial inspection and handling scenario includes: a data acquisition link, a multi-level map construction link, a semantic segmentation link, and a path planning link; First, multi-source perception data is acquired; the multi-source perception data includes: three-dimensional laser ranging data, visible light vision data, and infrared vision data; Secondly, the multi-source perception data is unified under a common framework for consistency representation and processing to construct a multi-level map; the common framework is a framework obtained based on the consistency description of multi-source perception data at different levels; the multi-level map includes: a data layer, a feature layer, and a semantic layer of the node scene; the data layer of the node scene is environmental perception data obtained based on a model of point cloud plus spatial octree grid; the feature layer is pose estimation and target recognition obtained based on a geometric representation model extracted from multi-source fusion data features; the semantic layer is a representation form abstracted based on the semantic map representation model on the basis of the three-dimensional scene semantic segmentation result; the consistency at different levels includes: the consistency of the data observation space and time of the inspection and handling robot, the modeling scale consistency, the scene feature consistency, and the scene semantic relationship consistency; Then, based on the semantic segmentation technology of deep learning, the semantic layer in the multi-level map is semantically segmented to obtain a semantic segmentation result; the semantic segmentation technology includes: a semantic segmentation model based on a fully convolutional neural network and a semantic segmentation model based on a fully connected conditional random field of a convolutional neural network; the semantic segmentation result is used to realize the understanding of complex dynamic scenes; Finally, according to the semantic segmentation result, based on the intelligent motion planning technology of reinforcement learning, combined with the learning of the reward function of intrinsic motivation, the collision-free shortest path in a complex dynamic environment is planned to achieve long-term autonomous navigation; the intelligent motion planning technology based on reinforcement learning refers to planning a collision-free shortest path to a target position through the self-observation of the inspection and handling robot in a complex dynamic environment; the learning of the reward function of intrinsic motivation is used to encourage the inspection and handling robot to explore unknown areas.

[0110] The present application improves the adaptability of the inspection and handling robot to large-scale, complex and diverse industrial plant scenarios, and realizes reliable three-dimensional scene modeling and reconstruction; through the multi-level map construction technology, long-term navigation and autonomous scene understanding are realized, overcoming the limitation that the inspection and handling robot cannot work autonomously for a long time; based on the semantic segmentation technology of deep learning, the ability to understand complex dynamic scenes is improved; based on the intelligent motion planning technology of reinforcement learning, combined with the learning of the reward function of intrinsic motivation, the collision-free shortest path planning in a complex dynamic environment is realized; it promotes the inspection and handling robot working in a complex dynamic environment to get rid of the dependence on remote control operation and gradually develop towards the direction of full autonomy.

[0111] Based on the same inventive concept, an embodiment of the present application further provides an industrial inspection and handling robot navigation device based on multi-sensor fusion for implementing the above-mentioned industrial inspection and handling robot navigation method based on multi-sensor fusion. The implementation solutions provided by this device to solve problems are similar to the implementation solutions described in the above method. Therefore, the specific limitations in one or more embodiments of the industrial inspection and handling robot navigation device based on multi-sensor fusion provided below can refer to the limitations on the industrial inspection and handling robot navigation method based on multi-sensor fusion in the above text, and will not be repeated here.

[0112] In an exemplary embodiment, as Figure 6 shown, an industrial inspection and handling robot navigation device based on multi-sensor fusion is provided. The industrial inspection and handling robot navigation device based on multi-sensor fusion includes:

[0113] A data acquisition unit M1, configured to acquire multi-source perception data.

[0114] A consistency characterization and processing unit M2, configured to unify the multi-source perception data into a common framework for consistency characterization and processing, and construct a multi-level map; the common framework is a framework obtained based on the consistency description of multi-source perception data at different levels.

[0115] A semantic segmentation unit M3, configured to perform semantic segmentation on the semantic layer in the multi-level map based on the semantic segmentation technology of deep learning, and obtain a semantic segmentation result; the semantic segmentation technology includes: a semantic segmentation model based on a fully convolutional neural network and a semantic segmentation model based on a fully connected conditional random field of a convolutional neural network; the semantic segmentation result is used to realize the understanding of complex dynamic scenes.

[0116] A path planning unit M4, configured to plan a collision-free shortest path in a complex dynamic environment based on the semantic segmentation result, based on the intelligent motion planning technology of reinforcement learning, combined with the learning of an intrinsic incentive reward function, to achieve long-term autonomous navigation; the intelligent motion planning technology based on reinforcement learning refers to planning a collision-free shortest path to a target position through the self-observation of the inspection and handling robot in a complex dynamic environment; the learning of the intrinsic incentive reward function is used to encourage the inspection and handling robot to explore unknown areas.

[0117] In an exemplary embodiment, a computer device is provided. The computer device can be a server or a terminal, and its internal structure diagram can be as Figure 7As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store multi-sensor data. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through a network connection. When the computer program is executed by the processor, it implements a navigation method for an industrial inspection and handling robot based on multi-sensor fusion.

[0118] Those skilled in the art can understand that Figure 7 the structure shown in is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0119] In an exemplary embodiment, a computer device is further provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the above method embodiments are implemented.

[0120] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by the processor, the above method embodiments are implemented.

[0121] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by the processor, the above method embodiments are implemented.

[0122] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0123] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memories can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memories can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0124] The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0125] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0126] Specific examples are used in this article to elaborate on the principles and implementation manners of the present application. The descriptions of the above embodiments are only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A navigation method for an industrial inspection and handling robot based on multi-sensor fusion, characterized in that: The industrial inspection and handling robot navigation method based on multi-sensor fusion includes: Acquire multi-sensory data; Unifying the multi-sensory data into a common framework for consistent representation and processing, and constructing a multi-level map; the common framework is a framework obtained based on the consistent description of the multi-sensory data at different levels; Based on the semantic segmentation technology of deep learning, the semantic layer in the multi-level map is semantically segmented to obtain a semantic segmentation result; the semantic segmentation technology includes: a semantic segmentation model based on a fully convolutional neural network and a semantic segmentation model based on a fully connected conditional random field of a convolutional neural network; the semantic segmentation result is used to realize the understanding of complex dynamic scenes; According to the semantic segmentation results, the intelligent motion planning technology based on reinforcement learning is combined with the reward function learning of intrinsic motivation to plan the collision-free shortest path in a complex dynamic environment to achieve long-term autonomous navigation; the intelligent motion planning technology based on reinforcement learning refers to planning a collision-free shortest path to the target position through the inspection and handling robot's own observation in a complex dynamic environment; the reward function learning of intrinsic motivation is used to encourage the inspection and handling robot to explore unknown areas.

2. The navigation method of the industrial inspection and handling robot based on multi-sensor fusion according to claim 1 is characterized in that: Acquire multi-sensory data, including: Using a three-dimensional laser sensor to measure distance and obtain three-dimensional laser distance measurement data; Using a visible light vision sensor to capture visible light vision data around the inspection and handling robot; the visible light vision data includes: visual features of the environment and shape and color information of the object; Use infrared vision sensors to perform industrial inspections and obtain infrared vision data.

3. The navigation method of the industrial inspection and handling robot based on multi-sensor fusion according to claim 1 is characterized in that: The multi-sensory data are unified into a common framework for consistent representation and processing, and a multi-level map is constructed, specifically including: A common framework is established based on the consistency of observation space and time, modeling scale, scene features and semantic relationship of the inspection and handling robot data. Using timestamp alignment technology to preliminarily integrate the multivariate sensing data to obtain integrated data; Jointly calibrating and paralleling the integrated data to obtain calibrated and paralleled data; Taking the spatial and temporal consistency of the inspection and handling robot data observation as the bottom data acquisition layer, taking the modeling scale consistency and the scene feature consistency as the middle knowledge generation layer, taking the scene semantic relationship consistency as the top scene understanding layer, optimizing the common framework, and obtaining an optimized framework; Based on the calibrated and parallelized data and the optimized framework, a multi-level map is constructed.

4. The navigation method of the industrial inspection and handling robot based on multi-sensor fusion according to claim 1 is characterized in that: The semantic segmentation model based on the fully convolutional neural network uses the results of fusing multiple deconvolution layers and intermediate convolution layers to ensure that the residual is effectively transmitted during model training, and uses the SoftMax layer as the final classification output layer; the semantic segmentation model based on the convolutional neural network fully connected conditional random field is used to mine deep global image features, and uses the fully connected conditional random field as the final classification output layer.

5. The navigation method of the industrial inspection and handling robot based on multi-sensor fusion according to claim 1 is characterized in that: According to the semantic segmentation results, intelligent motion planning technology based on reinforcement learning, combined with intrinsically motivated reward function learning, plans the shortest collision-free path in a complex dynamic environment to achieve long-term autonomous navigation, including: Using several consecutive frames of RGB images taken by the inspection and handling robot as reinforcement learning input and the target position as reinforcement learning output, the end-to-end learning navigation strategy is optimized to obtain an optimized strategy; the target position is specified by the user and represented by the RGB image taken by the inspection and handling robot as a first-person perspective; the end-to-end learning navigation strategy is used to reflect the relative position relationship between the robot's current position and the target position; Based on the intrinsic motivation reward function learning method, the image features of the environment observed by the inspection and handling robot during operation are stored in a memory buffer, and the similarity between the current scene and the image features of the scene in memory is judged through a deep neural network; Determine a reward or punishment result according to the similarity; if the current scene is a new scene, the inspection and handling robot is rewarded, otherwise it is punished; According to the semantic segmentation result, the optimized strategy and the reward and punishment result, a collision-free shortest path in a complex dynamic environment is planned to achieve long-term autonomous navigation.

6. The navigation method of the industrial inspection and handling robot based on multi-sensor fusion according to claim 1 is characterized in that: The industrial inspection and handling robot navigation method based on multi-sensor fusion also includes: Using the posture and viewpoint information of the inspection and handling robot to effectively fuse the multi-sensory data to obtain fused data; Based on the fused data, performing scene registration; Remove dynamic obstacle nodes in the scene and use 3D point cloud registration technology to solve the optimal posture of the inspection and handling robot; According to the optimal posture of the inspection and handling robot, the matching degree of effective background information in the semantic segmentation result is used to optimize the result of scene registration.

7. An industrial inspection and handling robot navigation device based on multi-sensor fusion, the industrial inspection and handling robot navigation device based on multi-sensor fusion is used to implement the industrial inspection and handling robot navigation method based on multi-sensor fusion according to any one of claims 1 to 6, characterized in that: The industrial inspection and handling robot navigation device based on multi-sensor fusion includes: A data acquisition unit, used for acquiring multi-sensory data; A consistency representation and processing unit, used to unify the multi-sensory data into a common framework for consistency representation and processing, and to construct a multi-level map; the common framework is a framework obtained based on the consistency description of the multi-sensory data at different levels; A semantic segmentation unit, which is used to perform semantic segmentation on the semantic layer in the multi-layer map based on the semantic segmentation technology of deep learning to obtain a semantic segmentation result; the semantic segmentation technology includes: a semantic segmentation model based on a fully convolutional neural network and a semantic segmentation model based on a convolutional neural network fully connected conditional random field; the semantic segmentation result is used to achieve understanding of complex dynamic scenes; A path planning unit is used to plan a collision-free shortest path in a complex dynamic environment according to the semantic segmentation result, based on the intelligent motion planning technology of reinforcement learning, combined with the learning of the reward function of intrinsic motivation, so as to achieve long-term autonomous navigation; the intelligent motion planning technology based on reinforcement learning refers to planning a collision-free shortest path to the target position through the inspection and handling robot's own observation in a complex dynamic environment; the reward function learning of intrinsic motivation is used to encourage the inspection and handling robot to explore unknown areas.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the industrial inspection and handling robot navigation method based on multi-sensor fusion as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the navigation method of an industrial inspection and handling robot based on multi-sensor fusion described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the navigation method of an industrial inspection and handling robot based on multi-sensor fusion described in any one of claims 1 to 6 is implemented.

Images