A safety inspection robot path planning method and system

Abstract

The application provides a kind of safety inspection robot path planning method and system, it is related to robot path planning technical field.The method comprises: setting the starting point of the inspection path of safety inspection robot, and setting multiple detection positions in the inspection path;Obtain multiple orientation detection images;Determine whether to add branch inspection path;Determine multiple branch detection positions, and determine branch inspection path;Obtain multiple orientation branch detection images;Determine alarm information;After safety inspection robot reaches the end point of branch inspection path, continue to go to the next detection position;In the case where it is determined that no branch inspection path is added, go directly to the next detection position.According to the application, whether to add branch inspection path can be determined based on the photographed image, and the branch inspection path is planned, and then the alarm information is determined, the inspection path can be adjusted according to abnormal conditions, the accuracy of safety inspection robot inspection path is improved, and the pertinence and efficiency of inspection are improved.

Description

Technical Field

[0001] This invention relates to the field of robot path planning technology, and in particular to a method and system for path planning of a safety inspection robot. Background Technology

[0002] In industrial production and facility management, safety inspection robots are commonly used to inspect production areas or equipment to promptly identify potential risks. However, current technologies typically rely on human staff to plan the robot's inspection paths. These paths are usually fixed, leading to the possibility of overlooking safety risks and low inspection efficiency. Furthermore, it's difficult to adjust the planned path to address equipment anomalies and to issue timely alarms.

[0003] The information disclosed in the background section of this application is intended only to enhance the understanding of the general background of this application and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0004] This invention provides a path planning method and system for a safety inspection robot, which can solve the technical problem that related technologies are unable to adjust the planned path in response to abnormal equipment conditions.

[0005] According to a first aspect of the present invention, a path planning method for a safety inspection robot is provided, comprising:

[0006] Set the starting point of the inspection path for the safety inspection robot, and set multiple detection positions in the inspection path, where the starting point of the inspection path is the first detection position;

[0007] At the i-th detection position, multiple detection images of different orientations are acquired through multiple cameras, including an RGB camera and an infrared camera. The detection images are a first detection image captured by the RGB camera and a second detection image captured by the infrared camera.

[0008] Based on the detected image and the orientation of the multiple cameras, as well as the path between the i-th detection position and the (i+1)-th detection position, determine whether to add a branch inspection path;

[0009] Given that an additional branch inspection path is to be added, multiple branch detection positions are determined, and the branch inspection path is determined, wherein the starting point and the ending point of the branch inspection path are both the i-th detection position.

[0010] At multiple branch detection locations, multiple cameras are used to acquire branch detection images facing multiple directions;

[0011] Based on the branch detection images at multiple branch detection locations, determine the alarm information;

[0012] After the safety inspection robot reaches the end of the branch inspection path, it continues to the (i+1)th inspection position.

[0013] If no additional branch inspection path is added, proceed directly to the (i+1)th inspection location.

[0014] According to the present invention, determining whether to add a branch inspection path includes:

[0015] The first image detection model is used to process the first detection images in various orientations to obtain the first region where the defect is located.

[0016] The second image detection model is used to process the second detection images in various orientations to obtain the second region of abnormal heating.

[0017] Based on the first and second regions, determine the location information to be observed;

[0018] Based on the location information to be observed, the i-th detection location, and the orientation of the multiple cameras, determine the target shooting location of the inspection robot;

[0019] Based on the target shooting location and the path between the i-th detection location and the (i+1)-th detection location, determine whether to add a branch inspection path.

[0020] According to the present invention, determining the location information to be observed includes:

[0021] If the coordinate ranges of the first region and the second region do not intersect, then the coordinates of the centroid of the first region in the first detection image and the coordinates of the centroid of the second region in the second detection image are both used as the location information to be observed.

[0022] If the coordinate ranges of the first region and the second region intersect, then the average coordinates of the centroid of the first region in the first detection image and the centroid of the second region in the second detection image are used as the location information to be observed.

[0023] According to the present invention, the multi-channel camera further includes a depth camera;

[0024] Based on the location information to be observed, the i-th detection location, and the orientation of the multiple cameras, the target shooting position of the inspection robot is determined, including:

[0025] Using a depth camera, determine the first distance between the actual observation location of the location information to be observed in real space and the multiple cameras;

[0026] Based on the coordinates of the i-th detection position in real space and the orientation of the multiple cameras, determine the calibration parameters of the RGB camera and the infrared camera.

[0027] Based on the first distance and the calibration parameters of the RGB camera and the infrared camera, determine the spatial coordinates of the actual observation position of the location information to be observed in the real space;

[0028] The target shooting position of the inspection robot is determined based on the spatial coordinates of the actual observation location.

[0029] According to the present invention, determining the target shooting position of the inspection robot includes:

[0030] According to the formula

[0031] ,

[0032] Obtain the planar coordinates of the first undetermined shooting position corresponding to the k-th actual observation position with the j-th orientation at the i-th detection position. ,in, Let be the spatial coordinates of the k-th actual observation position at the j-th orientation of the i-th detection position. Let i be the spatial coordinates of the i-th detection position. Let i be the planar coordinates of the (i+1)th detection position. For the height of multiple cameras, Preset observation angle;

[0033] According to the formula

[0034] ,

[0035] Obtain the planar coordinates of the second undetermined shooting position corresponding to the k-th actual observation position with the j-th orientation at the i-th detection position. ;

[0036] Determine the first shortest distance between the planar coordinates of the first undetermined shooting position and the planar coordinates of the i-th detection position and the (i+1)-th detection position, and determine the second shortest distance between the planar coordinates of the second undetermined shooting position and the planar coordinates of the i-th detection position and the (i+1)-th detection position.

[0037] If the first shortest distance is less than the second shortest distance, then the planar coordinates of the first undetermined shooting position are determined as the planar coordinates of the undetermined shooting position; otherwise, the planar coordinates of the second undetermined shooting position are determined as the planar coordinates of the undetermined shooting position.

[0038] Cluster the planar coordinates of multiple undetermined shooting locations to obtain the cluster centers of multiple clusters;

[0039] Multiple cluster centers were identified as target shooting locations.

[0040] According to the present invention, determining whether to add a branch inspection path includes:

[0041] Determine the first vector between the i-th detection position and each target shooting position, and the second vector between the (i+1)-th detection position and each target shooting position;

[0042] According to the formula

[0043] ,

[0044] Obtain the first condition Second condition ,in, Let be the first vector corresponding to the shooting position of the s-th target. Let be the second vector corresponding to the shooting position of the s-th target. For vectors The line in question The planar area where the inspection target is located;

[0045] If there exists that satisfies the first condition Second condition If any one of the target shooting positions is determined, then the branch inspection path is added.

[0046] According to the present invention, determining multiple branch detection locations and determining branch inspection paths includes:

[0047] The target shooting location is determined as the branch detection location;

[0048] Determine the second distance between the i-th detection position and each branch detection position;

[0049] Choose any branch detection position as the first branch detection position in the first round of sorting, and sort each branch detection position in clockwise order to obtain the first sequence number of each branch detection position in the first round of sorting;

[0050] According to the formula

[0051] ,

[0052] Obtain the filtering translation distance corresponding to the branch detection position with the first sequence number h. ,in, The second distance corresponds to the branch detection position where the first sequence number is h. The second distance is the branch detection position corresponding to the first sequence number h+1, and when h=m. Where m is the number of branch detection positions. The second distance is the branch detection position corresponding to the first sequence number 1;

[0053] Will The branch detection position corresponding to the first index when the minimum value is obtained is taken as the first branch detection position in the second round of sorting, and the branch detection positions are sorted in counterclockwise order to obtain the second index of each branch detection position in the second round of sorting.

[0054] Take the i-th detection position as the starting point of the branch inspection path, connect each branch detection position in sequence according to the second sequence number, and connect the branch detection position with the second sequence number m to the i-th detection position to obtain the branch inspection path.

[0055] According to a second aspect of the present invention, a path planning system for a safety inspection robot is provided, comprising:

[0056] The settings module allows you to set the starting point of the inspection path for the safety inspection robot and set multiple detection locations along the inspection path, where the starting point of the inspection path is the first detection location.

[0057] The detection image module acquires multiple orientation detection images at the i-th detection position through multiple cameras, including an RGB camera and an infrared camera. The detection images are a first detection image captured by the RGB camera and a second detection image captured by the infrared camera.

[0058] The judgment module determines whether to add a branch inspection path based on the detected image, the orientation of the multiple cameras, and the path between the i-th detection position and the (i+1)-th detection position.

[0059] The branch inspection path module determines multiple branch detection positions and the branch inspection path when it is determined that a branch inspection path will be added. The starting point and ending point of the branch inspection path are both the i-th detection position.

[0060] The branch detection image module acquires branch detection images from multiple directions at multiple branch detection locations using multiple cameras;

[0061] The alarm information module determines alarm information based on the branch detection images at multiple branch detection locations;

[0062] The first module continues to the (i+1)th inspection position after the safety inspection robot reaches the end of the branch inspection path.

[0063] The second module proceeds directly to the (i+1)th detection location if it determines that no additional branch inspection paths will be added.

[0064] By adopting the above technical solution, the present invention can achieve the following technical effects:

[0065] According to the present invention, multiple detection positions can be set in the inspection path. At each detection position, multiple cameras capture detection images from multiple directions to determine whether to add a branch inspection path. When adding a branch inspection path, multiple branch detection positions and branch inspection paths are determined, and alarm information is determined based on the branch detection images. The planned path can be adjusted in time according to the abnormalities that occur during the inspection, reducing the possibility of overlooking safety risks, improving the accuracy of the safety inspection robot's path planning, and enhancing the targeting and efficiency of safety inspections. Furthermore, multiple detection positions can be evenly set in the inspection path, and multiple cameras can capture RGB and infrared images of each detection position from various directions to comprehensively obtain the defects and abnormal heating of the inspected equipment, providing basic data for determining whether to add a branch inspection path. When determining whether to add a branch inspection path, the first distance between the actual observation position of the observed location information in real space and the multi-camera system can be determined based on the depth camera on the multi-camera system. Then, based on the coordinates of each detection position in real space, the orientation of the multi-camera system, the coordinates of the observed location information in the camera coordinate system, and the calibration parameters of the multi-camera system at each detection position, the spatial coordinates of the actual observation position of each observed location information in real space are obtained. Based on the spatial coordinates and orientation angles of the multi-camera system, the x- and y-axis components of the distance between the multi-camera system and each actual observation position are determined, thereby determining the first and second undetermined shooting positions. These undetermined shooting positions are then selected, and the target shooting position is obtained after clustering. Furthermore, based on the ease of reaching the target shooting position from the i-th detection position, it can be determined whether to add a branch inspection path. This improves the accuracy, objectivity, and comprehensiveness of determining whether to add a branch inspection path. Furthermore, the branch detection path with the shortest total distance can be selected based on the filtering translation distance, improving the efficiency of inspection through branch inspection. Furthermore, alarm information can be determined based on the branch detection images. After completing the branch inspection path task or if it is not necessary to add a branch inspection path at the current detection location, the robot proceeds to the next detection location. This improves the accuracy of path planning for the safety inspection robot, enhancing the targeting and efficiency of safety inspections.

[0066] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Other features and aspects of the invention will become clearer from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0068] Figure 1 An exemplary flowchart of a path planning method for a safety inspection robot according to an embodiment of the present invention is shown.

[0069] Figure 2 An exemplary flowchart illustrating the determination of whether to add a branch inspection path according to an embodiment of the present invention is shown;

[0070] Figure 3 A block diagram of a safety inspection robot path planning system according to an embodiment of the present invention is shown as an example. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0072] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0073] Figure 1 An exemplary flowchart illustrates a path planning method for a safety inspection robot according to an embodiment of the present invention, the method comprising:

[0074] Step S1: Set the starting point of the inspection path for the safety inspection robot and set multiple detection positions in the inspection path, wherein the starting point of the inspection path is the first detection position.

[0075] Step S2: At the i-th detection position, multiple detection images of different orientations are acquired through multiple cameras, including an RGB camera and an infrared camera. The detection images are a first detection image captured by the RGB camera and a second detection image captured by the infrared camera.

[0076] Step S3: Based on the detected image and the orientation of the multiple cameras, as well as the path between the i-th detection position and the (i+1)-th detection position, determine whether to add a branch inspection path.

[0077] Step S4: If it is determined that an additional branch inspection path is to be added, multiple branch detection positions are determined, and the branch inspection path is determined. The starting point and ending point of the branch inspection path are both the i-th detection position.

[0078] Step S5: At multiple branch detection locations, acquire branch detection images of multiple orientations using multiple cameras;

[0079] Step S6: Determine alarm information based on the branch detection images at multiple branch detection locations;

[0080] Step S7: After the safety inspection robot reaches the end of the branch inspection path, it continues to the (i+1)th inspection position.

[0081] Step S8: If it is determined that no additional branch inspection path will be added, proceed directly to the (i+1)th inspection position.

[0082] The safety inspection robot path planning method according to an embodiment of the present invention can set multiple detection positions in the inspection path. At each detection position, multiple cameras capture detection images from multiple directions to determine whether to add a branch inspection path. If adding a branch, multiple branch detection positions and branch inspection paths are determined, and alarm information is determined based on the branch detection images. The planned path can be adjusted in time according to the abnormalities that occur during the inspection, reducing the possibility of overlooking safety risks, improving the accuracy of safety inspection robot path planning, and enhancing the targeting and efficiency of safety inspection.

[0083] According to an embodiment of the present invention, in step S1, the starting point of the inspection path of the safety inspection robot is set, and multiple detection positions are set in the inspection path, wherein the starting point of the inspection path is the first detection position. The starting point can be randomly set on the inspection path of the safety inspection robot, which is the first detection position. Multiple detection positions are evenly set in the inspection path, with the same interval between adjacent detection positions. The interval between adjacent detection positions is determined by the equipment being inspected. For example, when inspecting pipes or lines, the interval between adjacent detection positions can be 2 meters; when inspecting steel structure frames, the interval between adjacent detection positions can be 4 meters. The present invention does not impose any limitations on this.

[0084] According to an embodiment of the present invention, in step S2, at the i-th detection position, multiple detection images in multiple orientations are acquired using a multi-channel camera, wherein the multi-channel camera includes an RGB camera and an infrared camera, and the detection images are a first detection image captured by the RGB camera and a second detection image captured by the infrared camera. At the i-th detection position, multiple detection images in multiple orientations (e.g., upward 45°, downward 45°, etc.) are acquired using a preset multi-channel camera. The detection images include a first detection image captured by the RGB camera (i.e., an RGB image) and a second detection image captured by the infrared camera (i.e., an infrared image). The first detection image can be used to determine the defect status of the device under inspection, and the second detection image can be used to determine the abnormal heating status of the device under inspection, and thus to determine whether it is necessary to add a branch inspection path.

[0085] In this way, multiple inspection positions can be set evenly along the inspection path, and RGB and infrared images of each inspection position in each direction can be captured by multiple cameras to comprehensively obtain the defects and abnormal heating of the inspected equipment, providing basic data for determining whether to add branch inspection paths.

[0086] Figure 2 An exemplary flowchart illustrating the determination of whether to add a branch inspection path according to an embodiment of the present invention is shown.

[0087] According to an embodiment of the present invention, in step S3, determining whether to add a branch inspection path based on the detected image, the orientation of the multiple cameras, and the path between the i-th detection position and the (i+1)-th detection position includes: step S31, processing the first detected images of each orientation using a first image detection model to obtain the first region where the defect is located; step S32, processing the second detected images of each orientation using a second image detection model to obtain the second region of abnormal heating; step S33, determining the location information to be observed based on the first region and the second region; step S34, determining the target shooting position of the inspection robot based on the location information to be observed, the i-th detection position, and the orientation of the multiple cameras; and step S35, determining whether to add a branch inspection path based on the target shooting position and the path between the i-th detection position and the (i+1)-th detection position.

[0088] According to an embodiment of the present invention, in step S31, the first detection images of each orientation are processed by a first image detection model to obtain the first region where the defect is located. By processing the first detection images of each orientation by a first image detection model (e.g., R-CNN model, SSD model, etc.), the region in the first detection image where the defect exists (e.g., broken, cracked, rusted, etc.) can be selected, which is the first region.

[0089] According to an embodiment of the present invention, in step S32, the second detection images of each orientation are processed using a second image detection model to obtain a second region of abnormal heating. By processing the second detection images of each orientation using a second image detection model (e.g., YOLO thermal imaging version), the region of abnormal heating in the first detection image can be selected, which is the second region. For example, a heat value threshold is set for each pixel, and pixels with heat values ​​exceeding the threshold are identified as pixels of abnormal heating, thereby determining the second region of abnormal heating.

[0090] According to an embodiment of the present invention, in step S33, determining the position information to be observed based on the first region and the second region includes: if the coordinate range of the first region and the coordinate range of the second region do not intersect, then the coordinates of the centroid of the first region in the first detection image and the coordinates of the centroid of the second region in the second detection image are both used as the position information to be observed; if the coordinate range of the first region and the coordinate range of the second region intersect, then the average coordinates of the coordinates of the centroid of the first region in the first detection image and the coordinates of the centroid of the second region in the second detection image are used as the position information to be observed.

[0091] According to an embodiment of the present invention, since the first region is in the first detection image and the second region is in the second detection image, the first region and the second region do not intersect. Therefore, whether the region of equipment defect and the region of abnormal heating intersect in the real world cannot be determined by whether the first region and the second region intersect. Therefore, the intersection of the region of equipment defect and the region of abnormal heating can be determined based on the coordinate range of the first region and the second region. If the coordinate range of the first region and the coordinate range of the second region do not intersect, the coordinates of the centroid of the first region in the first detection image (i.e., the average coordinates of the pixels in the first region in the first detection image) and the coordinates of the centroid of the second region in the second detection image (i.e., the average coordinates of the pixels in the second region in the second detection image) are both taken as abnormal locations that need to be focused on, which are the locations to be observed. For example, the coordinate range of the first region is 45-50 on the horizontal axis and 40-45 on the vertical axis. The second region has coordinates ranging from 35 to 40 on the x-axis and 30 to 35 on the y-axis. Since the coordinate ranges of the first and second regions do not intersect, and both represent anomalies, the coordinates of the centroid of the first region in the first detection image and the centroid of the second region in the second detection image are both used as the location information to be observed. Conversely, if the coordinate ranges of the first and second regions intersect, the average coordinates of the centroids of the first and second regions in the first and second detection images are used as the location information to be observed. In this case, the defect and the area of ​​abnormal heating can be considered close, indicating a correlation between the two anomalies. The average of the two coordinates can be used as the location information to be observed, thus simultaneously observing the anomalies in both regions. If no anomaly observation location exists (i.e., neither the first nor the second region exists), then it is determined that no additional branch inspection path is needed.

[0092] According to an embodiment of the present invention, in step S34, the multi-channel camera further includes a depth camera; determining the target shooting position of the inspection robot based on the location information to be observed, the i-th detection position, and the orientation of the multi-channel camera includes: determining a first distance between the actual observation position of the location information to be observed in real space and the multi-channel camera using the depth camera; determining the calibration parameters of the RGB camera and the infrared camera based on the coordinates of the i-th detection position in real space and the orientation of the multi-channel camera; determining the spatial coordinates of the actual observation position of the location information to be observed in real space based on the first distance and the calibration parameters of the RGB camera and the infrared camera; and determining the target shooting position of the inspection robot based on the spatial coordinates of the actual observation position.

[0093] According to an embodiment of the present invention, the multi-channel camera further includes a depth camera. The image captured by the depth camera is used to determine the distance between the multi-channel camera and the actual observation position of the location information to be observed in real space. Using the depth camera (e.g., a structured light camera, a TOF camera, etc.), the straight-line distance (i.e., the depth value) between the actual observation position of the location information to be observed in real space and the multi-channel camera can be measured, which is the first distance. Taking any point in the real world (e.g., the starting point of the inspection path) as the origin of the coordinate system, and determining the coordinates of the multi-channel camera in real space and the orientation angle of the multi-channel camera (e.g., the angle with the three coordinate axes of the coordinate system in the real world), the extrinsic parameters (e.g., rotation matrix and translation vector) of the RGB camera and the infrared camera can be determined. That is, the parameters that convert the coordinates in the coordinate system with the location of the multi-channel camera as the origin into the coordinates in the aforementioned coordinate system in the real world are used. The calibration parameters of the RGB camera and the infrared camera include the intrinsic and extrinsic parameters of the RGB camera and the infrared camera (the two have a high degree of integration, and it can be assumed that their intrinsic and extrinsic parameters are the same). The intrinsic parameters of the RGB and infrared cameras are their respective intrinsic parameters, and the intrinsic parameter matrix is ​​determined by the parameters set at the camera's factory (e.g., focal length, optical center, etc.). Using the SLAM algorithm, by substituting the coordinates and orientation angle of the observed position information in the camera coordinate system, the rotation matrix and translation vector of the camera's extrinsic parameters can be obtained, which are the camera's extrinsic parameters. The coordinates of the observed position information in a coordinate system with the location of the multiple cameras as the origin can be determined using the first distance, the coordinates of the observed position information, and the aforementioned intrinsic parameters. Then, using the extrinsic parameters, these coordinates can be converted to coordinates in the aforementioned real-world coordinate system to obtain the spatial coordinates of the actual observed position of the observed position information in real space.

[0094] According to an embodiment of the present invention, determining the target shooting position of the inspection robot based on the spatial coordinates of the actual observation position includes: obtaining the planar coordinates of the first undetermined shooting position corresponding to the k-th actual observation position with the j-th orientation at the i-th detection position according to formula (1). ,

[0095] (1)

[0096] in, Let be the spatial coordinates of the k-th actual observation position at the j-th orientation of the i-th detection position. Let i be the spatial coordinates of the i-th detection position. Let i be the planar coordinates of the (i+1)th detection position. For the height of multiple cameras, Preset observation angle;

[0097] According to formula (2), the planar coordinates of the second undetermined shooting position corresponding to the k-th actual observation position with the j-th orientation at the i-th detection position are obtained. ;

[0098] (2)

[0099] Determine the first shortest distance between the planar coordinates of the first undetermined shooting position and the planar coordinates of the i-th detection position and the (i+1)-th detection position, and determine the second shortest distance between the planar coordinates of the second undetermined shooting position and the planar coordinates of the i-th detection position and the (i+1)-th detection position.

[0100] If the first shortest distance is less than the second shortest distance, then the planar coordinates of the first undetermined shooting position are determined as the planar coordinates of the undetermined shooting position; otherwise, the planar coordinates of the second undetermined shooting position are determined as the planar coordinates of the undetermined shooting position. The planar coordinates of multiple undetermined shooting positions are clustered to obtain the cluster centers of multiple clusters. The multiple cluster centers are determined as the target shooting position.

[0101] According to an embodiment of the present invention, in formula (1), Let represent the vector pointing from the i-th detection position to the (i+1)-th detection position in the Cartesian coordinate system containing the ground. When the inspection robot photographs the actual observation position, it can be positioned directly in front of that position. A plane can be constructed through the actual observation position, and this plane intersects with the vector . Vertically, when the inspection robot is on this plane, it can be in front of the actual observation position. Since the inspection robot is on the ground, it can find the target shooting position on the straight line where the plane intersects the ground.

[0102] According to an embodiment of the present invention, the plane and vector where the target shooting position is located Vertical, therefore, the vector pointing from the target observation position to the target shooting position is perpendicular to... The dot product of two perpendicular vectors is 0. Therefore, with... The perpendicular vector is or .

[0103] According to an embodiment of the present invention, in formula (1), This represents the coordinates of the i-th detection position in the spatial coordinates perpendicular to the ground, i.e., the height of the robot's feet. Therefore, The sum of the height of the inspection robot's feet and the height of the multiple cameras represents the coordinates of the multiple cameras in their spatial coordinate system, perpendicular to the ground. Therefore, This can represent the height difference between the actual observation position and the multiple cameras. The angle (i.e., pitch angle) of the inspection robot towards the actual observation position in the vertical direction is... Therefore, when the inspection robot is at the target shooting position, the horizontal distance between it and the actual observation position is... The vector pointing from the target observation position to the target shooting position and... When the directions are the same, This can represent the distance along the horizontal x-axis between the inspection robot when it is at the target shooting position and the actual observation position. Therefore, This can represent the x-axis coordinate of the first undetermined shooting position of the target shooting location. Similarly, The y-axis coordinate of the first undetermined shooting position can be used to represent the target shooting position, thus obtaining the planar coordinates of the first undetermined shooting position.

[0104] According to an embodiment of the present invention, the above is a vector from the target observation position to the target shooting position. When the directions are the same, the planar coordinates of the first undetermined shooting position obtained can be used to obtain the vector pointing from the target observation position to the target shooting position in a similar manner. Planar coordinates of the second undetermined shooting position when the directions are the same .

[0105] According to an embodiment of the present invention, the distance between the plane coordinates of the first undetermined shooting position and the perpendicular line connecting the plane coordinates of the i-th detection position and the plane coordinates of the (i+1)-th detection position is determined as the first shortest distance, and the distance between the plane coordinates of the second undetermined shooting position and the perpendicular line connecting the plane coordinates of the i-th detection position and the plane coordinates of the (i+1)-th detection position is determined as the second shortest distance.

[0106] According to an embodiment of the present invention, if the first shortest distance is less than the second shortest distance, the planar coordinates of the first undetermined shooting position are determined as the planar coordinates of the undetermined shooting position; otherwise, the planar coordinates of the second undetermined shooting position are determined as the planar coordinates of the undetermined shooting position. Since the preset inspection path of the inspection robot includes a path from the i-th detection position to the (i+1)-th detection position, the actual observation position faces the inspection path. Based on the fact that the undetermined shooting position determined from the actual observation position is close to the inspection path, the coordinates corresponding to the smaller of the first and second shortest distances are determined as the undetermined shooting position.

[0107] According to embodiments of the present invention, based on the above method, the planar coordinates of the undetermined shooting position corresponding to multiple actual observation positions in multiple orientations at each detection position can be obtained. In images with different orientations, the same actual observation position may be captured, thus the corresponding undetermined shooting position can be determined based on the same actual observation position in images with different orientations. Furthermore, due to possible calculation errors (e.g., errors in extrinsic parameters), the undetermined shooting positions corresponding to the same actual observation position in different images may differ. On the other hand, there may also be cases where the undetermined shooting positions corresponding to different actual observation positions are close to each other. Therefore, the planar coordinates of multiple undetermined shooting positions can be clustered, grouping the planar coordinates of similar undetermined shooting positions into one category to obtain multiple clusters. The coordinates in each cluster are averaged to obtain the center of each cluster. Each cluster center is the corresponding optimal shooting position obtained after comprehensively considering all undetermined shooting positions in each cluster, and can be used as the target shooting position. The number of clusters can be manually specified or equal to the number of actual observation positions.

[0108] According to an embodiment of the present invention, in step S35, determining whether to add a branch inspection path based on the target shooting position and the path between the i-th detection position and the (i+1)-th detection position includes: determining a first vector between the i-th detection position and each target shooting position, and a second vector between the (i+1)-th detection position and each target shooting position; obtaining a first condition according to formula (3). Second condition ,

[0109] (3)

[0110] in, Let be the first vector corresponding to the shooting position of the s-th target. Let be the second vector corresponding to the shooting position of the s-th target. For vectors The line in question The area on the plane where the inspection target is located; if there exists an area that meets the first condition. Second condition If any one of the target shooting positions is determined, then the branch inspection path is added.

[0111] According to an embodiment of the present invention, the vector pointing from the i-th detection position to each target shooting position is the first vector. The vector pointing from the (i+1)-th detection position to each target shooting position is the second vector. In formula (3), Let the magnitude of the first vector corresponding to the s-th target's shooting position be considered as the distance from the s-th target's shooting position to the i-th detection position. Similarly, This can be considered as the distance from the s-th target shooting position to the (i+1)-th detection position. Under the first condition... In the middle, when When the distance from the s-th target shooting position to the i-th detection position is less than the distance from the s-th target shooting position to the (i+1)-th detection position, and the distance from the s-th target shooting position to the i-th detection position is shorter, it is more convenient to reach the s-th target shooting position from the i-th detection position compared to the (i+1)-th detection position. Therefore, a branch inspection path can be added at the i-th detection position. Under the second condition... middle, Representing vectors The intersection of the straight line and the plane region where the inspection target is located is not empty. That is, the route from the (i+1)th detection position to the (s)th target shooting position requires bypassing the inspection target, which is inconvenient. Therefore, a branch inspection path can be added at the i-th detection position. In other words, reaching the s-th target shooting position from the i-th detection position is more convenient than reaching it from the (i+1)-th detection position. Therefore, a branch inspection path is added at the i-th detection position. If there exists a path that satisfies the first condition... Second condition If any one of the target shooting positions is determined, then the branch inspection path is added.

[0112] In this way, the first distance between the actual observation position of the observed position information in real space and the multiple cameras can be determined based on the depth camera on the multiple cameras. Then, based on the coordinates of each detection position in real space, the orientation of the multiple cameras, the coordinates of the observed position information in the camera coordinate system, and the calibration parameters of the multiple cameras at each detection position, the spatial coordinates of the actual observation position of each observed position information in real space are obtained. Based on the spatial coordinates and orientation angles of the multiple cameras, the x- and y-axis components of the distance between the multiple cameras and each actual observation position are determined, thereby determining the first and second undetermined shooting positions. Undetermined shooting positions are then selected from these, and the target shooting position is obtained after clustering. Furthermore, based on the ease of reaching the target shooting position from the i-th detection position, it can be determined whether to add a branch inspection path. This improves the accuracy, objectivity, and comprehensiveness of determining whether to add a branch inspection path.

[0113] According to an embodiment of the present invention, in step S4, when it is determined that an additional branch inspection path is to be added, multiple branch detection positions are determined, and the branch inspection path is determined, including: determining the target shooting position as the branch detection position; determining the second distance between the i-th detection position and each branch detection position; arbitrarily selecting one branch detection position as the first branch detection position in the first round of sorting, and sorting each branch detection position in clockwise order to obtain the first sequence number of each branch detection position in the first round of sorting; obtaining the screening translation distance corresponding to the branch detection position with the first sequence number h according to formula (4). ,

[0114] (4)

[0115] in, The second distance corresponds to the branch detection position where the first sequence number is h. The second distance is the branch detection position corresponding to the first sequence number h+1, and when h=m. Where m is the number of branch detection positions. The second distance corresponds to the branch detection position where the first index is 1; The branch detection position corresponding to the first index when the minimum value is obtained is taken as the first branch detection position in the second round of sorting, and the branch detection positions are sorted in counterclockwise order to obtain the second index of each branch detection position in the second round of sorting; the i-th detection position is taken as the starting point of the branch inspection path, and the branch detection positions are connected in order of the second index, and the branch detection position with the second index m is connected to the i-th detection position to obtain the branch inspection path.

[0116] According to an embodiment of the present invention, all obtained target shooting positions are determined as positions that the branch inspection path needs to pass through and shoot at, i.e., branch detection positions. The length of the line connecting the i-th detection position and each branch detection position is determined, which is the second distance. One branch detection position is randomly selected as the first branch detection position in the first round of sorting, and each branch detection position is sorted in ascending order in a clockwise direction to obtain the sequence number of each branch detection position in the first round of sorting, which is the first sequence number.

[0117] According to an embodiment of the present invention, in formula (4), The sum of the second distance corresponding to the branch detection position with the first sequence number h and the second distance corresponding to the branch detection position with the first sequence number h+1 can be considered as the sum of the second distance corresponding to the branch detection position with the first sequence number h and the second distance corresponding to the next branch detection position in clockwise order. This can be used as the filtering translation distance corresponding to the branch detection position with the first sequence number h.

[0118] According to an embodiment of the present invention, the distance between each branch detection position remains constant. Therefore, the only factor affecting the shortest total distance of the branch inspection path is the sum of the distances from the starting point (i.e., the i-th detection position) to the first branch detection position and from the last branch detection position to the ending point (i.e., the i-th detection position). Thus, when the above-mentioned filtering translation distance is minimized, the sum of the second distance of the corresponding branch detection position and the second distance of the next branch detection position is minimized. In this case, the branch detection position is taken as the first branch detection position in the second round of sorting, and sorted counterclockwise. Then, the distances from the i-th detection position to the first branch detection position and from the last branch detection position to the i-th detection position are minimized, and the total distance of the branch inspection path is minimized. The i-th detection position can be taken as the starting point of the branch inspection path, and each branch detection position can be connected sequentially according to the second sequence number. The branch detection position with the second sequence number m can be connected to the i-th detection position to obtain the branch inspection path, i.e., the branch detection path with the shortest total distance.

[0119] In this way, the branch detection path with the shortest total distance can be selected based on the filtering translation distance, which improves the efficiency of inspection through branch inspection.

[0120] According to an embodiment of the present invention, in step S5, branch detection images of multiple orientations are acquired at multiple branch detection locations using multiple cameras. Similar to acquiring detection images, RGB and infrared images of each orientation can be acquired at each branch detection location using multiple cameras, which constitute the branch detection images.

[0121] According to an embodiment of the present invention, in step S6, alarm information is determined based on the branch detection images at multiple branch detection locations. The alarm information may include information such as the location of defects or abnormally high temperatures, thereby prompting maintenance personnel to perform maintenance.

[0122] According to an embodiment of the present invention, in step S7, after the safety inspection robot reaches the end of the branch inspection path, it continues to the (i+1)th detection position. After the safety inspection robot reaches the end of the branch inspection path (i.e., the i-th detection position), it can be considered that the inspection robot has completed the detection task on the branch inspection path and continues to the (i+1)-th detection position (i.e., the next detection position) to repeat similar detection.

[0123] According to an embodiment of the present invention, in step S8, if it is determined that no additional branch inspection path will be added, the system proceeds directly to the (i+1)th inspection position. If it is determined that no additional branch inspection path will be added, the system proceeds directly to the next inspection position, i.e., the (i+1)th inspection position.

[0124] In this way, alarm information can be determined based on branch detection images. The robot can then proceed to the next detection location after completing its branch inspection path task or when it is not necessary to add a branch inspection path at the current detection location. This improves the accuracy of path planning for safety inspection robots, enhancing the targeting and efficiency of safety inspections.

[0125] The safety inspection robot path planning method according to embodiments of the present invention can set multiple detection positions in the inspection path, and capture multiple orientation detection images at each detection position using multiple cameras, thereby determining whether to add a branch inspection path. If an addition is made, multiple branch detection positions and branch inspection paths are determined, and alarm information is determined based on the branch detection images. The planned path can be adjusted promptly based on anomalies encountered during inspection, reducing the possibility of overlooking safety risks, improving the accuracy of safety inspection robot path planning, and enhancing the targeting and efficiency of safety inspections. Furthermore, multiple detection positions can be evenly set in the inspection path, and RGB and infrared images of each detection position in each orientation can be captured using multiple cameras, comprehensively acquiring the defects and abnormal heating conditions of the inspected equipment, providing basic data for determining whether to add a branch inspection path. When determining whether to add a branch inspection path, the first distance between the actual observation position of the observed location information in real space and the multi-camera system can be determined based on the depth camera on the multi-camera system. Then, based on the coordinates of each detection position in real space, the orientation of the multi-camera system, the coordinates of the observed location information in the camera coordinate system, and the calibration parameters of the multi-camera system at each detection position, the spatial coordinates of the actual observation position of each observed location information in real space are obtained. Based on the spatial coordinates and orientation angles of the multi-camera system, the x- and y-axis components of the distance between the multi-camera system and each actual observation position are determined, thereby determining the first and second undetermined shooting positions. These undetermined shooting positions are then selected, and the target shooting position is obtained after clustering. Furthermore, based on the ease of reaching the target shooting position from the i-th detection position, it can be determined whether to add a branch inspection path. This improves the accuracy, objectivity, and comprehensiveness of determining whether to add a branch inspection path. Furthermore, the branch detection path with the shortest total distance can be selected based on the filtering translation distance, improving the efficiency of inspection through branch inspection. Furthermore, alarm information can be determined based on the branch detection images. After completing the branch inspection path task or if it is not necessary to add a branch inspection path at the current detection location, the robot proceeds to the next detection location. This improves the accuracy of path planning for the safety inspection robot, enhancing the targeting and efficiency of safety inspections.

[0126] Figure 3 An exemplary block diagram of a path planning system for a security inspection robot according to an embodiment of the present invention is shown, the system comprising:

[0127] The settings module allows you to set the starting point of the inspection path for the safety inspection robot and set multiple detection locations along the inspection path, where the starting point of the inspection path is the first detection location.

[0128] The detection image module acquires multiple orientation detection images at the i-th detection position through multiple cameras, including an RGB camera and an infrared camera. The detection images are a first detection image captured by the RGB camera and a second detection image captured by the infrared camera.

[0129] The judgment module determines whether to add a branch inspection path based on the detected image, the orientation of the multiple cameras, and the path between the i-th detection position and the (i+1)-th detection position.

[0130] The branch inspection path module determines multiple branch detection positions and the branch inspection path when it is determined that a branch inspection path will be added. The starting point and ending point of the branch inspection path are both the i-th detection position.

[0131] The branch detection image module acquires branch detection images from multiple directions at multiple branch detection locations using multiple cameras;

[0132] The alarm information module determines alarm information based on the branch detection images at multiple branch detection locations;

[0133] The first module continues to the (i+1)th inspection position after the safety inspection robot reaches the end of the branch inspection path.

[0134] The second module proceeds directly to the (i+1)th detection location if it determines that no additional branch inspection paths will be added.

[0135] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0136] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are merely examples and do not limit the present invention. The objectives of the present invention have been fully and effectively achieved. The functions and structural principles of the present invention have been demonstrated and explained in the embodiments, and any variations or modifications may be made to the implementation of the present invention without departing from the stated principles.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A path planning method for a safety inspection robot, characterized in that, include: Set the starting point of the inspection path for the safety inspection robot, and set multiple detection positions in the inspection path, where the starting point of the inspection path is the first detection position; At the i-th detection position, multiple detection images of different orientations are acquired through multiple cameras, including an RGB camera and an infrared camera. The detection images are a first detection image captured by the RGB camera and a second detection image captured by the infrared camera. Based on the detected image and the orientation of the multiple cameras, and the i-th detection position and the... The path between each detection location determines whether to add a branch inspection path; Given that an additional branch inspection path is to be added, multiple branch detection positions are determined, and the branch inspection path is determined, wherein the starting point and the ending point of the branch inspection path are both the i-th detection position. At multiple branch detection locations, multiple cameras are used to acquire branch detection images facing multiple directions; Based on the branch detection images at multiple branch detection locations, determine the alarm information; After the safety inspection robot reaches the end of the branch inspection path, it continues to the next... One detection location; If it is determined that no additional branch inspection routes will be added, proceed directly to the first... One detection location; Based on the detected image and the orientation of the multiple cameras, and the i-th detection position and the... The path between each detection location determines whether to add a branch inspection path, including: The first image detection model is used to process the first detection images in various orientations to obtain the first region where the defect is located. The second image detection model is used to process the second detection images in various orientations to obtain the second region of abnormal heating. Based on the first and second regions, determine the location information to be observed; Based on the location information to be observed, the i-th detection location, and the orientation of the multiple cameras, determine the target shooting location of the inspection robot; Based on the target shooting position, and the i-th detection position and the... The path between each detection location determines whether to add a branch inspection path; Based on the first and second regions, determine the location information to be observed, including: If the coordinate ranges of the first region and the second region do not intersect, then the coordinates of the centroid of the first region in the first detection image and the coordinates of the centroid of the second region in the second detection image are both used as the location information to be observed. If the coordinate ranges of the first region and the second region intersect, then the average coordinates of the centroid of the first region in the first detection image and the centroid of the second region in the second detection image are used as the location information to be observed. The multi-channel camera also includes a depth camera; Based on the location information to be observed, the i-th detection location, and the orientation of the multiple cameras, the target shooting position of the inspection robot is determined, including: Using a depth camera, determine the first distance between the actual observation location of the location information to be observed in real space and the multiple cameras; Based on the coordinates of the i-th detection position in real space and the orientation of the multiple cameras, determine the calibration parameters of the RGB camera and the infrared camera. Based on the first distance and the calibration parameters of the RGB camera and the infrared camera, determine the spatial coordinates of the actual observation position of the location information to be observed in the real space; The target shooting position of the inspection robot is determined based on the spatial coordinates of the actual observation location; Based on the target shooting position, and the i-th detection position and the... The path between each detection location determines whether to add a branch inspection path, including: Determine the first vector between the i-th detection position and each target capture position, and the second vector between the i-th detection position and the target capture position. The second vector between each detection location and each target shooting location; According to the formula ， Obtain the first condition Second condition ,in, Let be the first vector corresponding to the shooting position of the s-th target. Let be the second vector corresponding to the shooting position of the s-th target. For vectors The line in question The planar area where the inspection target is located; If there is a target shooting position that can meet any one of the first condition and the second condition a new branch inspection path is determined; If neither the first nor the second region exists, then it is determined that no additional branch inspection path is needed.

2. The path planning method for a safety inspection robot according to claim 1, characterized in that, Based on the spatial coordinates of the actual observation location, determine the target shooting position of the inspection robot, including: According to the formula ， Obtain the planar coordinates of the first undetermined shooting position corresponding to the k-th actual observation position with the j-th orientation at the i-th detection position. ,in, Let be the spatial coordinates of the k-th actual observation position at the j-th orientation of the i-th detection position. Let i be the spatial coordinates of the i-th detection position. For the first The planar coordinates of each detection location, For the height of multiple cameras, Preset observation angle; According to the formula , Obtain the planar coordinates of the second undetermined shooting position corresponding to the k-th actual observation position with the j-th orientation at the i-th detection position. ; Determine the planar coordinates of the first undetermined shooting position and the planar coordinates of the i-th detection position and the... The first shortest distance between the planar coordinates of the i-th detection position and the planar coordinates of the second undetermined shooting position is determined, and the planar coordinates of the i-th detection position and the i-th detection position are determined. The second shortest distance between the planar coordinates of each detection location; If the first shortest distance is less than the second shortest distance, then the planar coordinates of the first undetermined shooting position are determined as the planar coordinates of the undetermined shooting position; otherwise, the planar coordinates of the second undetermined shooting position are determined as the planar coordinates of the undetermined shooting position. Cluster the planar coordinates of multiple undetermined shooting locations to obtain the cluster centers of multiple clusters; Multiple cluster centers were identified as target shooting locations.

3. The path planning method for a safety inspection robot according to claim 1, characterized in that, Given the decision to add branch inspection paths, determine multiple branch inspection locations and the branch inspection paths, including: The target shooting location is determined as the branch detection location; Determine the second distance between the i-th detection position and each branch detection position; Choose any branch detection position as the first branch detection position in the first round of sorting, and sort each branch detection position in clockwise order to obtain the first sequence number of each branch detection position in the first round of sorting; According to the formula , Obtaining the first sequence number is The filtering translation distance corresponding to the branch detection position ,in, The first serial number is The second distance corresponding to the branch detection position. The first serial number is The second distance corresponding to the branch detection position, and in In this case, ,in, The number of branch detection locations, The second distance is the branch detection position corresponding to the first sequence number 1; Will The branch detection position corresponding to the first index when the minimum value is obtained is taken as the first branch detection position in the second round of sorting, and the branch detection positions are sorted in counterclockwise order to obtain the second index of each branch detection position in the second round of sorting. Take the i-th detection position as the starting point of the branch inspection path, connect each branch detection position sequentially according to the second sequence number, and set the second sequence number as... The branch detection position is connected to the i-th detection position to obtain the branch inspection path.

4. A path planning system for a safety inspection robot, characterized in that, include: The settings module allows you to set the starting point of the inspection path for the safety inspection robot and set multiple detection locations along the inspection path, where the starting point of the inspection path is the first detection location. The detection image module acquires multiple orientation detection images at the i-th detection position through multiple cameras, including an RGB camera and an infrared camera. The detection images are a first detection image captured by the RGB camera and a second detection image captured by the infrared camera. The judgment module, based on the detected image and the orientation of the multiple cameras, and the relationship between the i-th detection position and the... The path between each detection location determines whether to add a branch inspection path; The branch inspection path module determines multiple branch detection positions and the branch inspection path when it is determined that a branch inspection path will be added. The starting point and ending point of the branch inspection path are both the i-th detection position. The branch detection image module acquires branch detection images from multiple directions at multiple branch detection locations using multiple cameras; The alarm information module determines alarm information based on the branch detection images at multiple branch detection locations; The first module continues to the next module after the safety inspection robot reaches the end of the branch inspection path. One detection location; The second module, if it is determined that no additional branch inspection paths will be added, will directly proceed to the first... One detection location; Based on the detected image and the orientation of the multiple cameras, and the i-th detection position and the... The path between each detection location determines whether to add a branch inspection path, including: The first image detection model is used to process the first detection images in various orientations to obtain the first region where the defect is located. The second image detection model is used to process the second detection images in various orientations to obtain the second region of abnormal heating. Based on the first and second regions, determine the location information to be observed; Based on the location information to be observed, the i-th detection location, and the orientation of the multiple cameras, determine the target shooting location of the inspection robot; Based on the target shooting position, and the i-th detection position and the... The path between each detection location determines whether to add a branch inspection path; Based on the first and second regions, determine the location information to be observed, including: If the coordinate ranges of the first region and the second region do not intersect, then the coordinates of the centroid of the first region in the first detection image and the coordinates of the centroid of the second region in the second detection image are both used as the location information to be observed. If the coordinate ranges of the first region and the second region intersect, then the average coordinates of the centroid of the first region in the first detection image and the centroid of the second region in the second detection image are used as the location information to be observed. The multi-channel camera also includes a depth camera; Based on the location information to be observed, the i-th detection location, and the orientation of the multiple cameras, the target shooting position of the inspection robot is determined, including: Using a depth camera, determine the first distance between the actual observation location of the location information to be observed in real space and the multiple cameras; Based on the coordinates of the i-th detection position in real space and the orientation of the multiple cameras, determine the calibration parameters of the RGB camera and the infrared camera. Based on the first distance and the calibration parameters of the RGB camera and the infrared camera, determine the spatial coordinates of the actual observation position of the location information to be observed in the real space; The target shooting position of the inspection robot is determined based on the spatial coordinates of the actual observation location; Based on the target shooting position, and the i-th detection position and the... The path between each detection location determines whether to add a branch inspection path, including: Determine the first vector between the i-th detection position and each target capture position, and the second vector between the i-th detection position and the target capture position. The second vector between each detection location and each target shooting location; According to the formula , Obtain the first condition Second condition ,in, Let be the first vector corresponding to the shooting position of the s-th target. Let be the second vector corresponding to the shooting position of the s-th target. For vectors The line in question The planar area where the inspection target is located; If there exists that satisfies the first condition Second condition If any one of the target shooting positions is determined, then the branch inspection path is determined; If neither the first nor the second region exists, then it is determined that no additional branch inspection path is needed.

Images