An autonomous positioning and mapping method for inspection robots based on plant and station semantic information

Abstract

The present invention discloses an autonomous positioning and mapping method for an inspection robot based on semantic information of a plant or station, comprising data annotation of plant or station images, construction of a specific semantic image dataset applicable to a variety of image segmentation and target detection algorithms, combination of target detection, optical flow motion consistency detection and ORB feature extraction, and consideration of pure vision and visual odometer schemes combining vision with IMU, thereby maximizing the accuracy of the inspection robot in calculating its own posture in dynamic scenes, integrating a dual-channel coordinate attention mechanism into a real-time semantic segmentation network BiSeNet to optimize its network structure, and mapping the image segmentation results into the ultimately generated point cloud, thereby completing the construction of a semantic point cloud map and a semantic octree map and realizing map reuse. The present invention reduces the inspection robot's dependence on the network, so that its positioning and subsequent path planning tasks no longer need to rely on network maps.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an autonomous positioning and mapping method for an inspection robot based on semantic information of a plant or station. Background Art

[0002] The development of smart grids has become a key strategic component of the nation's power industry development. Equipment health monitoring and the development of intelligent inspection equipment have become hot topics in recent years, demonstrating enormous growth potential and development opportunities. Routine inspection and maintenance of power equipment are fundamental to ensuring reliable and stable grid operation. Promptly understanding equipment anomalies and proactively managing production risks are key research areas for various power sectors. Compared to manual inspections, which can lead to missed inspections, unauthorized inspections, and non-compliant inspections, automated inspections using inspection robots can significantly reduce errors, avoid dangers and accidents, and enhance inherent safety on-site.

[0003] In order for inspection robots to operate autonomously in scenarios like power plants and substations, they must possess positioning and perception capabilities. However, in these environments, network signals are poor, network maps are unusable, GPS signals are weak, data drift is significant indoors, and local map perception is impossible. Therefore, inspection robots must be able to autonomously construct maps and perform positioning. Summary of the Invention

[0004] The purpose of this section is to summarize some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of this application to avoid blurring the purpose of this section, the abstract and the title of the invention, and such simplifications or omissions should not be used to limit the scope of the present invention.

[0005] In view of the above problems and / or problems existing in existing inspection robots, the present invention is proposed.

[0006] Therefore, the purpose of the present invention is to provide an autonomous positioning and mapping method for inspection robots based on plant and station semantic information. By constructing multiple maps containing semantic information, map reuse is achieved without relying on network maps, reducing dependence on the network and improving the accuracy of inspection robots.

[0007] To solve the above technical problems, according to one aspect of the present invention, the present invention provides the following technical solutions:

[0008] A method for autonomous positioning and mapping of an inspection robot based on semantic information of a plant or station, comprising:

[0009] S1. Label the plant and station images and construct a specific semantic image dataset that can be applied to various image segmentation and target detection algorithms;

[0010] S2. By combining feature extraction with target detection and optical flow calculation, the pose calculation of the inspection robot itself in dynamic scenes is optimized;

[0011] S3. Integrate the dual-channel coordinate attention mechanism into the real-time semantic segmentation network BiSeNet to form an improved BiSeNet network structure;

[0012] S4. Based on the constructed specific semantic image dataset, the optimized posture calculation of the inspection robot's own motion in dynamic scenes, and the improved BiSeNet network structure, the ORB-SLAM3 algorithm combining vision and IMU is used to construct a static point cloud map.

[0013] As a preferred solution of the autonomous positioning and mapping method of an inspection robot based on plant station semantic information described in the present invention, step S1 also includes preprocessing the brightness and clarity of the image, and the preprocessing of brightness and clarity includes: increasing brightness, reducing brightness, flipping, reducing clarity and increasing noise.

[0014] As a preferred embodiment of the autonomous positioning and mapping method for an inspection robot based on plant station semantic information described in the present invention, in step S2, the steps of optimizing the posture calculation of the inspection robot's own motion in a dynamic scene by combining feature extraction with target detection and optical flow calculation are as follows:

[0015] Optical flow calculation: Obtain image frames, and restore the relative position change between the two frames by matching the optical flow feature points between the previous and next frames. Then, calculate the basic matrix F based on the above optical flow feature point set.

[0016] After obtaining the basic matrix, determine whether a certain optical flow point in the current frame is a dynamic point;

[0017] Object detection: Using the YOLOv5s object detection network, the keyframe images used for pose calculation are normalized and resized, and pixel values ​​are mapped to a range of 0-1 as input parameters for object detection. Network inference generates the following data: Detection Result, bbox, score, and class_id. The bbox is the two diagonal coordinates that describe a rectangle storing the object's position in the image. The score is the probability of the identified object class, and the class_id is the class ID.

[0018] Feature extraction: The left eye image of the binocular camera or the color image of the RGB-D camera is used as the input for feature extraction, that is, FAST corner points are extracted and BRIEF descriptors are calculated. The corner points and descriptors are combined to form ORB feature points.

[0019] Perform dynamic feature point judgment: If the dynamic optical flow point calculated by the optical flow is located in the bbox coordinate area of ​​the target detection output whose class_id is the dynamic category, the object is judged to be in motion, and then the ORB feature points in the area are deleted. Conversely, if the dynamic optical flow point is not in the bbox area, these ORB feature points are retained.

[0020] As a preferred solution of the autonomous positioning and mapping method of an inspection robot based on plant station semantic information described in the present invention, after obtaining the basic matrix, the steps of determining whether a certain optical flow point in the current frame is a dynamic point are as follows: determining whether the distance D between the current frame p2 and the epipolar line is less than the empirical constant δ; if so, it is defined as a static optical flow point; otherwise, it is defined as a dynamic optical flow point;

[0021] Among them, the epipolar line refers to a straight line that must pass through the point p2 in the current frame. The calculation formula of D is:

[0022]

[0023] Among them, P1 is the coordinate point of the previous frame on the normalized plane.

[0024] As a preferred solution of the autonomous positioning and mapping method of an inspection robot based on plant and station semantic information described in the present invention, the correct matching points are screened out from the set of matching point pairs containing erroneous points through RANSAC for the obtained basic matrix F.

[0025] As a preferred solution of the autonomous positioning and mapping method of an inspection robot based on plant station semantic information described in the present invention, in step S3, the construction of coordinate attention includes:

[0026] Coordinate information embedding: Use a pooling kernel of size (H, 1) or (1, W) to encode each channel along the x-axis and y-axis respectively. The output of the c-th channel with height h is:

[0027]

[0028] The output of the cth channel with width w is:

[0029]

[0030] Attention Generation: Connection and Two feature maps are passed to a 1×1 shared convolution transformation function F1, producing:

[0031] f=δ(F1([z h ,z w ]));

[0032] Among them, [z h ,z w ] represents the connection operation along the spatial dimension, δ is the nonlinear activation function;

[0033] Refer to the reduction rate r of the control module size in SE Attention and split f into two separate tensors along the spatial dimension. and Then use two 1×1 convolution f h and f w The feature map f h and f w Transform to the same number of channels as input X and get:

[0034]

[0035] Among them, g h and g w That is, as the attention weight, the output expression of the coordinate attention module is:

[0036]

[0037] As a preferred solution of the autonomous positioning and mapping method of an inspection robot based on plant station semantic information described in the present invention, a real-time depth generation algorithm is also included. The real-time depth generation algorithm is constructed as follows:

[0038] The image is preprocessed using the horizontal Sobel operator, and each pixel on the processed image is mapped to

[0039] Projected into a new image, the pixel value of the new image is defined as:

[0040]

[0041] Where P is the pixel value of the point, preFilterCap is a constant parameter, and the gradient information of the image is obtained;

[0042] When the pixel position is offset by b from left to right, and the disparity of pixel p is d, the minimum cost value L is b (p,d) is expressed as:

[0043]

[0044] The parameters P1 and P2 are defined as:

[0045]

[0046] C(p,d) is calculated by the following formula:

[0047]

[0048] Between the current pixel p and the pixel q after shifting d, after half-pixel interpolation, find the minimum grayscale or RGB difference between the two pixels and use it as the value of C(p,d);

[0049] Calculate L for the eight neighborhoods of the pixel according to the above steps. b (p, d), then accumulate the 8 cost values, select the disparity with the smallest cumulative cost as the final disparity of the pixel, accumulate the 8 cost values ​​for each pixel, select the disparity with the smallest cumulative cost as the final disparity of the pixel, and form the disparity map of the entire image;

[0050] According to the obtained disparity map, the depth of the entire image is restored, where a point p on the disparity map c (u,v) corresponds to p in three-dimensional space w The (x,y,z) coordinates are given by the following formula:

[0051]

[0052] where f x 、f y 、c x 、c y is the camera intrinsic matrix parameter, baseline is the distance between the optical centers of the left and right cameras, and desparity.at(v,u) is the value of the coordinate (u,v) corresponding to point p on the disparity map.

[0053] As a preferred solution of the autonomous positioning and mapping method of the inspection robot based on plant station semantic information described in the present invention, when screening the effective map point p w When , the voxels corresponding to the dynamic category of semantic segmentation are deleted.

[0054] Compared with the existing technology, the present invention has the following beneficial effects: first, the present invention constructs a specific semantic image data set that can be applied to a variety of image segmentation and target detection algorithms by data annotation of plant and station pictures. Secondly, the ORB-SLAM3 algorithm combining vision and IMU is adopted, and in the feature extraction stage, target detection and optical flow calculation threads are added to further determine whether the object is in motion. While effectively avoiding the calculation of posture being affected by moving objects, the static feature points are retained to the maximum extent, thereby improving the accuracy of the calculated posture. Thirdly, the dual-channel coordinate attention is integrated into BiSeNet, and while constructing a sparse feature point map, different power equipment are distinguished and a semantic point cloud map is generated. Finally, the point cloud is optimized into an octree map, which reduces the map storage size while more effectively improving the reusability of the map. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below in conjunction with the accompanying drawings and detailed embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without inventive efforts. Among them:

[0056] Figure 1 This is an example of the plant image annotation of the present invention;

[0057] Figure 2 This is a comparison chart of the effects of data enhancement performed on image datasets by the present invention;

[0058] Figure 3 This is a flowchart of posture calculation optimization in dynamic scenes of the present invention;

[0059] Figure 4 Schematic diagram of the improved BiSeNet network structure of the present invention;

[0060] Figure 5 This is a schematic diagram of the comparison of dynamic / static scene point cloud maps of the present invention. DETAILED DESCRIPTION

[0061] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0062] Next, the present invention is described in detail with reference to schematic diagrams. For ease of illustration, cross-sectional views of device structures may be partially enlarged and not to scale when describing the embodiments of the present invention. Furthermore, the schematic diagrams are merely illustrative and should not limit the scope of protection of the present invention. Furthermore, in actual production, three-dimensional dimensions, including length, width, and depth, should be included.

[0063] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0064] Currently, inspection robots operate autonomously in power plants, substations, and other related scenarios, mainly relying on network maps. However, in these environments, network signals are poor, network maps cannot be used, GPS signals are weak, data drift is large indoors, and local map perception is impossible. Therefore, inspection robots need to have the ability to autonomously build maps and locate themselves.

[0065] Based on this, the present invention provides an autonomous positioning and mapping method for inspection robots based on plant and station semantic information. By constructing multiple maps containing semantic information, map reuse is achieved without relying on network maps, reducing dependence on the network and improving the accuracy of the inspection robots.

[0066] This autonomous positioning and mapping method for inspection robots based on plant and station semantic information mainly includes four stages:

[0067] S1. Label the plant and station images and construct a specific semantic image dataset that can be applied to various image segmentation and target detection algorithms;

[0068] S2. By combining feature extraction with target detection and optical flow calculation, the pose calculation of the inspection robot itself in dynamic scenes is optimized;

[0069] S3. Integrate the dual-channel coordinate attention mechanism into the real-time semantic segmentation network BiSeNet to form an improved BiSeNet network structure;

[0070] S4. Based on the constructed specific semantic image dataset, the optimized posture calculation of the inspection robot's own motion in dynamic scenes, and the improved BiSeNet network structure, the ORB-SLAM3 algorithm combining vision and IMU is used to construct a static point cloud map.

[0071] In step S1, to ensure accurate target detection and image segmentation in special scenarios such as power plants, it is necessary to construct a corresponding image dataset. That is, all pixels in the image are categorized and labeled, and semantic or instance labels are set. According to the different equipment, devices, and personnel in the power plant, a total of 44 categories can be divided. See Table 1 for details.

[0072] Table 1 Classification of plant image dataset

[0073]

[0074]

[0075] In particular, considering the differences in equipment and facilities in different plants and stations, exclusive image data collection and annotation should be carried out for different scenes to ensure that the inspection robot can map and locate more accurately in that specific scene. Figure 1 Shows examples of semantic annotation of some images in the dataset.

[0076] In order to simulate the diversity of plant environments, improve the integrity of the data set, and enhance the generalization of neural network reasoning, the image brightness and clarity are preprocessed, and the data set is expanded. Figure 2As shown in the figure, the preprocessing of brightness and clarity includes: (a) original image, (b) improving brightness, (c) reducing brightness, (d) flipping, (e) reducing clarity, and (f) adding noise. At this point, the plant station semantic image dataset is constructed. According to different tasks such as target detection, semantic and instance segmentation, different model files can be obtained by training with this network dataset for various image inference requirements in SLAM work.

[0077] Among them, in step S2, a three-thread concurrent algorithm (optical flow calculation, target detection and feature extraction) is designed to optimize the posture calculation of the inspection robot's own motion in dynamic scenes, such as Figure 3 shown to reduce errors.

[0078] Thread 1 (optical flow calculation) first recovers the relative position change between the two frames by matching the optical flow feature points between the previous and next frames, and then calculates the fundamental matrix F through the above optical flow feature point set.

[0079] Assume that there are sub-pixel corner points that match each other in the previous and next frames, and their coordinates are expressed as p1(u1,v1,1) in the normalized plane. T and p2(u2,v2,1) T Then Fp1 represents a straight line (polar line) that must pass through point p2 in the current frame. That is, all matching points of the previous and next frames, p1 and p2, satisfy:

[0080] p2 T Fp1=0(1)

[0081] For the estimation of F, incorrectly matched points will cause the estimated value to deviate greatly from the true value. Therefore, the RANSAC method is used to filter out the correct matching points (inliers) from the set of matching point pairs containing incorrect points (outliers).

[0082] After obtaining the basic matrix, each static point p of the two frames before and after i All of them should satisfy the constraints of equation (1). Since the basic matrix is ​​an estimate, the mapping of point pairs cannot strictly satisfy the equation, that is, the coordinate p2 in the current frame is not necessarily on the epipolar line Fp1. At this time, it is determined whether the distance D of the epipolar line of p2 is less than the empirical constant δ. If it is less than the threshold, it is defined as a static optical flow point, otherwise it is defined as a dynamic optical flow point. The distance D is described by equation (2):

[0083]

[0084] From this we can determine whether a certain optical flow point in the current frame is a dynamic point.

[0085] Thread 2 (object detection) uses the YOLOv5s object detection network to identify various dynamic objects in images, including but not limited to people and vehicles, in real time, frame by frame. Specifically, the keyframe images used for pose calculation are normalized and resized, and pixel values ​​are mapped to a range of 0-1 as input parameters for this thread. Through network inference, data with the structure shown in Table 2 is generated.

[0086] Table 2 Target detection output

[0087] DetectionResult bbox score class_id

[0088] Where bbox is the two diagonal coordinates that describe a rectangle and store the location of the object in the image, score is the probability of the identified object class, and class_id is the class ID. Depending on the scenario and dataset, class_id needs to be modified accordingly.

[0089] Thread 3 (feature extraction) first performs the ORB feature extraction task: the left eye image of the binocular camera or the color image of the RGB-D camera is used as the input of this thread, that is, the FAST (Features from Accelerated SegmentTest) corner points are extracted and the BRIEF descriptors are calculated. The corner points and descriptors are combined to form ORB feature points. The dynamic feature point determination task is then performed: if the dynamic optical flow point of thread 1 is within the bbox coordinate area of ​​the dynamic category output by thread 2, the object is determined to be in motion and the ORB feature points in this area are subsequently deleted. Conversely, if the dynamic optical flow point is not within the bbox area, these ORB feature points are retained. This effectively optimizes the ORB-SLAM pose calculation while retaining as many valid feature points as possible.

[0090] In step S3, a dual-channel coordinate attention mechanism is integrated into the real-time semantic segmentation network BiSeNet to generate semantic point cloud images, accumulate them into a point cloud map, and ultimately convert them into an octree map more suitable for navigation and object tracking. BiSeNet is a bilateral segmentation network that combines spatial and contextual paths. The bilateral structure is used to address spatial information loss and reduced receptive field, respectively. A feature fusion module is used to effectively combine features from these two paths, ultimately forming a complete semantic segmentation network framework.

[0091] Its feature fusion module and attention refinement module use a channel-wise attention structure similar to SE (Squeeze-and-Excitation) Attention. This structure takes into account the limited computational power of lightweight networks. However, the attention features derived from 2D global pooling only consider the encoding of inter-channel information and ignore the importance of positional information, which is crucial for semantic segmentation tasks that require capturing target structure. Therefore, this section introduces dual-channel coordinate attention to construct a new network model that further improves semantic segmentation while ensuring real-time performance.

[0092] Coordinate attention is divided into two steps: coordinate information embedding and coordinate attention generation. The coordinate information embedding module allows the location information of the feature map to be stored in the network for a long time. First, global pooling is decomposed into two one-dimensional feature encoding operations: using pooling kernels of size (H, 1) or (1, W) to encode each channel along the x-axis and y-axis respectively. The output of the c-th channel with height h can be expressed as:

[0093]

[0094] Similarly, the output of the cth channel with width w can be written as:

[0095]

[0096] The above two transformations realize feature aggregation along two spatial directions respectively, and obtain a pair of direction-aware feature maps, which greatly simplifies the network's computational workload for searching the region of interest and can achieve precise positioning. Its effect is far better than the SE module that outputs a single feature vector.

[0097] Obtained and This completes the task of the coordinate information embedding module, and then generates the attention map through the coordinate attention generation module. First, connect and The two feature maps are then passed to a 1×1 shared convolution transformation function F1, producing:

[0098] f=δ(F1([z h ,z w ]))(5)

[0099] Among them, [z h ,z w ] represents the connection operation along the spatial dimension, and δ is a nonlinear activation function. Next, referring to the reduction rate r of the control module size in SEAttention, f is split into two separate tensors along the spatial dimension and Then use two 1×1 convolution F h and F w The feature map f h and f w Transform to the same number of channels as the input X and obtain the result of formula (6).

[0100]

[0101] Among them, g h and g w That is, as the attention weight. At this point, the output of the coordinate attention module is expressed as:

[0102]

[0103] After completing the construction of coordinate attention, the present invention uses the feature fusion method based on BiSeNet to pass the features into the coordinate attention module after 16 times and 32 times downsampling of the context path, thereby retaining the position information of the feature map and optimizing the expression of the original attention refinement module of BiSeNet. At the same time, after the convolution operation of the feature fusion module, the coordinate attention is connected to further achieve the precise positioning of the area of ​​interest, reduce the amount of network calculation, and form the following Figure 4 The network structure further improves the reasoning performance of the network model.

[0104] In step S4, the visual SLAM algorithm is ORB-SLAM3. Compared with ORB-SLAM2, it adds a visual-IMU mode, which helps reduce problems such as posture drift caused by rapid movement or camera shake. Compared with pure visual SLAM, combining IMU data with visual data can bring the following advantages:

[0105] More robust: IMU data enables ORB-SLAM3 to better handle camera motion estimation in moving scenes and calculate the camera's acceleration and angular velocity, better predicting the camera's motion state. This enables ORB-SLAM3 to better handle problems such as fast movement, vibration, and motion blur.

[0106] Higher positioning accuracy: IMU data can provide acceleration and angular velocity information of the camera in three-dimensional space, thereby improving ORB-SLAM3's ability to predict the camera's motion state and improving positioning accuracy.

[0107] Wider application scenarios: Based on the improvement of robustness and positioning accuracy, ORB-SLAM3 can be applied to more complex environments or equipment, such as high-dynamic scenes and mounted on high-speed drones.

[0108] Furthermore, generating a point cloud map requires three parameters: the robot's current position, an RGB image, and depth information. While depth information can be easily obtained using RGB-D cameras using laser or pulse cameras, the maximum depth achieved by these cameras is generally only around 10 meters. To enable inspection robots to easily obtain depth information for each pixel even in relatively open areas, the present invention also uses a binocular camera as input. However, the ORB-SLAM3 algorithm cannot directly obtain depth from binocular images. Therefore, a real-time depth generation algorithm was designed to address this issue.

[0109] First, we use the SGBM algorithm to better express the complete disparity value of the entire image. Its core is the SGM

[76] (Semi-Global Matching) algorithm, which sets a disparity map consisting of the disparity of each pixel. We use the horizontal Sobel operator to preprocess the image, and then map each pixel on the processed image into a new image. The pixel value of the new image is defined as:

[0110]

[0111] Where P is the pixel value of the point, and preFilterCap is a constant parameter, from which the gradient information of the image is obtained. Based on the sampling method, we divide the cost to be calculated into the gradient cost generated by the image gradient information and the SAD cost of the original image. When the pixel position is offset by b from left to right, when the disparity of pixel p is d, its minimum cost value L b (p,d) is expressed as:

[0112]

[0113] The parameters P1 and P2 are defined as:

[0114]

[0115] C(p,d) is calculated by the following formula:

[0116]

[0117] After half-pixel interpolation between the current pixel p and the pixel q shifted by d, the minimum grayscale or RGB difference between the two pixels is found. This is used as the value of C(p, d), which is the minimum cost for a pixel in one direction (from left to right) when taking a certain disparity value. This method is then applied to each of the pixel's eight neighborhoods. The eight cost values ​​are then accumulated, and the disparity with the minimum cumulative cost is selected as the final disparity for that pixel. This operation is repeated for each pixel to form a disparity map for the entire image.

[0118] After obtaining the disparity map, the depth of the entire image can be restored. c (u,v) corresponds to the coordinate p in three-dimensional space w (x,y,z) can be obtained by the following formula:

[0119]

[0120] where f x 、f y 、c x 、c y is the camera intrinsic matrix parameter, baseline is the distance between the optical centers of the left and right cameras, and desparity.at(v,u) is the value of the coordinate (u,v) corresponding to point p on the disparity map.

[0121] The construction of this map requires the semantic segmentation results, key frame images and the pose of the corresponding frame obtained above. The category information and spatial coordinates p of each pixel obtained by semantic segmentation are w (x, y, z) correspond one to one, so that the semantics of each voxel in the world coordinate system can be obtained, thereby constructing a semantic map.

[0122] When there are dynamic objects such as people and vehicles in the scene, it will seriously interfere with the construction of the point cloud map, causing the 3D scene to become blurred, and these types of objects are very easy to change relative to the scene map. Therefore, it is necessary to delete these objects when generating the point cloud. w When , the voxels corresponding to the dynamic category of semantic segmentation should be deleted, so as to generate a point cloud containing only static scenes, such as Figure 5 As shown, the left side does not use pose optimization and does not remove dynamic category voxels, the middle side uses pose optimization but does not remove dynamic category voxels, and the right side uses pose optimization and removes dynamic category voxels.

[0123] In summary, the present invention first constructs a specific semantic image dataset that can be applied to a variety of image segmentation and target detection algorithms by data annotation of plant and station images. Secondly, the ORB-SLAM3 algorithm combining vision and IMU is adopted, and in the feature extraction stage, target detection and optical flow calculation threads are added to further determine whether the object is in motion. While effectively avoiding the calculation of posture being affected by moving objects, the static feature points are retained to the maximum extent, thereby improving the accuracy of the calculated posture. Thirdly, the dual-channel coordinate attention is integrated into BiSeNet, and while constructing a sparse feature point map, different power equipment are distinguished and a semantic point cloud map is generated. Finally, the point cloud is optimized into an octree map, which reduces the map storage size while more effectively improving the reusability of the map.

[0124] Moreover, the data set constructed by the present invention can achieve almost full pixel semantic coverage for the plant scene where the inspection robot is located. Therefore, when constructing a point cloud map, the map points passed to the point cloud mapping thread can not only transmit the RGB value of the original image, but also transmit category color information. In this way, a full semantic scene can be constructed, and a variety of power equipment can be highlighted. Furthermore, by converting the point cloud voxels into an octree structure, a more excellent three-dimensional environment model can be constructed. The octree map is based on the octree structure and uses probabilistic occupancy estimation. It can clearly represent the space occupied by voxels and occupies much less hard disk space than the point cloud. It can also play a better role in tasks such as path planning and navigation of the inspection robot.

[0125] Although the present invention has been described above with reference to embodiments, various modifications may be made thereto and equivalent components may be substituted without departing from the scope of the present invention. In particular, as long as there are no structural conflicts, the various features of the embodiments disclosed herein may be combined with each other in any manner, and the omission of an exhaustive description of such combinations in this specification is solely for the sake of space and resource conservation. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A method for autonomous positioning and mapping of inspection robots based on plant and station semantic information, characterized in that: include: S1. Label the plant and station images and construct a specific semantic image dataset that can be applied to various image segmentation and target detection algorithms; S2. By combining feature extraction with target detection and optical flow calculation, the pose calculation of the inspection robot itself in dynamic scenes is optimized; S3. Integrate the dual-channel coordinate attention mechanism into the real-time semantic segmentation network BiSeNet to form an improved BiSeNet network structure. The specific improvements are as follows: Based on the feature fusion method of BiSeNet, the features are passed to the coordinate attention module after 16 times and 32 times downsampling of the context path, respectively, to preserve the position information of the feature map and optimize the expression of the original attention refinement module of BiSeNet; S4, based on the constructed specific semantic image dataset, the optimized pose calculation of the inspection robot's own motion in dynamic scenes, and the improved BiSeNet network structure, the ORB-SLAM3 algorithm combining vision and IMU is used to construct a static point cloud map; In step S2, the steps for optimizing the posture calculation of the inspection robot in dynamic scenes by combining feature extraction with target detection and optical flow calculation are as follows: Optical flow calculation: Obtain image frames, and restore the relative position change between the two frames by matching the optical flow feature points between the previous and next frames. Then, calculate the basic matrix F based on the above optical flow feature point set. After obtaining the basic matrix, determine whether a certain optical flow point in the current frame is a dynamic point; Object detection: Using the YOLOv5s object detection network, the keyframe images used for pose calculation are normalized and resized, and pixel values ​​are mapped to a range of 0-1 as input parameters for object detection. Network inference generates the following data: Detection Result, bbox, score, and class_id. The bbox is the two diagonal coordinates that describe a rectangle storing the object's position in the image. The score is the probability of the identified object class, and the class_id is the class ID. Feature extraction: The left eye image of the binocular camera or the color image of the RGB-D camera is used as the input for feature extraction, that is, FAST corner points are extracted and BRIEF descriptors are calculated. The corner points and descriptors are combined to form ORB feature points. Perform dynamic feature point judgment: If the dynamic optical flow point calculated by the optical flow is located in the bbox coordinate area of ​​the target detection output whose class_id is the dynamic category, the inspection robot is judged to be in motion, and then the ORB feature points in the area are deleted. Conversely, if the dynamic optical flow point is not in the bbox area, these ORB feature points are retained.

2. The autonomous positioning and mapping method of an inspection robot based on plant and station semantic information according to claim 1 is characterized in that: The step S1 further includes pre-processing the brightness and clarity of the image, wherein the pre-processing of the brightness and clarity includes: increasing brightness, decreasing brightness, flipping, decreasing clarity, and increasing noise.

3. The autonomous positioning and mapping method of an inspection robot based on plant and station semantic information according to claim 1 is characterized in that: After obtaining the basic matrix, the steps to determine whether a certain optical flow point in the current frame is a dynamic point are as follows: determine whether the distance D between the current frame p2 and the epipolar line is less than the empirical constant δ. If it is less than the empirical constant δ, it is defined as a static optical flow point, otherwise it is defined as a dynamic optical flow point; Among them, the epipolar line refers to a straight line that must pass through the point p2 in the current frame. The calculation formula of D is: Among them, p1 is the coordinate point of the previous frame on the normalized plane.

4. The autonomous positioning and mapping method for an inspection robot based on plant and station semantic information according to claim 1 is characterized in that: For the obtained basic matrix F, the correct matching points are screened out from the set of matching point pairs containing error points through RANSAC.

5. The autonomous positioning and mapping method of an inspection robot based on plant and station semantic information according to claim 1 is characterized in that: In step S3, the construction of coordinate attention includes: Coordinate information embedding: Use a pooling kernel of size (H, 1) or (1, W) to encode each channel along the x-axis and y-axis respectively. The output of the c-th channel with height h is: The output of the cth channel with width w is: Attention Generation: Connection and Two feature maps are passed to a 1×1 shared convolution transformation function F1, producing: f=δ(F1([z h ,z w ])); Among them, [z h ,z w ] represents the connection operation along the spatial dimension, δ is the nonlinear activation function; According to the reduction rate r of the control module size in SE Attention, f is split into two separate tensors along the spatial dimension. and Then use two 1×1 convolution f h and f w The feature map f h and f w Transform to the same number of channels as input X and get: Among them, g h and g w That is, as the attention weight, the output expression of the coordinate attention module is:

6. The autonomous positioning and mapping method of an inspection robot based on plant and station semantic information according to claim 1 is characterized in that: Also included is a real-time depth generation algorithm, which is constructed as follows: The horizontal Sobel operator is used to preprocess the image, and each pixel on the processed image is mapped into a new image. The pixel value of the new image is defined as: Where P is the pixel value of the point, preFilterCap is a constant parameter, and the gradient information of the image is obtained; When the pixel position is offset by b from left to right, and the disparity of pixel p is d, the minimum cost value L is b (p,d) is expressed as: The parameters P1 and P2 are defined as: C(p,d) is calculated by the following formula: Between the current pixel p and the pixel q after shifting d, after half-pixel interpolation, find the minimum grayscale or RGB difference between the two pixels and use it as the value of C(p,d); Calculate L for the eight neighborhoods of the pixel according to the above steps. b (p, d), then accumulate the 8 cost values, select the disparity with the smallest cumulative cost as the final disparity of the pixel, accumulate the 8 cost values ​​for each pixel, select the disparity with the smallest cumulative cost as the final disparity of the pixel, and form the disparity map of the entire image; According to the obtained disparity map, the depth of the entire image is restored, where a point p on the disparity map c (u,v) corresponds to p in three-dimensional space w The (x,y,z) coordinates are given by the following formula: where f x 、f y 、c x 、c y is the camera intrinsic matrix parameter, baseline is the distance between the optical centers of the left and right cameras, and desparity.at(v,u) is the value of the coordinate (u,v) corresponding to point p on the disparity map.

7. The autonomous positioning and mapping method for an inspection robot based on plant and station semantic information according to claim 6 is characterized in that: In screening valid map points p w When , the voxels corresponding to the dynamic category of semantic segmentation are deleted.

Images