A closed loop inspection method, device, recording medium and system for a visual inspection robot

Abstract

The present application belongs to the technical field of electric field test, and particularly relates to a closed loop detection method of a visual inspection robot, comprising the following steps: target enhancement is performed on a perception aliasing image, a contrast branch is added, the accuracy of a convolutional neural network in identifying the perception aliasing image is improved, and the robustness of the closed loop detection is enhanced. The present application also discloses a non-transient readable recording medium storing a program of the method and a system comprising the medium, and the program can be called by a processing circuit to execute the above method.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of visual positioning and mapping, and discloses a closed-loop detection method for a visual patrol robot, a recording medium and a system. BACKGROUND

[0002] With the continuous development of the national economy, the overall scale of the power grid is expanding, and the traditional transformer maintenance production mode has been unable to meet the needs of the development of the power grid. Compared with the traditional mode of transformer operation and transformer maintenance division, the transformer station operation and maintenance integration requires the operation and maintenance work area to carry out daily maintenance maintenance business. Due to the lack of operation personnel, in actual work, it is easy to appear that the equipment inspection is not in place, and the safety management work is not in place. Therefore, the visual robot is used in the transformer station, which can replace the operation and maintenance personnel to carry out the conventional auxiliary work to a certain extent, reduce the time of the operation personnel going back and forth between each transformer station, and improve the operation and maintenance efficiency.

[0003] SLAM (simultaneous localization and mapping) is the full name of real-time positioning and mapping or concurrent mapping and positioning. Its main function is to enable robots to complete localization, mapping and path planning in unknown environments. At present, SLAM technology is widely used in robots, unmanned aerial vehicles, unmanned vehicles, AR, VR and other fields, and can realize the functions of autonomous positioning, mapping and path planning of machines by relying on sensors. Among them, closed-loop detection can determine whether the current position has been visited by a mobile robot, and is a key link in the SLAM process. Accurate detection of the closed loop can effectively reduce the cumulative error of the robot pose estimation, which is beneficial to building a more accurate map and ensuring the consistency of the generated map.

[0004] In the field of visual SLAM closed-loop detection, existing methods in recent years can be roughly divided into two categories: traditional feature-based methods and deep learning-based methods, and the main achievements are as follows:

[0005] (1) Traditional feature-based method

[0006] Traditional methods are mainly based on the manual feature extraction of the image feature expression. Among them, the local feature provides the invariance to a certain illumination and affine transformation, which is suitable for solving image matching, retrieval and other problems, and has been widely used in visual loop closure detection. The scale invariant feature transform (SIFT) is used to extract image features, and a number of clusters are formed by clustering method. The frequency of visual vocabulary can be obtained by calculating the number of feature points of each center. The disadvantage is that the calculation process is very time-consuming, which is difficult to meet the real-time requirements of SLAM. As an improvement of SIFT feature extraction, some other features, such as oriented FaSt and rotated BRIEF (ORB) and speeded-up robust feature (SUFR) are recommended, which greatly improves the calculation speed while properly reducing the robustness. However, the common disadvantage of manual feature description is that it cannot effectively cope with scene changes such as season, viewing angle, illumination and dynamic objects in the scene. In addition, the traditional method based on appearance completely depends on the appearance without using geometric information, so it is easy to misjudge similar images as loop closure in the loop closure detection process.

[0007] (2) Deep learning based method

[0008] In recent years, deep learning technology has made great progress in visual scene recognition and classification, providing a new way to solve the loop closure detection problem. Using a pre-trained convolutional neural network (CNN) to extract image features and then comparing their similarities, it is found that the features extracted by CNN are better at dealing with environmental changes than the manually extracted features. The deep convolutional neural network model usually uses a supervised learning method in the field of scene recognition, which requires a large amount of data with labels, and there are great difficulties in actual operation. A method of using a stacked denoising sparse auto-encoder model for loop closure detection is proposed for similarity comparison, but it does not consider the spatial local characteristics of the image; the PCA-Net deep network extracts image features as image descriptions, the network is simple and easy to debug, but the input image pixels are too small, which makes it difficult to judge the similarity between images in actual operation. As the mobile robot moves for a long time, the number of frames transmitted by the sensor accumulates, and the calculation time of the convolutional neural network in loop closure detection also gradually increases; as the size of the deep neural network model increases, the feature descriptor extracted has a large dimension, and the real-time performance is not good, and the robustness of the loop closure detection in the presence of multiple moving targets outdoors is weak.

[0009] At the same time, the algorithm based on deep learning has a large amount of calculation, and there are still factors such as the change of the visual angle of the mobile robot in the real environment, so how to further improve the detection speed and enhance the robustness in the research of closed-loop detection is still a problem to be solved in the future. SUMMARY

[0010] In view of the above problems, the application provides a closed-loop detection method for a visual patrol robot, and the specific scheme comprises the following steps:

[0011] S1. Grabbing sample images of various scenes from a known reference environment data set, adding labels to the sample images, performing feature extraction, enhancing the sample images with perceptual aliasing, adding the obtained images and their labeled position information to a sample library, and extracting a training set and a test set from the sample library.

[0012] S2. Building a convolutional neural network model based on the training set and the test set for training and testing, decomposing a standard convolution into a depth convolution and a point-by-point convolution during training, the depth convolution acting on each channel of the input picture, and the point-by-point convolution acting on the output feature map of the depth convolution; the image obtained during the training process is first input into a feature extraction network to obtain a corresponding feature map, and then the feature map is input into a detection branch and a comparison branch as input, the output of the detection branch is a predicted target box and its category, and the detection loss is calculated by comparing the target box with the real box; in the comparison branch, comparison learning samples are constructed, and the aliasing feature map is compared with the feature map of a randomly selected background region to calculate the comparison loss.

[0013] S3. Defining a function for calculating the similarity of the image features and setting a threshold, and determining the closed loop when the similarity reaches the threshold, and performing closed loop correction if no closed loop is detected.

[0014] Preferably, the enhancement comprises at least one of the following three ways: for each target in the image, enlarging or reducing by different proportions, and then pasting multiple times at random positions; randomly selecting multiple targets in the image, enlarging or reducing by different proportions, and then pasting multiple times at random positions; randomly selecting multiple targets in the data set, enlarging or reducing by different proportions, and randomly pasting multiple times on the randomly selected images.

[0015] Further, the test comprises detecting by extracting feature maps of different scales through SSD, setting different prior boxes in each unit of SSD according to the scales or aspect ratios, taking these prior boxes as the basis for the predicted bounding boxes, for each prior box in each unit, outputting a set of independent detection values corresponding to a bounding box, and the detection values include the confidence of each category and the center coordinates, width and height of the bounding box.

[0016] Based on the same inventive concept, the application provides a closed-loop detection method of a visual patrol robot, comprising the following functional modules:

[0017] A data module is configured to capture sample images of various scenes from a known reference environment dataset, add labels to the sample images, perform feature extraction, enhance the sample images with perception aliasing, add the obtained images and their labeled position information to a sample library, and extract a training set and a test set from the sample library.

[0018] A learning module is configured to build a convolutional neural network model based on the training set and the test set for training and testing, decompose a standard convolution into a depth convolution and a point-wise convolution during training, the depth convolution acts on each channel of an input picture, and the point-wise convolution acts on an output feature map of the depth convolution; the images obtained during the training process are first input into a feature extraction network to obtain corresponding feature maps, and then the feature maps are input into a detection branch and a comparison branch as inputs, the output of the detection branch is a predicted target frame and its category, and a detection loss is calculated by comparing the target frame with a real frame; in the comparison branch, comparison learning samples are constructed, and a comparison loss is calculated by comparing an aliasing feature map with a feature map of a randomly selected background region.

[0019] A detection module is configured to define a function for calculating the similarity of image features and set a threshold, and determine a closed loop when the similarity reaches the threshold, and perform closed loop correction if no closed loop is detected.

[0020] Preferably, the enhancement comprises at least one of the following three methods: expanding or reducing each target in the image by different scales, and then pasting multiple times at random positions; randomly selecting multiple targets in the image, expanding or reducing them by different scales, and then pasting multiple times at random positions; and randomly selecting multiple targets in the dataset, expanding or reducing them by different scales, and then pasting multiple times at random positions.

[0021] Further, the test comprises detecting by extracting feature maps of different scales through SSD, each unit of the SSD is provided with prior frames with different scales or aspect ratios, the predicted boundary frame is based on these prior frames, for each prior frame of each unit, a set of independent detection values is output, the detection values include the confidence of each category and the center coordinates, width and height of the boundary frame.

[0022] Another scheme of the application provides a non-transitory readable recording medium for storing one or more programs comprising a plurality of instructions, when the instructions are executed, the processing circuit will execute the above-mentioned closed-loop detection method of a visual patrol robot.

[0023] Still another aspect of the present application provides a closed-loop detection system of a visual inspection robot, comprising a processing circuit and a memory electrically coupled to the processing circuit, wherein the memory is configured to store at least one program, the program comprising a plurality of instructions, and the processing circuit executes the program to perform the above-mentioned closed-loop detection method of a visual inspection robot.

[0024] Compared with the prior art, the present application has the following beneficial effects:

[0025] In the SLAM closed-loop detection in a harsh environment, target enhancement is performed on the perception aliasing image, and a contrast branch is added to improve the accuracy of the convolutional neural network in identifying the perception aliasing image.

[0026] In the model training process, a contrast branch and a new loss function are added to extend the contrast learning to strong supervision, so that the normalized embedding distance of the same object is closer, and the distance of the negative sample pair is farther.

[0027] The CNN (convolutional neural network) is applied to the SLAM closed-loop detection, the basic network can adopt the MobileNet network suitable for mobile terminals, and the target detection adopts the SSD (Single Shot MultiBox Detector) algorithm trained in a single stage, which helps the power transmission line inspection robot to quickly perform closed-loop detection and improves the convergence efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 A flowchart of the closed-loop detection process of the visual inspection robot based on strong supervision learning in the embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme of the embodiments of the present application will be described below in combination with the drawings of the embodiments of the present application, and the described embodiments are part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making innovative efforts fall within the scope of protection of the present application.

[0030] The specific steps of the closed-loop detection method of the visual inspection robot provided by the present application are as follows:

[0031] Color and depth images are obtained by RGB-D sensor, and ORB algorithm is used to extract the continuous images. Then, the pose estimation of the camera is obtained by solving PnP problem using Ransac algorithm combined with depth images. Since there is a lot of redundant information in continuous images, the strategy based on pose constraint is adopted to screen the key frames. The basic idea is that if the relative pose change of the robot when collecting two images is large, the new frame is determined as a key frame. These key frames will be sent to G2O link in the form of nodes and edges for SLAM backend optimization.

[0032] The image size obtained by RGB-D camera is 640x480 pixels, which will bring high computational complexity due to its high dimension. Therefore, the image is reduced, and the size of the preprocessed image is 320x240 pixels. Then, the reduced images are batch processed for image enhancement. The New College and City Centre datasets in Oxford dataset are widely used to evaluate the performance of visual SLAM loop closure detection. The two datasets contain 2146 frames and 2474 frames of images, respectively.

[0033] Referring to the drawings Figure 1 First, sample images of various scenes are captured from New College and City Centre, respectively. Then, the images are labeled, features are extracted, image enhancement is performed, and image preprocessing is carried out. The processed images are divided into training set and test set. The training set is used to train the image classification model and region generation model, and the test set is used to test the recognition effect of the model. A supervised contrast branch is added in the training stage. The feature map is compared with the feature map of the randomly selected background region on the feature map, which improves the detection effect of the detection algorithm on the perception aliasing. For the input sequence image data, the similarity of the scene is judged by the trained MobileNet-SSD and the optimized features, and then it is judged whether a loop is formed.

[0034] The model introduces a contrast branch on the target detection model MobileNet-SSD, without changing the original feature extraction network and detection branch. In the training process, the image first enters the feature extraction network to obtain the corresponding feature map, and then the feature map is input into the detection branch and the contrast branch. The output of the detection branch is the predicted target box and its category. The classification loss and target box regression loss are calculated by comparing the predicted box with the real box. The sum of the classification loss and the target box regression loss is the detection loss. In the contrast branch, we use the existing real box label to sample the target region corresponding feature and the background region corresponding part of the feature from the overall feature map of the image. In a training batch, two target samples of the same category are positive sample pairs, target samples of different categories are positive sample pairs, and any target sample and background region sample are negative sample pairs. The distance between positive and negative samples is used to calculate the contrast loss. In the model training process, the detection loss and the contrast loss affect the training of the feature extraction network through gradient backpropagation. The contrast branch uses the contrast loss to reduce the distance between positive samples and increase the distance between negative samples to help improve the feature extraction network's ability to represent mixed targets, thereby improving the model's detection effect.

[0035] In the actual implementation process, there are mainly 3 stages:

[0036] First, establish a closed-loop detection sample library system.

[0037] We perform target augmentation on the images in the New College and City Centre datasets, mainly using the following methods:

[0038] 1. For each target in the image, enlarge or reduce it by different proportions, and then paste it multiple times at random positions;

[0039] 2. Randomly select multiple targets in the image, enlarge or reduce them by different proportions, and then paste them multiple times at random positions;

[0040] 3. Randomly select multiple targets in the dataset, enlarge or reduce them by different proportions, and randomly paste them multiple times on randomly selected images;

[0041] Add the image and the marked position to the sample library through operator confirmation, complete the construction of the sample library, and divide the training set and the test set.

[0042] Second, train the MobileNet-SSD model based on supervised contrast.

[0043] Lightweight network MobileNet is adopted, and based on deep separable convolution, the standard convolution is decomposed into depth convolution and point-wise convolution to reduce the parameter quantity and calculation amount. The depth convolution part acts on each channel of the input picture, and the point-wise convolution part acts on the output feature map of the depth convolution.

[0044] Different scale feature maps are extracted by SSD for detection, and large scale feature maps are used to detect small objects, and small size feature maps are used to detect large objects. SSD sets different prior boxes with different scales or aspect ratios for each unit, and the predicted bounding box is based on these prior boxes to reduce the training difficulty. For each prior box of each unit, a set of independent detection values corresponding to a bounding box is output. The output contains two parts: the first part is the confidence of each class, and the second part is the specific position value (cx, cy, w, h) of the bounding box, which are the center coordinates, width and height of the bounding box respectively. The loss function adopts the weighted sum of position error and confidence error. The position error adopts Smooth L1 loss function, and the confidence error adopts softmax loss function.

[0045] In the contrast learning training process, in order to ensure that each target can correspond to at least one pair of positive sample pairs in a batch, the input of the network is adjusted. Assuming that the size of a training batch is n in normal training (without contrast branch), the input images of the network are x1, …, xn, and the output of the network is y1, …, yn. n After adding the contrast learning branch, for each image x i , two kinds of random data augmentation T1, T2 are needed to obtain new two images x i1 , x i2 , and then the new images are input into the network. After obtaining the candidate region position and object category of the network output, the actual candidate region position and object category are compared to calculate the corresponding cost function contrast loss, and the contrast loss function is:

[0046]

[0047] P(i) = {p e A(i)} represents all samples of the same class i, and in the supervised contrast task, multiple pairs of positive samples may appear in a training batch. For any target sample block to be detected, the unit vector z i after pooling, there is at least another unit vector z a of the same class as a positive sample pair, and z i · z a is the cosine distance between the positive sample pairs, and there is a unit vector z p of different classes or background regions as a negative sample pair, and z i · z pwhere τ is a temperature constant.

[0048] Third, loop closure detection is performed. Suppose there are two keyframes p i and p j , each keyframe can be represented by t features after MobileNet:

[0049]

[0050] A similarity function δ is defined to measure the similarity of features using the cosine of the angle:

[0051] A threshold F a is set in advance. If the similarity of the current frame with a certain keyframe is more than F a times the similarity of the current frame with the last keyframe, it is determined that a loop closure is detected. If no loop closure is detected, the loop closure correction stage is entered.

[0052] Those skilled in the art will understand that embodiments of the present application can be provided as methods, systems, or computer program products. Thus, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0053] The present application is described in reference to the flowcharts and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, a special purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions, which are executed via the processor of the computer or other programmable data processing apparatus, generate an apparatus that creates the flows specified in the flowcharts and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the present application. Figure 1 The flowcharts and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the present application. Figure 1 The flowcharts and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the present application.

[0054] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including instruction apparatus, which implements the functions specified in the flowcharts and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the present application. Figure 1 The flowcharts and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the present application. Figure 1 The flowcharts and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the present application. The flowcharts and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the present application.

[0055] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable data processing devices to generate a computer implemented process, so that the instructions executed on the computer or other programmable data processing devices provide a process for implementing the functions specified in the flowchart Figure 1 one flow or multiple flows and / or the functions specified in the block Figure 1 one block or multiple blocks.

[0056] The above method steps are assembled into a program and stored in a hard disk or other non-transient storage medium, which constitutes the "non-transient readable recording medium" technical solution of the present application. The storage medium is electrically connected with the computer processor, and the closed loop detection of the visual patrol robot can be completed through data processing, which constitutes the "closed loop detection system of the visual patrol robot" technical solution of the present application.

[0057] Finally, it should be pointed out that: the above only for the preferred embodiments of the present application, and not for limiting the present application, although the present application has been described in detail with reference to the foregoing embodiments, for those skilled in the art, it still can be modified to the technical solutions recorded in the foregoing embodiments, or equivalent replacement of part of the technical features. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A closed-loop detection method for a visual inspection robot, characterized in that... Includes the following steps: S1. Extract sample images of various scenes from a known reference environment dataset, add labels to the sample images, extract features, enhance the sample images with perceptual aliasing, add the obtained images and their labeled location information to the sample library, and extract training and test sets from the sample library. S2. Based on the training set and test set, a convolutional neural network model is built for training and testing. During training, the standard convolution is decomposed into depthwise convolution and pointwise convolution. The depthwise convolution part is applied to each channel of the input image, and the pointwise convolution part is applied to the output feature map of the depthwise convolution. During training, the images obtained first enter the feature extraction network to obtain the corresponding feature maps. Then, the feature maps are used as inputs to the detection branch and the contrast branch respectively. The output of the detection branch is the predicted target box and its category. The detection loss is calculated by comparing the target box with the ground truth box. In the contrast branch, contrast learning samples are constructed, and the contrast loss is calculated by comparing the aliased feature map with the feature map of a randomly selected background region. S3. Define a function to calculate the similarity of image features and set a threshold. When the similarity reaches the threshold, a loop closure is determined. If no loop closure is detected, loop closure correction is performed.

2. The closed-loop detection method for a visual inspection robot according to claim 1, characterized in that, The enhancement includes at least one of the following three methods: enlarging or shrinking each target in the image by different proportions, and then copying and pasting it multiple times at random locations; randomly selecting multiple targets in the image, enlarging or shrinking them by different proportions, and then copying and pasting them multiple times at random locations; randomly selecting multiple targets in the dataset, enlarging or shrinking them by different proportions, and then randomly pasting them multiple times on the randomly selected image.

3. The closed-loop detection method for a visual inspection robot according to claim 2, characterized in that, The test includes detecting feature maps of different scales extracted by SSD. Each unit of SSD is set with a prior box of different scales or aspect ratios. The predicted bounding box is based on these prior boxes. For each prior box of each unit, a set of independent detection values ​​is output for each bounding box. The detection values ​​include the confidence of each category and the center coordinates, width and height of the bounding box.

4. A closed-loop detection device for a visual inspection robot, characterized in that... Includes the following functional modules: The data module is used to capture sample images of various scenes from a known reference environment dataset, add labels to the sample images, extract features, enhance the sample images with perceptual aliasing, add the obtained images and their labeled location information to the sample library, and extract training and test sets from the sample library. The learning module is used to build a convolutional neural network model based on the training set and the test set for training and testing. During training, the standard convolution is decomposed into depthwise convolution and pointwise convolution. The depthwise convolution part is applied to each channel of the input image, and the pointwise convolution part is applied to the output feature map of the depthwise convolution. During training, the images obtained first enter the feature extraction network to obtain the corresponding feature maps. Then, the feature maps are used as inputs to the detection branch and the contrast branch respectively. The output of the detection branch is the predicted target box and its category. The detection loss is calculated by comparing the target box with the ground truth box. In the contrast branch, contrast learning samples are constructed, and the contrast loss is calculated by comparing the aliased feature map with the feature map of a randomly selected background region. The detection module defines a function to calculate the similarity of image features and sets a threshold. When the similarity reaches the threshold, a loop closure is determined. If no loop closure is detected, loop closure correction is performed.

5. The closed-loop detection device for a visual inspection robot according to claim 4, characterized in that, The enhancement includes at least one of the following three methods: enlarging or shrinking each target in the image by different proportions, and then copying and pasting it multiple times at random locations; randomly selecting multiple targets in the image, enlarging or shrinking them by different proportions, and then copying and pasting them multiple times at random locations; randomly selecting multiple targets in the dataset, enlarging or shrinking them by different proportions, and then randomly pasting them multiple times on the randomly selected image.

6. The closed-loop detection device for a visual inspection robot according to claim 5, characterized in that, The test includes detecting feature maps of different scales extracted by SSD. Each unit of SSD is set with a prior box of different scales or aspect ratios. The predicted bounding box is based on these prior boxes. For each prior box of each unit, a set of independent detection values ​​is output for each bounding box. The detection values ​​include the confidence of each category and the center coordinates, width and height of the bounding box.

7. A non-transitory readable recording medium for storing one or more programs containing multiple instructions, characterized in that, When the instruction is executed, the processing circuit will perform the closed-loop detection method of the visual inspection robot according to any one of claims 1-3.

8. A closed-loop detection system for a visual inspection robot, comprising a processing circuit and a memory electrically coupled thereto, characterized in that, The memory is configured to store at least one program, the program containing multiple instructions, and the processing circuit runs the program to execute the closed-loop detection method of a visual inspection robot according to any one of claims 1-3.

Images