Robot-based inspection method, device, equipment and medium for abnormal objects in passages

Abstract

This invention provides a robot-based inspection method for abnormal objects in passageways. The method obtains images of inspection points taken by the robot during inspections, with each inspection point corresponding to multiple inspection point images. These images are then aligned through image processing. Feature values ​​are then extracted from the aligned images using a pre-trained deep learning model. The feature values ​​are then used to determine whether abnormal images exist, thereby determining whether abnormal objects exist in the inspected passageway. This method, which uses a trained model to obtain image feature information and determine feature differences, overcomes environmental dependence compared to traditional image frame difference methods, resulting in a more robust arrangement and an unlimited range of abnormal object detection.

Description

Technical Field

[0001] The present invention belongs to the field of robot inspection, and in particular relates to a robot-based inspection method, device and medium for abnormal objects in a passage. Background Art

[0002] With the development of science and technology, artificial intelligence (AI) has gradually come to the fore. Inspection robots are a key component of AI. Using robots to inspect for unusual accumulations in various scenarios, replacing traditional manual inspections, has broad application value. For example, fire escapes should not be blocked by large objects such as cartons and garbage. Similarly, at certain streets or intersections, we cannot allow the accumulation of debris such as garbage or cartons to affect the cityscape.

[0003] When robots inspect abnormal accumulations in the above-mentioned different scenarios, common methods in the prior art include:

[0004] 1) Using deep learning to detect abnormal objects as targets, the learning process must first learn a fixed category of abnormal objects as detection targets, for example, pre-setting detection to only include cardboard boxes and garbage bags in aisles. This existing solution has a drawback: it limits the types of abnormal objects to be detected. In reality, the number of foreign objects is vast and even difficult to enumerate. This means that this solution can only address specific needs in specific scenarios, resulting in significant abnormal object detection limitations.

[0005] 2) Using image differencing to detect non-specific abnormal objects. This solution primarily uses an image of the abnormal scene and a normal scene to perform a differential analysis. It then uses classic image processing methods such as edge detection and image filtering to detect and identify foreign objects. This solution is highly sensitive to image position, ambient lighting, or scene changes. It also requires a fixed camera and is heavily dependent on hardware. This method can result in numerous false positives with even the slightest change in ambient lighting or scene background, making it less robust to environmental changes.

[0006] Therefore, the technical problems existing in the prior art at least are:

[0007] (-) Deep learning detection methods that first learn fixed categories of abnormal objects as detection targets are highly dependent on the number of types of abnormal objects, making it difficult to cover real-world scenarios and having relatively large limitations on the range of foreign objects to be detected;

[0008] (2) Using image difference to detect non-specific abnormal objects is highly dependent on hardware equipment and environmental conditions and is not very robust. Summary of the Invention

[0009] The present invention proposes a robot-based inspection method for abnormal objects in a channel to solve the problems existing in the prior art of being greatly affected by the environment, having poor robustness, and having a limited range for detecting abnormal objects.

[0010] The present invention is achieved through the following technical solutions:

[0011] A robot-based inspection method for abnormal objects in a passage comprises the following steps:

[0012] S1. Obtain inspection point images taken by the robot during inspection. One inspection point corresponds to multiple inspection point images.

[0013] S2. Performing image alignment on the multiple inspection point images of the same inspection point to obtain multiple aligned images;

[0014] S3. Extracting feature values ​​from the multiple images according to a pre-trained deep learning model to obtain image feature values;

[0015] S4. Determine whether there is an abnormal object based on the image feature value.

[0016] Preferably, the inspection point pictures taken during the inspection include:

[0017] Take pictures at multiple inspection points along the preset inspection route to obtain multiple inspection point pictures of the same inspection point.

[0018] Furthermore, the inspection points include fixed shooting points and fixed shooting parameters.

[0019] Preferably, performing image alignment processing on the multiple inspection point pictures of the same inspection point includes:

[0020] For the multiple inspection point images of the same inspection point, one image is preset as a reference image for image alignment, and the image to be aligned and calibrated is the target image;

[0021] Acquire feature points on the reference image and the target image, and match the feature points;

[0022] The target image is mapped to the reference image according to the feature point matching result.

[0023] Preferably, the presetting of a picture as a reference image for image alignment includes:

[0024] Preselect a picture or use the first picture taken by the robot at the inspection point as the reference image for image alignment.

[0025] Preferably, the training process of the deep learning model includes:

[0026] Obtaining a training sample set, wherein the training sample set includes a plurality of original images;

[0027] Each original image is processed and three types of images are generated, including a first image, a second image, and a third image. The first image is the original image, the second image is an image obtained by performing image transformation on the first image, and the third image is an image obtained by randomly pasting image content on the first image.

[0028] Generating corresponding labels for the first picture, the second picture, and the third picture according to a preset rule, wherein the preset rule is that the label setting for the second picture is the same as that for the first picture, and the label setting for the third picture is different from that for the first picture;

[0029] The first image, the second image, and the third image are used to train the label classifier Circle Loss to obtain the deep learning network model.

[0030] Preferably, in step S4, judging whether there is an abnormal object according to the image feature value includes:

[0031] A hierarchical clustering analysis is performed on the image feature values ​​to obtain several clusters; the average of the maximum and minimum values ​​of the vertical coordinates after clustering is used as a threshold to separate the clusters to obtain abnormal features. The abnormal feature is an image feature value greater than the threshold, and an abnormal object exists in the image containing the abnormal feature.

[0032] A robot-based inspection device for abnormal objects in a passage, comprising:

[0033] The acquisition module obtains the inspection point pictures taken by the robot during the inspection. One inspection point corresponds to multiple inspection point pictures;

[0034] An alignment module performs image alignment processing on the multiple inspection point images of the same inspection point to obtain multiple aligned images;

[0035] A feature value extraction module extracts feature values ​​from the multiple images based on a pre-trained deep learning model to obtain image feature values;

[0036] The abnormal object judgment module judges whether there is an abnormal object according to the image feature value.

[0037] The present invention also includes a non-volatile storage medium comprising one or more computer instructions, wherein the one or more computer instructions implement the above-mentioned measurement method when executed.

[0038] A terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the robot-based channel abnormal object inspection method as described above when executing the computer program.

[0039] Compared with the prior art, the present invention has at least the following beneficial effects or advantages:

[0040] The present invention uses a trained model to obtain feature information of an image to determine feature differences. Compared with the traditional image frame difference method, it overcomes the dependence on the environment, has a more robust sorting scheme, and has an unlimited range of detection for abnormal objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The present invention will be described in further detail below with reference to the accompanying drawings;

[0042] Figure 1 This is a flow chart of the method for inspecting abnormal objects in a passage based on a robot according to the present invention;

[0043] Figure 2 A schematic diagram of image feature point matching and alignment for robot-based inspection of abnormal objects in a channel according to the present invention;

[0044] Figure 3 Schematic diagram of hierarchical clustering analysis results of the robot-based inspection of abnormal objects in passages according to the present invention;

[0045] Figure 4 Schematic diagram of a device for inspecting abnormal objects in a passage based on a robot according to the present invention; DETAILED DESCRIPTION

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0047] like Figure 1 As shown, a robot-based inspection method for abnormal objects in a channel includes the following steps:

[0048] S1. Obtain inspection point images taken by the robot during inspection. One inspection point corresponds to multiple inspection point images.

[0049] S2. Performing image alignment on the multiple inspection point images of the same inspection point to obtain multiple aligned images;

[0050] S3. Extracting feature values ​​from the multiple images according to a pre-trained deep learning model to obtain image feature values;

[0051] S4. Determine whether there is an abnormal object based on the image feature value.

[0052] Based on a specific robot inspection scenario, the present invention is based on an inspection robot, in which a camera is placed on the robot's head. Multiple images are taken at fixed inspection points with fixed shooting parameters, and the multiple images are aligned through image processing. The aligned images are then subjected to feature value extraction using a pre-trained deep learning model. The feature values ​​are then used to determine whether there are abnormal images, thereby determining whether there are abnormal objects in the inspection channel. The present invention uses a technical solution to obtain feature information from images using a trained model to determine feature differences. Compared with the traditional image frame difference method, this overcomes dependence on the environment, is more robust, and has an unlimited range of abnormal object detection.

[0053] Furthermore, the embodiment of the present invention uses robots for inspection, which does not require learning fixed categories of abnormal objects as detection targets, can cover the entire scene, and is not affected by the detection range of abnormal objects.

[0054] According to a preferred embodiment, in step S1, pictures are taken at multiple inspection points in a loop according to a preset inspection route to obtain multiple inspection point pictures of the same inspection point. Specifically, the inspection route can be pre-arranged and customized according to the inspection channel scene, and fixed shooting points and corresponding fixed shooting parameters can be pre-set. The shooting parameters are not limited to the angle, exposure and focal length during shooting. Each time the robot conducts an inspection, a picture will be taken at a fixed inspection point at a fixed angle. After the shooting is completed, the robot will run to the next fixed inspection point according to the instructions to shoot. After shooting all the inspection points, the robot will start from the beginning according to the navigation closed loop and repeat the first inspection route and image shooting from the first fixed inspection point. As the robot continues to repeat the inspection route multiple times according to the regulations, multiple inspection point pictures taken in chronological order are obtained at the same inspection point. In this embodiment, the method of obtaining multiple pictures of the same inspection point at different time points through robot inspection reduces the number of cameras in the layout compared to the solution in the prior art where the camera is fixed at the inspection point for shooting. This not only reduces the cost of the camera, but also greatly improves the coverage of the detection.

[0055] According to a specific embodiment, certain errors may occur during the robot's inspection and navigation process, which may result in some slight position or angle deviations in the pictures taken at the same inspection point. In order to reduce the impact of this deviation on the subsequent judgment of abnormal objects, for the multiple inspection point pictures of the same inspection point, the deviations of the multiple inspection point pictures are calibrated by image alignment. When calibrating and aligning the images, it is necessary to pre-select a picture as the reference image for image alignment, and the other images to be aligned and calibrated are the target images. Preferably, a picture can be manually pre-selected or the first picture taken by the robot at the inspection point can be used as the reference image for image alignment. Figure 2 As shown, when implementing the image calibration and alignment of step S2, a picture is preset as the reference image for image alignment, the image to be aligned and calibrated is the target image, feature points on the reference image and the target image are obtained, the feature points are matched, and according to the feature point matching results, the target image is mapped to the reference image.

[0056] According to the above technical solution, the feature-based image alignment method of the present invention calculates a set of feature points on the reference image and the target image respectively, matches the feature points using a certain matching principle, and then calculates a single-projection affine transformation matrix of an image based on the matching result, and then maps each point of the original image to the new image through the affine transformation through the operation of matrix multiplication, and the new image formed is the final aligned image, thereby mapping a target image to the reference image to achieve image alignment. Preferably, the acquisition of feature points uses the more traditional SIFT scale-invariant feature transformation algorithm, and the feature point matching algorithm uses the KNN nearest neighbor matching algorithm. The technical solution of the embodiment of the present invention corrects the position and angle deviations caused by taking multiple pictures during the robot inspection and navigation process, and reduces the error in judging abnormal objects.

[0057] According to a specific embodiment, in step S3, the training process of the deep learning model includes:

[0058] Obtaining a training sample set, wherein the training sample set includes a plurality of original images;

[0059] Each original image is processed and three types of images are generated, including a first image, a second image, and a third image. The first image is the original image, the second image is an image obtained by performing image transformation on the first image, and the third image is an image obtained by randomly pasting image content on the first image.

[0060] Generating corresponding labels for the first picture, the second picture, and the third picture according to a preset rule, wherein the preset rule is that the label setting for the second picture is the same as that for the first picture, and the label setting for the third picture is different from that for the first picture;

[0061] The first image, the second image, and the third image are used to train the label classifier Circle Loss to obtain the deep learning network model.

[0062] In this embodiment, preferably, the following specific implementation methods are included:

[0063] Data preparation stage

[0064] The present invention uses the open source network dataset Imagenet. During training, three types of images are generated for each training image: the first type is the original image without any operation on the image, denoted as img_src; the second type is the image that has some image transformations such as translation, scaling, and rotation based on img_src to make it have a certain deviation from img_src, denoted as img_pos; the third type is the image that uses some other images to be scaled a certain proportion and randomly pasted on img_src, denoted as img_neg.

[0065] Label generation stage:

[0066] During training, the embodiment of the present invention automatically generates corresponding labels based on the image type. The generated rule is to set the labels of img_src and img_pos to be the same, and set the labels of img_src and img_neg to be different. For example, if an original image img_src1 is selected and its label is set to 1, then the above rule is used to generate the corresponding images img_pos1 and img_neg1 from img_src1. The label of img_pos1 is set to 1, and the label of img_neg1 is set to a number other than 1.

[0067] Metric learning model training phase:

[0068] The loss function used in this paper is CircleLoss, a massive label classifier loss with relatively high accuracy in recent years. This loss is mainly used in general fine-grained search tasks such as face recognition. In some cases, the abnormal objects in this paper may appear in a smaller granularity, so this method is more suitable. The following is the original formula of CircleLoss:

[0069]

[0070] According to the above embodiments, the present invention employs unsupervised contrastive learning to train a deep neural network model. This deep learning network model then uses image feature differences to replace the image frame differences used in existing techniques to determine the presence of abnormal objects. Compared to specific abnormal target detection schemes, the present invention is completely independent of the type of abnormal target, thus adapting to a wider range of abnormal object detection options. It also has low environmental dependence and is more robust than schemes that calculate image frame differences.

[0071] According to a specific embodiment, in step S4, judging whether there is an abnormal object based on the image feature value includes:

[0072] A hierarchical clustering analysis is performed on the image feature values ​​to obtain several clusters; the average of the maximum and minimum values ​​of the vertical coordinates after clustering is used as a threshold to separate the clusters to obtain abnormal features. The abnormal feature is an image feature value greater than the threshold, and an abnormal object exists in the image containing the abnormal feature.

[0073] Specifically, after obtaining a series of features of multiple images at the same inspection point, a hierarchical cluster analysis method is used to find the corresponding outlier features. In the prior art, hierarchical cluster analysis mainly refers to clustering layer by layer, which can merge small clusters from bottom to top, or split large clusters from top to bottom. Figure 3 As shown, the present invention uses a bottom-up clustering method: specifically, each time the two clusters with the shortest distance are found, and then merged into a large cluster until all are merged into one cluster. The whole process is equivalent to building a tree structure. The result is as follows Figure 3 As shown. In this embodiment, since the present invention uses a metric learning method with edge differences, generally speaking, the average threshold can be used to distinguish abnormal feature values, and the image containing abnormal feature values ​​is the image with abnormal objects. The threshold here refers to the score threshold between each cluster obtained by the vertical coordinate after hierarchical clustering. For example, Figure 3 As shown, if we choose 25 as the threshold, the data can be divided into two categories, Figure 3 The dashed lines enclose the sample as one class, and the remaining unenclosed line segments as one class. This clearly shows that samples 13 and 14 are outliers. Preferably, the threshold in the present invention is the average of the maximum and minimum values ​​of the ordinate. Samples with values ​​greater than this threshold are considered outliers, while those with values ​​less than this threshold are considered normal. Since we use MetricLearning with Margin, the separability between classes is relatively high, so setting an average value can effectively separate outliers.

[0074] According to a specific embodiment of the present invention, Figure 4 As shown, a robot-based abnormal object inspection device is also provided, comprising:

[0075] The acquisition module is used to obtain inspection point images taken by the robot during inspection. One inspection point corresponds to multiple inspection point images;

[0076] An alignment module is used to perform image alignment processing on the multiple inspection point pictures of the same inspection point to obtain multiple aligned images;

[0077] A feature value extraction module is used to extract feature values ​​from the multiple images according to a pre-trained deep learning model to obtain image feature values;

[0078] The abnormal object judgment module is used to judge whether there is an abnormal object based on the image feature value. Specifically, the process of the device for detecting abnormal objects in a channel based on a robot corresponds to the process of the method for detecting abnormal objects in a channel, and will not be described in detail here.

[0079] The present invention also provides a non-volatile storage medium, which includes one or more computer instructions, and the one or more computer instructions implement the above inspection method when executed.

[0080] The specific embodiments described above further illustrate the objectives, technical solutions, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present invention also fall within the scope of protection of the present invention.

Claims

1. A robot-based inspection method for abnormal objects in a channel, characterized in that: The following steps are involved: S1. Obtain inspection point images taken by the robot during inspection. One inspection point corresponds to multiple inspection point images. S2. Performing image alignment on the multiple inspection point images of the same inspection point to obtain multiple aligned images; S3. Extracting feature values ​​from the multiple images according to a pre-trained deep learning model to obtain image feature values; S4. Determine whether there is an abnormal object based on the image feature value; The training process of the deep learning model includes: Obtaining a training sample set, wherein the training sample set includes a plurality of original images; Each original image is processed and three types of images are generated, including a first image, a second image, and a third image. The first image is the original image, the second image is an image obtained by performing image transformation on the first image, and the third image is an image obtained by randomly pasting image content on the first image. Generating corresponding labels for the first, second, and third images according to a preset rule, wherein the preset rule is that the second image and the first image have the same label setting, and the third image and the first image have different label settings; The first, second and third images are used as input, a Circle Loss loss function is adopted, and unsupervised contrastive learning is used to train the deep learning model.

2. The robot-based inspection method for abnormal objects in a passage according to claim 1, characterized in that: The inspection point pictures taken during the inspection include: Take pictures at multiple inspection points along the preset inspection route to obtain multiple inspection point pictures of the same inspection point.

3. The robot-based inspection method for abnormal objects in a passage according to claim 1, characterized in that: The inspection points include fixed shooting points and fixed shooting parameters.

4. The robot-based inspection method for abnormal objects in a passage according to claim 1, characterized in that: The image alignment processing of the multiple inspection point pictures of the same inspection point includes: For the multiple inspection point images of the same inspection point, one image is preset as a reference image for image alignment, and the image to be aligned and calibrated is the target image; Acquire feature points on the reference image and the target image, and match the feature points; The target image is mapped to the reference image according to the feature point matching result.

5. The robot-based inspection method for abnormal objects in a passage according to claim 4 is characterized in that: The preset image as the reference image for image alignment includes: Preselect a picture or use the first picture taken by the robot at the inspection point as the reference image for image alignment.

6. The robot-based inspection method for abnormal objects in a passage according to claim 1, characterized in that: In step S4, judging whether there is an abnormal object according to the image feature value includes: A hierarchical clustering analysis is performed on the image feature values ​​to obtain several clusters; the average of the maximum and minimum values ​​of the vertical coordinates after clustering is used as a threshold to separate the clusters to obtain abnormal features. The abnormal feature is an image feature value greater than the threshold, and an abnormal object exists in the image containing the abnormal feature.

7. A robot-based inspection device for abnormal objects in a passage, characterized in that: include: The acquisition module obtains the inspection point pictures taken by the robot during the inspection. One inspection point corresponds to multiple inspection point pictures; An alignment module performs image alignment processing on the multiple inspection point images of the same inspection point to obtain multiple aligned images; A feature value extraction module extracts feature values ​​from the multiple images based on a pre-trained deep learning model to obtain image feature values; An abnormal object judgment module is used to judge whether there is an abnormal object based on the image feature value; The training process of the deep learning model includes: Obtaining a training sample set, wherein the training sample set includes a plurality of original images; Each original image is processed and three types of images are generated, including a first image, a second image, and a third image. The first image is the original image, the second image is an image obtained by performing image transformation on the first image, and the third image is an image obtained by randomly pasting image content on the first image. Generating corresponding labels for the first, second, and third images according to a preset rule, wherein the preset rule is that the second image and the first image have the same label setting, and the third image and the first image have different label settings; The first, second and third images are used as input, a Circle Loss loss function is adopted, and unsupervised contrastive learning is used to train the deep learning model.

8. A non-volatile storage medium, characterized in that: It comprises one or more computer instructions, which, when executed, implement the robot-based channel abnormal object inspection method described in any one of claims 1 to 6.

9. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the robot-based channel abnormal object inspection method according to any one of claims 1 to 6 is implemented.

Images