An intelligent inspection robot system based on the Internet of Things

Abstract

The present application relates to the field of patrol robot systems. The present application provides an intelligent patrol robot system based on the Internet of Things, including: a patrol robot and a cloud platform; the patrol robot includes a visual acquisition module, a visual feature reconstruction module, a communication module, a control module and a power module; the visual acquisition module is used to collect real-time video images taken by the patrol robot; the visual feature reconstruction module is used to extract the final feature map of the real-time video image based on a neural network; the communication module is used to transmit the final feature map of the real-time video image to the cloud platform in real time; the cloud platform is used to identify the abnormal situation of the final feature map of the real-time video image and issue abnormal instructions; the control module is used to receive the abnormal instructions of the cloud platform to drive the patrol robot to execute the corresponding action of the abnormal instructions; the power module is used to provide power to the patrol robot. The present application constructs an intelligent patrol robot system based on the Internet of Things, and realizes the abnormal identification and processing of real-time video images.

Description

Technical Field

[0001] The present invention belongs to the field of inspection robot systems, and in particular relates to an intelligent inspection robot system based on the Internet of Things. Background Art

[0002] The intelligent inspection robot system of the Internet of Things is a system that combines the Internet of Things technology and robotics technology for automated inspections. It is mainly used to monitor and maintain certain specific environments, equipment or infrastructure, such as factories, substations, transmission lines, pipelines, and other places that require regular inspections.

[0003] Traditional IoT-based intelligent inspection robots will perform anomaly analysis on the real-time video images collected by the robots. Real-time video images contain many details, among which the video images are very rich and the amount of data is huge. Directly transmitting and processing them not only takes up network resources, but also increases processing complexity. In addition, there are many unnecessary details in the video images, such as background, light changes, etc. This information will interfere with the subsequent recognition of anomalies in the video images. Therefore, inspection robots usually have basic image processing functions, but are usually limited to some simple algorithms, such as edge detection, color analysis, etc. These algorithms can detect some obvious anomalies, but cannot effectively detect small and potential faults, such as fine cracks on the surface of the equipment and gradually worn parts.

[0004] In addition, traditional IoT-based intelligent inspection robots have difficulty extracting depth information from image data in complex environments. For example, in the case of uneven lighting, complex backgrounds, or noise on the surface of equipment, simple image processing methods have difficulty identifying and separating important visual information. Summary of the invention

[0005] In view of the above-mentioned defects of the prior art, the present invention proposes an intelligent inspection robot system based on the Internet of Things. The technical solution designed by the present invention includes:

[0006] Inspection robots and cloud platforms;

[0007] The inspection robot includes a visual acquisition module, a visual feature reconstruction module, a communication module, a control module and a power module;

[0008] The visual acquisition module is used to collect real-time video images of the monitored object taken by the inspection robot during the inspection process;

[0009] The visual feature reconstruction module is used to extract the final feature map of the real-time video image, including constructing a residual unit, an upsampling layer and a final feature map generation layer in a convolutional neural network; the residual unit is used to downsample the input real-time video image to generate a primary feature map; the upsampling layer is used to restore the resolution of the primary feature map; the final feature map generation layer is used to perform convolution processing on the restored primary feature map and output the final feature map of the real-time video image;

[0010] The communication module is used to transmit the final feature map of the real-time video image to the cloud platform in real time;

[0011] The cloud platform is used to identify abnormal conditions of the final feature map of the real-time video image and issue abnormal instructions;

[0012] The control module is used to receive abnormal instructions from the cloud platform and drive the inspection robot to perform actions corresponding to the abnormal instructions;

[0013] The power module is used to provide power to the inspection robot.

[0014] Preferably, the final feature map generation layer is used to perform convolution processing on the restored primary feature map and output the final feature map of the real-time video image, including:

[0015] The final feature map generation layer includes a convolution filter and a 1×1 convolution operation;

[0016] Each pixel value in the restored primary feature map is mapped to a corresponding element in a matrix, and the convolution filter performs sliding calculation on the matrix to output a final feature map of the real-time video image through a 1×1 convolution operation.

[0017] Preferably, the convolution filter performs sliding calculation on the matrix, and the formula is as follows:

[0018]

[0019] In the formula, is the final feature map of the real-time video image, is the offset function, are the corresponding elements in the matrix, is the convolution kernel, and is the height and width of the convolution kernel, is the step size of the convolution kernel, and is the internal index of the convolution kernel.

[0020] Preferably, the residual unit is used to downsample the input real-time video image to generate a primary feature map, and the formula is as follows:

[0021]

[0022] In the formula, is the primary feature map, is the operation function of the residual unit, The input real-time video image.

[0023] Preferably, the inspection robot further comprises a feature map detection module;

[0024] The feature map detection module is used to store the final feature map of the historical video images without abnormalities, and calculate the repetition value between the final feature map of the real-time video image and the final feature map of the historical video images without abnormalities, and judge whether the final feature map of the real-time video image currently acquired is a repeated feature map based on the repetition value, and exclude the repeated feature map.

[0025] Preferably, the calculation of the repetition value of the final feature map of the real-time video image and the final feature map of the historical video image without abnormality is performed by the following formula:

[0026]

[0027] In the formula, is a repeated value, is the corner point position of the final feature map of the real-time video image, is the corner point position of the final feature map of the historical video image without abnormalities, is the number of corner points.

[0028] Preferably, judging whether the final feature map of the real-time video image currently acquired is a repeated feature map based on the repetition value includes:

[0029] Determine whether the repetition value of the final feature map of the current acquired real-time video image is within a preset repetition range. If so, the final feature map of the current acquired real-time video image is marked as a repeated feature map. If not, it is output normally.

[0030] Preferably, the cloud platform is used to identify abnormal conditions of the final feature map of the real-time video image and issue abnormal instructions, including:

[0031] The cloud platform receives the final feature map of the real-time video image transmitted by the inspection robot through the communication module, compares it with the feature map library under the normal state saved historically, and identifies the anomaly in the final feature map of the real-time video image based on the classification model. When an anomaly is detected, the cloud platform generates a corresponding abnormal instruction according to the abnormal situation.

[0032] Beneficial effects:

[0033] 1. The present invention compares the real-time feature graph transmitted by the inspection robot with the historical normal feature graph library through the cloud platform, accurately identifies anomalies based on the classification model, and generates corresponding abnormal instructions. This method makes anomaly detection more intelligent and accurate, reduces false positives and missed positives, and improves the speed and pertinence of abnormal response;

[0034] 2. The visual feature reconstruction module of the present invention not only greatly improves the inspection robot's ability to process visual data, but also enhances its ability to detect minor anomalies and faults through advanced convolutional neural networks and deep learning technologies. The introduction of this module effectively makes up for the deficiencies in the visual processing capabilities of traditional inspection robots, especially in terms of automated and high-precision anomaly detection;

[0035] 3. The feature map detection module of the present invention analyzes the features of the image and compares it with the feature maps in the historical records to determine whether the current image is a duplicate of a previously recorded image. If duplicate images are detected, the system will exclude these duplicate images without the need to perform redundant processing or storage on them again. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a structural schematic diagram of a preferred embodiment of the present invention;

[0037] Figure 2 It is a schematic structural diagram of an inspection robot according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0038] The embodiments of the present invention are described in detail below. The following embodiments are implemented on the premise of the technical solution of the present invention, and detailed implementation methods and specific operation processes are given, but the protection scope of the present invention is not limited to the following embodiments.

[0039] The present invention designs an intelligent inspection robot system based on the Internet of Things. Figure 1 As shown, the technical solution specifically includes:

[0040] Inspection robots and cloud platforms;

[0041] The inspection robot includes a visual acquisition module, a visual feature reconstruction module, a communication module, a control module and a power module;

[0042] The visual acquisition module is used to collect real-time video images of the monitored object taken by the inspection robot during the inspection process;

[0043] The visual feature reconstruction module is used to extract the final feature map of the real-time video image, including constructing the residual unit, upsampling layer and final feature map generation layer in the convolutional neural network; the residual unit is used to downsample the input real-time video image to generate a primary feature map; the upsampling layer is used to restore the resolution of the primary feature map; the final feature map generation layer is used to perform convolution processing on the restored primary feature map and output the final feature map of the real-time video image;

[0044] The communication module is used to transmit the final feature map of the real-time video image to the cloud platform in real time;

[0045] The cloud platform is used to identify abnormal conditions in the final feature map of the real-time video image and issue abnormal instructions;

[0046] The control module is used to receive abnormal instructions from the cloud platform and drive the inspection robot to execute actions corresponding to the abnormal instructions;

[0047] The power module is used to provide power to the inspection robot.

[0048] Preferably, the final feature map generation layer is used to perform convolution processing on the restored primary feature map and output the final feature map of the real-time video image, including:

[0049] The final feature map generation layer contains convolution filters and 1×1 convolution operations;

[0050] Each pixel value in the restored primary feature map is mapped to the corresponding element in the matrix, and the convolution filter performs sliding calculation on the matrix to output the final feature map of the real-time video image through a 1×1 convolution operation.

[0051] Preferably, the convolution filter is calculated by sliding on the matrix, and the formula is as follows:

[0052]

[0053] In the formula, is the final feature map of the real-time video image, is the offset function, are the corresponding elements in the matrix, is the convolution kernel, and is the height and width of the convolution kernel, is the step size of the convolution kernel, and is the internal index of the convolution kernel.

[0054] Preferably, the residual unit is used to downsample the input real-time video image to generate a primary feature map, and the formula is as follows:

[0055]

[0056] In the formula, is the primary feature map, is the operation function of the residual unit, The input real-time video image.

[0057] Specifically, the visual feature reconstruction module uses a convolutional neural network (CNN). When processing real-time video images, CNN can extract spatial position features very well. The convolution layer of CNN scans local areas in real-time video images and extracts local features such as edges and corners, which can capture information at different positions in real-time video images. Among them, corner points are areas with more dramatic changes in real-time video images, which are the intersection of two edges. Corner points play a very critical role in the structure of objects, have high stability, and are suitable for comparison. Therefore, the subsequent feature map detection module uses corner points for comparison when calculating duplicate values.

[0058] In addition, the convolution filter is applied to slide on the matrix as follows: the corresponding elements in the input matrix It is obtained from the previous network layer and contains preliminary features extracted from the original video image. For each position in the feature map , calculate the offset, and adjust the sampling point of the convolution kernel according to the calculated offset. , the points that should have been sampled Adjust to , and perform convolution operations on the sampled points after applying the offset adjustment. The adjusted input points are combined with the dynamically generated convolution kernel Perform a dot product operation to obtain the convolution result, and calculate the final feature map through the adjusted convolution operation. Traditional convolution calculations usually perform sampling on a fixed grid of the input feature map. As the step size slides on the feature map according to certain rules, the receptive field of the convolution kernel is no longer limited to a regular grid by adding an offset function. The convolution kernel can adjust the sampling position according to the specific features of the input, and flexibly respond to irregular deformations, bending and other structures in the image, which meets the irregular inspection conditions of the inspection robot in response to abnormal situations. The offset function can help the convolution kernel better capture these changes and improve the inspection robot's ability to collect images during inspections. For example, due to the adaptive ability of the offset function, the convolution kernel can more accurately capture important corner points in the image collected by the inspection robot, avoiding corner point deviations caused by image deformation, blurring and other problems, thereby facilitating the subsequent and It will also be more accurate. In addition, the shape and angle of the inspected object may change in different perspectives or frames. The offset function can adjust the sampling position of the convolution kernel according to these changes to help better identify new corner points.

[0059] In addition, CNN extracts local features through convolution operations, and the final feature map generated has a smaller size than the original video image but retains key information. This efficient feature map not only reduces the burden of data processing on the cloud platform, but also improves the processing speed. The cloud platform does not need to conduct a comprehensive analysis of each frame of video image, but only needs to process the streamlined feature map generated by the visual feature reconstruction module. In this way, the cloud platform can simply make a quick judgment based on the classification model without consuming a lot of computing resources to process redundant information.

[0060] Preferably, if Figure 2 As shown, the inspection robot also includes a feature map detection module;

[0061] The feature map detection module is used to store the final feature map of historical video images without abnormalities, and calculate the repetition value between the final feature map of the real-time video image and the final feature map of the historical video image without abnormalities. Based on the repetition value, it is determined whether the final feature map of the real-time video image currently acquired is a repeated feature map, and the repeated feature maps are excluded.

[0062] Preferably, the repetition value of the final feature map of the real-time video image and the final feature map of the historical video image without abnormality is calculated, and the formula is as follows:

[0063]

[0064] In the formula, is a repeated value, is the corner point position of the final feature map of the real-time video image, is the corner point position of the final feature map of the historical video image without abnormalities, is the number of corner points.

[0065] Preferably, judging whether the final feature map of the real-time video image currently acquired is a repeated feature map based on the repetition value includes:

[0066] Determine whether the repetition value of the final feature map of the current acquired real-time video image is within a preset repetition range. If so, the final feature map of the current acquired real-time video image is marked as a repeated feature map. If not, it is output normally.

[0067] Specifically, compared with the traditional video image matching method (pixel level or histogram comparison), the formula of the repeated value can more accurately capture the detailed changes of the image through the corner position and cosine calculation, especially when processing feature maps with corner changes (such as rotation and scaling). Although cosine calculation is not a new concept in image feature matching, combining it with the corner position and the final feature map output by CNN, and making quick judgments through a simplified formula structure, can improve the calculation efficiency while maintaining accuracy, and is particularly suitable for use in real-time video processing scenarios. In addition, for the area that the robot needs to inspect, there are multiple robots that synchronously collect the area that needs to be inspected at different angles within a certain period of time, resulting in the image features being mirror-inverted or partially inverted. The above formula for repeated values ​​can output close to half of the original repeated value in the case of mirror inversion, and local inversion occurs. The repeated value formula can also minimize errors, thereby improving the accuracy of repeated value judgment and thus improving the accuracy of repeated feature map judgment.

[0068] also, Here, t represents time, which refers to the repetition value calculated at time point t, and is used to represent the final feature map of the real-time video image at that moment. Therefore, the judgment process of judging whether the final feature map of the real-time video image currently acquired is a repetition feature map of the historical record based on the repetition value also has a time attribute. The specific judgment process is that whenever a new real-time video image is acquired (time point t), its repetition value is calculated. , and compare this repetition value with the preset repetition range, and add a time threshold to define whether the image within this time threshold can be considered as repeated. This threshold can be a fixed time period (such as a few seconds or minutes) or a time interval relative to a reference time point. The calculated time difference Δt is compared with the preset time threshold. If Δt is less than or equal to the time threshold, the current image is considered to be close to the historical image in time and repeatability can be determined; otherwise, it is considered that they are not within a reasonable time range and no further comparison is required.

[0069] Preferably, the cloud platform is used to identify abnormal conditions of the final feature map of the real-time video image and issue abnormal instructions, including:

[0070] The cloud platform receives the final feature map of the real-time video image transmitted by the inspection robot through the communication module, compares it with the feature map library under the normal state saved historically, and identifies the anomaly in the final feature map of the real-time video image based on the classification model. When an anomaly is detected, the cloud platform generates corresponding abnormal instructions according to the abnormal situation.

[0071] Specifically, once the cloud platform detects an abnormality, it generates corresponding abnormal instructions, including equipment failure, temperature abnormality, surface damage, etc., evaluates the severity of the abnormality, and divides it into minor abnormalities, serious abnormalities, etc., and generates different response strategies based on different levels, and stores the abnormal feature map and related information in the database for subsequent analysis or historical data tracking; the generated abnormal instructions include stopping the robot, rechecking, alarming, taking images from more angles, and further monitoring the abnormal parts. Later, through the communication module, the cloud platform sends the generated abnormal instructions back to the inspection robot to guide the robot drive, such as returning to the specified position, alarming, adjusting the inspection path, and collecting more detailed images.

[0072] The preferred specific embodiments of the present invention are described in detail above. It should be understood that ordinary technicians in the field can make many modifications and changes based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by technicians in the technical field based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. An intelligent inspection robot system based on the Internet of Things, characterized in that: include: Inspection robots and cloud platforms; The inspection robot includes a visual acquisition module, a visual feature reconstruction module, a communication module, a control module and a power module; The visual acquisition module is used to collect real-time video images of the monitored object taken by the inspection robot during the inspection process; The visual feature reconstruction module is used to extract the final feature map of the real-time video image, including constructing a residual unit, an upsampling layer and a final feature map generation layer in a convolutional neural network; the residual unit is used to downsample the input real-time video image to generate a primary feature map; the upsampling layer is used to restore the resolution of the primary feature map; The final feature map generation layer is used to perform convolution processing on the restored primary feature map and output the final feature map of the real-time video image; The communication module is used to transmit the final feature map of the real-time video image to the cloud platform in real time; The cloud platform is used to identify abnormal conditions of the final feature map of the real-time video image and issue abnormal instructions; The control module is used to receive abnormal instructions from the cloud platform and drive the inspection robot to perform actions corresponding to the abnormal instructions; The power module is used to provide power to the inspection robot; The inspection robot also includes a feature map detection module; The feature map detection module is used to store the final feature map of the historical video image without abnormality, and calculate the repetition value of the final feature map of the real-time video image and the final feature map of the historical video image without abnormality, and judge whether the final feature map of the real-time video image currently acquired is a repeated feature map based on the repetition value, and exclude the repeated feature map; The formula for calculating the repetition value of the final feature map of the real-time video image and the final feature map of the historical video image without abnormality is as follows: In the formula, Duplicate(t) is the duplicate value, w k is the corner point position of the final feature map of the real-time video image, W k is the corner point position of the final feature map of the historical video image without abnormalities, and d is the number of corner points; The step of judging whether the final feature map of the real-time video image currently acquired is a repeated feature map based on the repetition value includes: Determine whether the repetition value of the final feature map of the current acquired real-time video image is within a preset repetition range. If so, the final feature map of the current acquired real-time video image is marked as a repeated feature map. If not, it is output normally.

2. According to the Internet of Things-based intelligent inspection robot system of claim 1, it is characterized in that: The final feature map generation layer is used to perform convolution processing on the restored primary feature map and output the final feature map of the real-time video image, including: The final feature map generation layer includes a convolution filter and a 1×1 convolution operation; Each pixel value in the restored primary feature map is mapped to a corresponding element in a matrix, and the convolution filter performs sliding calculation on the matrix to output a final feature map of the real-time video image through a 1×1 convolution operation.

3. The intelligent inspection robot system based on the Internet of Things according to claim 2 is characterized in that: The convolution filter performs sliding calculation on the matrix, and the formula is as follows: Where Z(i, j) is the final feature map of the real-time video image, Δ(i, j) is the offset function, Q(i, j) is the corresponding element in the matrix, W(x, y) is the convolution kernel, A and B are the height and width of the convolution kernel, s is the step size of the convolution kernel, and m and n are the internal indexes of the convolution kernel.

4. The intelligent inspection robot system based on the Internet of Things according to claim 1 is characterized in that: The residual unit is used to downsample the input real-time video image to generate a primary feature map, and the formula is as follows: C=G(v)+v In the formula, C is the primary feature map, G is the operation function of the residual unit, and v is the input real-time video image.

5. The intelligent inspection robot system based on the Internet of Things according to claim 1 is characterized in that: The cloud platform is used to identify abnormal conditions of the final feature map of the real-time video image and issue abnormal instructions, including: The cloud platform receives the final feature map of the real-time video image transmitted by the inspection robot through the communication module, compares it with the feature map library under the normal state saved historically, and identifies the anomaly in the final feature map of the real-time video image based on the classification model. When an anomaly is detected, the cloud platform generates a corresponding abnormal instruction according to the abnormal situation.

Images