Intelligent inspection method and system, inspection robot and medium

Abstract

The invention discloses an intelligent inspection method and system, an inspection robot and a medium, and relates to the technical field of inspection robots, and the intelligent inspection method comprises the steps: obtaining multi-modal data, and carrying out the preprocessing of the multi-modal data, and obtaining the preprocessed multi-modal data; extracting each data feature in the preprocessed multi-modal data, and fusing each data feature to obtain a comprehensive data feature; the comprehensive data features are input into a preset large model, an inspection decision is obtained, operation is carried out according to the inspection decision, and the large model is obtained through training by taking preset actual comprehensive data features as model input data and taking a preset actual inspection decision corresponding to the actual comprehensive data features as a model training label. The technical scheme of the invention aims to solve the technical problem of how to improve the inspection efficiency of the inspection robot.

Description

Technical Field

[0001] The present application relates to the technical field of inspection robots, and in particular to an intelligent inspection method, system, inspection robot and medium. Background Art

[0002] Industrial sites such as mines and factories have complex and changeable working environments, such as narrow spaces, irregular terrain, various obstacles and potential safety hazards. These factors make it difficult for inspection robots to go deep into specific environments to conduct comprehensive and detailed inspections, resulting in low inspection efficiency of inspection robots.

[0003] Therefore, how to improve the inspection efficiency of inspection robots is a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0004] The main purpose of this application is to provide an intelligent inspection method, system, inspection robot and medium, aiming to solve the technical problem of how to improve the inspection efficiency of the inspection robot.

[0005] To achieve the above objectives, the present application proposes an intelligent inspection method, which is applied to an inspection robot and includes:

[0006] Acquiring multimodal data, and preprocessing the multimodal data to obtain preprocessed multimodal data;

[0007] Extracting each data feature from the preprocessed multimodal data, and fusing each data feature to obtain a comprehensive data feature;

[0008] The comprehensive data features are input into a preset big model to obtain an inspection decision, and the big model is run according to the inspection decision, wherein the big model is trained using the preset actual comprehensive data features as model input data and using the actual comprehensive data features corresponding to the preset actual inspection decisions as model training labels.

[0009] In one embodiment, the multimodal data includes: Li DAR (Light Detection And Ranging) data, GNSS (Global Navigation Satellite System) data and image data;

[0010] The step of preprocessing the multimodal data to obtain preprocessed multimodal data includes:

[0011] Filtering the Li DAR data by a statistical filtering method to obtain filtered Li DAR data;

[0012] Correcting the GNSS data by a differential GNSS method to obtain corrected GNSS data;

[0013] The image data is enhanced and denoised in sequence to obtain corrected image data.

[0014] In one embodiment, the step of extracting each data feature from the preprocessed multimodal data includes:

[0015] Extracting shape features and distance features from the filtered Li DAR data;

[0016] Extracting position features, velocity features and direction features from the corrected GNSS data;

[0017] The color features, edge contour features and texture features in the corrected image data are extracted.

[0018] In one embodiment, the step of extracting shape features and distance features from the filtered Li DAR data comprises:

[0019] Extracting point cloud geometric features from the filtered Li DAR data, and calculating the Euclidean distance between points in the filtered Li DAR data;

[0020] A distance feature is obtained according to the Euclidean distance, and a shape feature is determined according to the point cloud geometric feature and the Euclidean distance.

[0021] In one embodiment, the step of extracting the position features, speed features and direction features from the corrected GNSS data comprises:

[0022] Converting the corrected GNSS data into plane coordinates and extracting position features in the plane coordinates;

[0023] determining a velocity characteristic based on a rate of change of position information in the corrected GNSS data;

[0024] A directional feature is determined based on the direction of a line between consecutive position points in the corrected GNSS data.

[0025] In one embodiment, the step of extracting color features, edge contour features and texture features from the corrected image data comprises:

[0026] Extracting color features in the corrected image data by a color histogram method;

[0027] Extracting edge contour features in the corrected image data by using a Canny edge detection algorithm;

[0028] The texture features in the corrected image data are extracted through local binary patterns.

[0029] In one embodiment, the step of fusing the data features to obtain a comprehensive data feature includes:

[0030] The data features are fused through the Kalman filter algorithm to obtain comprehensive data features.

[0031] In addition, to achieve the above purpose, the present application also proposes an intelligent inspection system, which is applied to an inspection robot and includes:

[0032] A preprocessing module, used to acquire multimodal data and preprocess the multimodal data to obtain preprocessed multimodal data;

[0033] A feature extraction module, used to extract each data feature from the preprocessed multimodal data, and fuse each data feature to obtain a comprehensive data feature;

[0034] The decision module is used to input the comprehensive data features into a preset big model to obtain an inspection decision and operate according to the inspection decision, wherein the big model is trained with the preset actual comprehensive data features as model input data and the actual comprehensive data features corresponding to the preset actual inspection decisions as model training labels.

[0035] In addition, to achieve the above objectives, the present application also proposes a patrol robot, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the intelligent patrol method described above.

[0036] In addition, to achieve the above objectives, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the intelligent inspection method described above are implemented.

[0037] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the intelligent inspection method described above are implemented.

[0038] In the present application, by acquiring multimodal data and preprocessing the multimodal data to obtain preprocessed multimodal data, the accuracy of the multimodal data used for environmental perception can be improved; then, by extracting each data feature in the preprocessed multimodal data and fusing the data features to obtain comprehensive data features, the complementary advantages of different data in the multimodal data can be achieved, thereby further improving the accuracy of the data; then, the comprehensive data features are input into a preset large model to obtain inspection decisions, and the inspection decisions are operated, wherein the large model is trained with preset actual comprehensive data features as model input data, and with actual comprehensive data features corresponding to preset actual inspection decisions as model training labels. The fused features can be deeply learned through the large model, the surrounding environment can be modeled, and state inference and prediction can be performed.

[0039] In this application, by comprehensively analyzing multimodal data, abnormal conditions in the environment (such as equipment failure, environmental changes, etc.) can be detected. On this basis, according to the three-dimensional environmental information and position information, this application can also plan the robot's movement path to avoid collisions and obstacles. Therefore, this application can achieve comprehensive perception of environmental information through multimodal data and large models, and make relevant decisions based on the status of the equipment, so that the inspection robot can adapt to various complex working environments, thereby improving the inspection efficiency of the inspection robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0042] Figure 1 A flowchart of the first embodiment of the intelligent inspection method of the present application is provided;

[0043] Figure 2 This is a schematic diagram of the LiDAR data feature extraction process of an embodiment of the intelligent inspection method of this application;

[0044] Figure 3 This is a schematic diagram of the GNSS data feature extraction process of an embodiment of the intelligent inspection method of the present application;

[0045] Figure 4 This is a schematic diagram of the image data feature extraction process of an embodiment of the intelligent inspection method of the present application;

[0046] Figure 5 This is a flowchart of a specific embodiment of the intelligent inspection method of this application;

[0047] Figure 6 This is a schematic diagram of the module structure of the intelligent inspection system according to an embodiment of the present application;

[0048] Figure 7 Schematic diagram of the equipment structure of the hardware operating environment involved in the intelligent inspection method in the embodiment of the present application.

[0049] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0051] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0052] Nowadays, industrial sites such as mines and factories have complex and changeable working environments, such as narrow spaces, irregular terrain, various obstacles and potential safety hazards. These factors make it difficult for inspection robots to go deep into specific environments to conduct comprehensive and detailed inspections, resulting in low inspection efficiency of inspection robots.

[0053] Therefore, how to improve the inspection efficiency of inspection robots is a technical problem that needs to be urgently solved by technical personnel in this field.

[0054] In the present application, by acquiring multimodal data and preprocessing the multimodal data to obtain preprocessed multimodal data, the accuracy of the multimodal data used for environmental identification can be improved; then, by extracting each data feature in the preprocessed multimodal data and fusing the data features to obtain comprehensive data features, the complementary advantages of different data in the multimodal data can be achieved, thereby further improving the accuracy of the data; then, the comprehensive data features are input into a preset large model to obtain an inspection decision, and the inspection decision is operated, wherein the large model is trained with preset actual comprehensive data features as model input data and with actual comprehensive data features corresponding to preset actual inspection decisions as model training labels. The fused features can be deeply learned through the large model, the surrounding environment can be modeled, and state inference and prediction can be performed.

[0055] In this application, by comprehensively analyzing multimodal data, abnormal conditions in the environment (such as equipment failure, environmental changes, etc.) can be detected. On this basis, according to the three-dimensional environmental information and position information, this application can also plan the robot's movement path to avoid collisions and obstacles. Therefore, this application can achieve comprehensive perception of environmental information through multimodal data and large models, and make relevant decisions based on the equipment status, so that the inspection robot can adapt to various complex working environments, thereby improving the inspection efficiency.

[0056] It should be noted that the execution subject of the intelligent inspection method of the present application may be an inspection robot with data processing, network communication and program running functions. The following takes the inspection robot as an example to illustrate various embodiments of the intelligent inspection method of the present application.

[0057] The present application embodiment provides an intelligent inspection method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the intelligent inspection method of the present application.

[0058] In this embodiment, the intelligent inspection method includes steps S10 to S30:

[0059] Step S10, acquiring multimodal data, and preprocessing the multimodal data to obtain preprocessed multimodal data;

[0060] It should be noted that multimodal data refers to data collected by multiple different sensors. Various sensors for sensing the environment can be pre-installed on the inspection robot. For example, LiDAR, camera, and GNSS sensor can be pre-installed. Then, the inspection robot can sense its working environment based on LiDAR, camera, and GNSS sensor.

[0061] In this embodiment, after acquiring the data collected by each sensor, since there are noise points in the collected data, it is necessary to pre-process the data collected by each sensor to improve the data quality, thereby increasing the probability of the inspection robot correctly perceiving the environment.

[0062] In a feasible implementation manner, the multimodal data includes: LiDAR data, GNSS data and image data, and the above step S10 further includes:

[0063] Step S101, filtering the LiDAR data by a statistical filtering method to obtain filtered LiDAR data;

[0064] Step S102, correcting the GNSS data by a differential GNSS method to obtain corrected GNSS data;

[0065] Step S103, performing enhancement and noise reduction processing on the image data in sequence to obtain corrected image data.

[0066] It is understandable that, since different sensors have different characteristics, the data collected by different sensors can be processed by different preprocessing methods, thereby further improving the quality of the data.

[0067] In this embodiment, the preprocessing includes filtering, correction, noise reduction, enhancement and other processing. For Li DAR data, the mean and standard deviation of the point cloud data can be calculated using a statistical filtering method, and then the discrete points in the point cloud data can be determined based on the mean and standard deviation, and then the discrete points can be removed. Specifically, in a feasible implementation, points that deviate from the mean or standard deviation by more than a certain threshold can be regarded as noise points and removed. For GNSS data, error correction can be performed using a differential GNSS method. Specifically, the reference station receives signals from multiple satellites and calculates the difference between the actual receiving position and the theoretical position of these signals, that is, the error; then, the reference station sends the error information to the inspection robot through a radio signal or other means; after receiving the error information, the inspection robot combines the error information with its own GNSS data, and corrects its positioning result through a correction algorithm (for example, Kalman filtering), thereby improving the positioning accuracy. For image data, since the image data is obtained through a camera, and the camera has lens distortion, before obtaining the image data, it is also necessary to eliminate the lens distortion through camera calibration, including radial distortion and tangential distortion, to improve the geometric accuracy of the image. Next, after the camera is calibrated, the image data acquired by the camera can be enhanced by using histogram equalization technology to enhance the contrast of the image and improve the overall brightness and clarity of the image. Finally, the enhanced image is denoised by using Gaussian filtering to remove random noise in the image, thereby obtaining the corrected image data.

[0068] Step S20, extracting each data feature in the preprocessed multimodal data, and fusing each data feature to obtain a comprehensive data feature;

[0069] It should be noted that data features refer to meaningful information extracted from data that can be used for further analysis or in machine learning models. Comprehensive data features are data features obtained by integrating various data features.

[0070] Step S30, input the comprehensive data features into a preset big model to obtain an inspection decision, and operate according to the inspection decision, wherein the big model is trained with the preset actual comprehensive data features as model input data and the actual comprehensive data features corresponding to the preset actual inspection decisions as model training labels.

[0071] It should be noted that the actual comprehensive data features refer to the comprehensive data features used to test the large model, the actual inspection decision refers to the inspection decision corresponding to the actual comprehensive data features, and the inspection decision refers to the operation strategy of the inspection robot. It is understandable that in order to obtain the operation strategy of the inspection robot that meets the current environment based on the large model, the large model needs to be trained in advance.

[0072] Specifically, the training process can be: pre-acquire the data obtained by the camera, LiDAR, and GNSS receiver on the inspection robot, and fuse the extracted features after filtering, denoising, etc., so as to obtain the actual comprehensive data features. Then, obtain the manually input operation strategy corresponding to the actual comprehensive data features. Then, input the actual comprehensive data features into the large model, and compare the output of the large model with the actual inspection decision. If there is a difference between the output of the large model and the actual inspection decision, fine-tune the parameters of the large model until the gap between the output of the large model and the actual inspection decision meets expectations.

[0073] In a feasible implementation manner, the above step S20 further includes:

[0074] Step S201, extracting shape features and distance features from the filtered LiDAR data;

[0075] It should be noted that LiDAR obtains 3D point cloud data by scanning the surrounding environment in real time, wherein the 3D point cloud data can reflect the 3D structural information of the environment. Since the inspection robot needs to adapt to different environments, it is necessary to determine the shape of the surrounding obstacles (i.e., shape features) based on the filtered LiDAR data, and it is also necessary to determine the density of objects around the inspection robot (i.e., distance features) based on the filtered LiDAR data.

[0076] Step S202, extracting position features, speed features and direction features from the corrected GNSS data;

[0077] It should be noted that the position feature refers to the position of the inspection robot, the speed feature refers to the feature related to the movement speed of the inspection robot, and the direction feature refers to the feature related to the running direction of the inspection robot.

[0078] Step S203: extracting color features, edge contour features and texture features from the corrected image data.

[0079] It should be noted that the color feature refers to the color of the object in the image data, the edge contour feature refers to the contour of the object, and the texture feature refers to the texture of the object.

[0080] In this embodiment, the shape features, distance features, position features, speed features, direction features, color features, edge contour features, and texture features all represent specific conditions of the environment. Therefore, it is necessary to plan inspection decisions for the inspection robot based on the above features so that the inspection robot can go deep into the working environment, thereby improving the inspection efficiency.

[0081] Please refer to Figure 2 In a feasible implementation manner, the above step S201 includes:

[0082] Step S2011, extracting point cloud geometric features from the filtered LiDAR data, and calculating the Euclidean distance between points in the filtered LiDAR data;

[0083] It should be noted that the geometric features of point clouds include but are not limited to: point position (the coordinates of each point in three-dimensional space), point density (the number of points per unit volume), and curvature (the curvature of a point on the point cloud surface, which can be used to describe the degree of curvature of the object surface).

[0084] Step S2012, obtaining a distance feature according to the Euclidean distance, and determining a shape feature according to the point cloud geometric feature and the Euclidean distance.

[0085] In this embodiment, distance features can be extracted by statistically analyzing the Euclidean distances between points in the point cloud. Specifically, distance features include but are not limited to the average distance, maximum distance, minimum distance, and distance distribution between each Euclidean distance. Shape features include but are not limited to the boundary contour and internal structure of an object. In this embodiment, shape features can be determined by analyzing the geometric features of the point cloud and the Euclidean distance.

[0086] In a feasible implementation, the point cloud geometric features and Euclidean distances may be processed by a pre-trained machine learning model to output shape features.

[0087] In another feasible implementation, the step of obtaining shape features may also be: points in the point cloud may be grouped into different clusters through a clustering algorithm (such as K-means), each cluster representing an object or a part of an object, and then the shape features of these objects may be extracted, such as boundaries, contours, volumes, surface areas, etc.

[0088] Please refer to Figure 3 In a feasible implementation manner, the above step S202 includes:

[0089] Step S2021, converting the corrected GNSS data into plane coordinates, and extracting position features in the plane coordinates;

[0090] It should be noted that GNSS data usually refers to coordinates in three-dimensional space. Since GNSS data is used for navigation in this embodiment, it is necessary to convert GNSS data into plane coordinates. Then, in the plane coordinate system, each point has its specific position characteristics, such as the offset of the coordinate value (x, y) relative to a reference point, the relative position to a specific path or area, etc. Therefore, after obtaining the plane coordinates corresponding to the inspection robot, the position characteristics corresponding to the plane coordinates can also be obtained.

[0091] Step S2022, determining a velocity characteristic based on a rate of change of the position information in the corrected GNSS data;

[0092] It should be noted that at consecutive time points, GNSS data will provide a series of position information. By comparing the changes in these position information over time, the rate of change of the position can be calculated. Then, the rate of change of the inspection robot's speed can be calculated based on the rate of change of the inspection robot's position, and the rate of change of the speed can be used as the speed feature.

[0093] Step S2022: determining a direction feature based on the direction of a line between consecutive position points in the corrected GNSS data.

[0094] It should be noted that in GNSS data, continuous position points can be regarded as points on the trajectory of an object moving in space. By connecting these points, the object's movement trajectory can be obtained. The connection direction refers to the direction indicated by the line between two adjacent position points, which reflects the movement direction of the object at a certain moment.

[0095] In this embodiment, the directional features generally include moving direction, direction change rate (ie, turning speed or angular velocity), and the like.

[0096] In a feasible implementation, the moving direction can be obtained by calculating the direction angle of the line between adjacent position points, and the direction change rate can be obtained by calculating the change in the direction angle in adjacent time periods and dividing it by the time interval.

[0097] Please refer to Figure 4 In a feasible implementation manner, the above step S203 includes:

[0098] Step S2031, extracting color features in the corrected image data by using a color histogram method;

[0099] It should be noted that the color histogram is a method of statistically analyzing the color distribution in an image. It divides the color space (such as RGB, HSV, etc.) in the image into several color intervals (or "bins"), and then counts the number or proportion of pixels in each color interval.

[0100] In this embodiment, in the corrected image data, it is first necessary to convert the image from the original color space (such as RGB) to a color space suitable for color histogram statistics (such as HSV, because it is closer to the way humans perceive colors). Then, according to the selected color space and the required color resolution, the color space is divided into a certain number of color intervals. Next, traverse each pixel in the image, map its color to the corresponding color interval, and count the number or proportion of pixels in each interval. The color histogram finally obtained is the color feature of the image.

[0101] Step S2032, extracting edge contour features in the corrected image data using a Canny edge detection algorithm;

[0102] In this embodiment, in the corrected image data, a Gaussian filter is first applied to smooth the image to reduce the impact of noise on edge detection. Then, the gradient strength and direction of each pixel in the image are calculated to identify possible edge points. Next, through the non-maximum suppression (NMS) process, the local maximum points in the gradient direction are retained as candidate edge points. Finally, a dual threshold detection strategy is used to determine the final edge points, and these points are connected to form an edge contour.

[0103] Step S2033: extracting texture features in the corrected image data through local binary patterns.

[0104] It should be noted that the local binary pattern is a simple and effective feature descriptor for texture classification. It extracts texture information by binary encoding the pixels in the local area of ​​the image. In the local binary pattern, a central pixel is first selected, and the pixels around it (usually the pixels in a circular neighborhood) are compared with the central pixel. If the grayscale value of the surrounding pixels is greater than or equal to the grayscale value of the central pixel, the position is marked as 1; otherwise it is marked as 0. In this way, a binary number is obtained, that is, the local binary pattern value of the central pixel. By converting each pixel in the image into a local binary pattern value and counting the local binary pattern histogram of the entire image, the texture features of the image can be obtained.

[0105] Exemplarily, in the corrected image data, the neighborhood size and shape (usually circular or square) of the local binary pattern is first determined. Then, each pixel in the image is traversed, taken as the central pixel, and its local binary pattern value is calculated. Next, the local binary pattern histogram of the entire image is counted, that is, the local binary pattern values ​​of all pixels are counted to obtain the frequency of occurrence of each local binary pattern value. The local binary pattern histogram finally obtained is the texture feature of the image.

[0106] In a feasible implementation manner, the above step S20 further includes:

[0107] Step S204, fusing the data features through a Kalman filter algorithm to obtain a comprehensive data feature.

[0108] It is understandable that in the process of the inspection robot moving, it is necessary to obtain multimodal data in real time, so the amount of data processing in the inspection robot is extremely large. In order to fuse the extracted data features in a relatively short time, the present application fuses the data features through the Kalman filter algorithm, so that the data processing of the present application is more real-time, thereby also facilitating the inspection robot to adjust the inspection strategy in real time according to different environments.

[0109] Please refer to Figure 5 , Figure 5 This is a schematic diagram of a specific embodiment of the intelligent inspection method of this application. Figure 5 In the process, the perception layer is used for data collection and data preprocessing. After the preprocessing is completed, the feature extraction layer is used to extract the point cloud and distance features in the Li DAR data, the position, speed, and direction features in the GNSS data, and the color, shape, and texture features in the camera data (image data). Then, the features extracted in the feature extraction layer are fused by the Kalman filter algorithm in the fusion layer, and then the fused features are input into the large model in the decision layer, and the large model is used to detect anomalies and make decisions. Then, the control layer can control the robot's actuator according to the instructions of the decision layer to realize actions such as moving, turning, and stopping, and timely warn of detected anomalies and faults and intelligently plan the inspection path to improve the inspection efficiency and accuracy. And in the process of executing the task, the motion state is monitored in real time to ensure that the task is executed according to the planned path, and the safety and reliability of the inspection are guaranteed. Thus, the present application can make the inspection robot suitable for various working environments, so that the present application improves the inspection efficiency of the inspection robot.

[0110] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the intelligent inspection method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0111] The present application also provides an intelligent inspection system, which is applied to an inspection robot. Figure 6 , the intelligent inspection system comprises:

[0112] A preprocessing module 10 is used to acquire multimodal data and preprocess the multimodal data to obtain preprocessed multimodal data;

[0113] A feature extraction module 20 is used to extract each data feature in the preprocessed multimodal data and fuse each data feature to obtain a comprehensive data feature;

[0114] The decision module 30 is used to input the comprehensive data features into a preset big model to obtain an inspection decision and operate according to the inspection decision, wherein the big model is trained with the preset actual comprehensive data features as model input data and the actual comprehensive data features corresponding to the preset actual inspection decisions as model training labels.

[0115] In one embodiment, the multimodal data includes: Li DAR data, GNSS data and image data, and the preprocessing module 10 is further used to:

[0116] Filtering the Li DAR data by a statistical filtering method to obtain filtered Li DAR data;

[0117] Correcting the GNSS data by a differential GNSS method to obtain corrected GNSS data;

[0118] The image data is enhanced and denoised in sequence to obtain corrected image data.

[0119] In one embodiment, the feature extraction module 20 is further used for:

[0120] Extracting shape features and distance features from the filtered Li DAR data;

[0121] Extracting position features, velocity features and direction features from the corrected GNSS data;

[0122] The color features, edge contour features and texture features in the corrected image data are extracted.

[0123] In one embodiment, the feature extraction module 20 is further used for:

[0124] Extracting point cloud geometric features from the filtered Li DAR data, and calculating the Euclidean distance between points in the filtered Li DAR data;

[0125] A distance feature is obtained according to the Euclidean distance, and a shape feature is determined according to the point cloud geometric feature and the Euclidean distance.

[0126] In one embodiment, the feature extraction module 20 is further used for:

[0127] Converting the corrected GNSS data into plane coordinates and extracting position features in the plane coordinates;

[0128] determining a velocity characteristic based on a rate of change of position information in the corrected GNSS data;

[0129] A directional feature is determined based on the direction of a line between consecutive position points in the corrected GNSS data.

[0130] In one embodiment, the feature extraction module 20 is further used for:

[0131] Extracting color features in the corrected image data by a color histogram method;

[0132] Extracting edge contour features in the corrected image data by using a Canny edge detection algorithm;

[0133] The texture features in the corrected image data are extracted through local binary patterns.

[0134] In one embodiment, the feature extraction module 20 is further used for:

[0135] The data features are fused through the Kalman filter algorithm to obtain comprehensive data features.

[0136] The intelligent inspection system provided by the present application adopts the intelligent inspection method in the above embodiment, which can solve the technical problem of how to improve the inspection efficiency of the inspection robot. Compared with the prior art, the beneficial effects of the intelligent inspection system provided by the present application are the same as the beneficial effects of the intelligent inspection method provided by the above embodiment, and other technical features of the intelligent inspection system are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0137] The present application provides an inspection robot, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the intelligent inspection method in the above-mentioned embodiment one.

[0138] Reference below Figure 7 , which shows a structural schematic diagram of an inspection robot suitable for implementing an embodiment of the present application. Figure 7 The inspection robot shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0139] like Figure 7As shown, the inspection robot may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read On ly Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. Various programs and data required for the operation of the inspection robot are also stored in RAM1004. The processing device 1001, ROM1002, and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the inspection robot to communicate with other devices wirelessly or by wire to exchange data. Although the inspection robot with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0140] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0141] The inspection robot provided by the present application adopts the intelligent inspection method in the above embodiment, which can solve the technical problem of how to improve the inspection efficiency of the inspection robot. Compared with the prior art, the beneficial effects of the inspection robot provided by the present application are the same as the beneficial effects of the intelligent inspection method provided by the above embodiment, and the other technical features of the inspection robot are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0142] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0143] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0144] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the intelligent inspection method in the above-mentioned embodiment.

[0145] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM: Erasable Programmab le Read On ly Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read On ly Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0146] The computer-readable storage medium may be included in the inspection robot; or may exist independently without being assembled into the inspection robot.

[0147] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the inspection robot, the inspection robot is enabled to: obtain multimodal data, and preprocess the multimodal data to obtain preprocessed multimodal data; extract each data feature in the preprocessed multimodal data, and fuse each data feature to obtain comprehensive data features; input the comprehensive data features into a preset large model to obtain inspection decisions, and operate according to the inspection decisions, wherein the large model is trained using preset actual comprehensive data features as model input data, and using the actual comprehensive data features corresponding to preset actual inspection decisions as model training labels.

[0148] The computer program code for performing the operation of the present application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0149] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0150] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0151] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned intelligent inspection method, and can solve the technical problem of how to improve the inspection efficiency of the inspection robot. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the intelligent inspection method provided by the above-mentioned embodiment, and will not be repeated here.

[0152] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. An intelligent inspection method, characterized in that: The intelligent inspection method is applied to an inspection robot, and the intelligent inspection method comprises: Acquiring multimodal data, and preprocessing the multimodal data to obtain preprocessed multimodal data; Extracting each data feature from the preprocessed multimodal data, and fusing each data feature to obtain a comprehensive data feature; The comprehensive data features are input into a preset big model to obtain an inspection decision, and the big model is run according to the inspection decision, wherein the big model is trained using the preset actual comprehensive data features as model input data and using the actual comprehensive data features corresponding to the preset actual inspection decisions as model training labels.

2. The intelligent inspection method according to claim 1, characterized in that: The multimodal data includes: LiDAR data, GNSS data and image data; The step of preprocessing the multimodal data to obtain preprocessed multimodal data includes: Filtering the LiDAR data by a statistical filtering method to obtain filtered LiDAR data; Correcting the GNSS data by a differential GNSS method to obtain corrected GNSS data; The image data is enhanced and denoised in sequence to obtain corrected image data.

3. The intelligent inspection method according to claim 2, characterized in that: The step of extracting each data feature from the preprocessed multimodal data comprises: Extracting shape features and distance features from the filtered LiDAR data; Extracting position features, velocity features and direction features from the corrected GNSS data; The color features, edge contour features and texture features in the corrected image data are extracted.

4. The intelligent inspection method according to claim 3, characterized in that: The step of extracting shape features and distance features from the filtered LiDAR data comprises: Extracting point cloud geometric features from the filtered LiDAR data, and calculating the Euclidean distance between points in the filtered LiDAR data; A distance feature is obtained according to the Euclidean distance, and a shape feature is determined according to the point cloud geometric feature and the Euclidean distance.

5. The intelligent inspection method according to claim 3, characterized in that: The step of extracting the position feature, speed feature and direction feature from the corrected GNSS data comprises: Converting the corrected GNSS data into plane coordinates and extracting position features in the plane coordinates; determining a velocity characteristic based on a rate of change of position information in the corrected GNSS data; A directional feature is determined based on the direction of a line between consecutive position points in the corrected GNSS data.

6. The intelligent inspection method according to claim 3, characterized in that: The step of extracting color features, edge contour features and texture features from the corrected image data comprises: Extracting color features in the corrected image data by a color histogram method; Extracting edge contour features in the corrected image data by using a Canny edge detection algorithm; The texture features in the corrected image data are extracted through local binary patterns.

7. The intelligent inspection method according to claim 1, characterized in that: The step of fusing the data features to obtain comprehensive data features includes: The data features are fused through the Kalman filter algorithm to obtain comprehensive data features.

8. An intelligent inspection system, characterized in that: The intelligent inspection system is applied to an inspection robot, and the intelligent inspection system comprises: A preprocessing module, used to acquire multimodal data and preprocess the multimodal data to obtain preprocessed multimodal data; A feature extraction module, used to extract each data feature from the preprocessed multimodal data, and fuse each data feature to obtain a comprehensive data feature; The decision module is used to input the comprehensive data features into a preset big model to obtain an inspection decision and operate according to the inspection decision, wherein the big model is trained with the preset actual comprehensive data features as model input data and the actual comprehensive data features corresponding to the preset actual inspection decisions as model training labels.

9. A patrol robot, characterized in that: The inspection robot comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the intelligent inspection method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the intelligent inspection method according to any one of claims 1 to 7 are implemented.

Images