Intelligent Inspection Method, Control Device, System and Storage Medium for Wheeled Robot

Abstract

The present application relates to vision detection technology, and discloses an intelligent inspection method, a control device, a system and a storage medium for a wheeled robot, including: performing image stitching on the monitoring images collected by each monitoring device to generate an overall monitoring image; analyzing the overall monitoring image to extract a workshop map of the production workshop; based on the workshop map, controlling the wheeled robot to perform a global scan of the production workshop to generate a 3D global image; analyzing the missing area part of the overall monitoring image compared with the 3D global image, and combining the latest workshop map to identify each monitoring dead angle area; based on the charging position of the wheeled robot and the positions of each monitoring dead angle area, planning an inspection path for the wheeled robot; and controlling the wheeled robot to perform intelligent inspection on the production workshop based on the inspection path. The present application aims to achieve efficient, intelligent and comprehensive monitoring of the production workshop.

Description

Technical Field

[0001] The present application relates to the technical field of visual detection, and particularly relates to an intelligent patrol inspection method, a control device, a control system and a computer-readable storage medium for a wheeled robot. Background Art

[0002] In modern industrial production, the efficient, stable and safe operation of the production workshop is the core goal pursued by enterprises. The production workshop is densely equipped with equipment, has complex technological processes, and various potential risk factors such as equipment failures, illegal operations, and safety hazards constantly threaten the normal progress of production. Conducting comprehensive, real-time and accurate monitoring of the production workshop can promptly detect and handle these problems, avoid the occurrence of production accidents, reduce equipment damage and downtime, improve product quality, and reduce production costs.

[0003] Currently, in order to achieve the monitoring of the production workshop, enterprises usually install multiple monitoring devices such as cameras and sensors in the workshop. These monitoring devices can collect data such as images, temperature, humidity, and pressure in the workshop, providing a certain basis for the management and maintenance of the workshop.

[0004] However, due to the extremely complex environment of the production workshop, there are many large-scale equipment, cargo stacks, and narrow passages. The existing monitoring devices are usually installed in fixed positions, and their monitoring vision ranges are limited, making it difficult to cover every corner of the workshop (for example, behind some large-scale mechanical equipment, narrow gaps between equipment, and the bottom of high-level shelves and other areas often become blind spots for monitoring. These monitoring dead corners may hide safety hazards such as equipment failures, material leaks, and illegal operations by personnel, and cannot be discovered and processed in a timely manner, posing great risks to production safety), and cannot meet the requirements of modern industrial production for efficient, intelligent, and comprehensive monitoring.

[0005] The above content is only used to assist in understanding the technical solution of the present application, and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main purpose of the present application is to provide an intelligent patrol inspection method, a control device, a control system and a computer-readable storage medium for a wheeled robot, aiming to achieve efficient, intelligent and comprehensive monitoring of the production workshop.

[0007] To achieve the above purpose, the present application provides an intelligent patrol inspection method for a wheeled robot, including the following steps:

[0008] Based on monitoring devices distributed at multiple different positions in the production workshop, collect monitoring images of the production workshop, and perform image stitching on the monitoring images collected by each monitoring device to generate an overall monitoring image;

[0009] Analyze the overall monitoring image, extract the workshop map of the production workshop, and mark the obstacles and passages in the workshop map;

[0010] Based on the workshop map, control the wheeled robot to conduct a global scan of the production workshop to generate a 3D global image; wherein, during the global scan, the wheeled robot supplements and updates the workshop map according to the detection data collected by the detection sensors integrated in the body; the detection sensors at least include an image sensor, a radar sensor, and a collision detection sensor;

[0011] Analyze the missing area part of the overall monitoring image compared with the 3D global image, and combine the latest workshop map to identify the missing area part to obtain each monitoring blind spot area in the production workshop;

[0012] Based on the charging position of the wheeled robot and the locations of each monitoring blind spot area, use a multi-objective optimization algorithm to plan the patrol path of the wheeled robot; wherein, the optimization objectives planned by the multi-objective optimization algorithm include the shortest single-round patrol time and the minimum single-round patrol energy consumption;

[0013] Based on the patrol path, control the wheeled robot to conduct an intelligent patrol of the production workshop; wherein, based on the detection data collected in real time by the wheeled robot during the patrol, control the wheeled robot to conduct intelligent obstacle avoidance.

[0014] To achieve the above object, the present application also provides a control device, including:

[0015] A monitoring module, configured to collect the monitoring images of the production workshop based on the monitoring devices distributed at multiple different positions in the production workshop, and perform image stitching on the monitoring images collected by each monitoring device to generate an overall monitoring image;

[0016] An extraction module, configured to analyze the overall monitoring image, extract the workshop map of the production workshop, and mark the obstacles and passages in the workshop map;

[0017] A scanning module, configured to control the wheeled robot to conduct a global scan of the production workshop based on the workshop map to generate a 3D global image; wherein, during the global scan, the wheeled robot supplements and updates the workshop map according to the detection data collected by the detection sensors integrated in the body; the detection sensors at least include an image sensor, a radar sensor, and a collision detection sensor;

[0018] An analysis module, configured to analyze the missing area part of the overall monitoring image compared with the 3D global image, and combine the latest workshop map to identify the missing area part to obtain each monitoring blind spot area in the production workshop;

[0019] A planning module, configured to plan a patrol path of the wheeled robot based on the charging position of the wheeled robot and the positions of various blind spots for monitoring, by using a multi-objective optimization algorithm; wherein, the optimization objectives of the multi-objective optimization algorithm include the shortest single-round patrol time and the minimum single-round patrol energy consumption.

[0020] A patrol module, configured to control the wheeled robot to perform intelligent patrol on the production workshop based on the patrol path; wherein, based on the detection data collected in real time during the patrol process of the wheeled robot, the wheeled robot is controlled to perform intelligent obstacle avoidance.

[0021] To achieve the above object, the present application further provides a control system, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the computer program is executed by the processor, the steps of the intelligent patrol method of the wheeled robot as described above are implemented.

[0022] To achieve the above object, the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the intelligent patrol method of the wheeled robot as described above are implemented.

[0023] The intelligent patrol method, control device, control system, and computer-readable storage medium of the wheeled robot provided by the present application are based on collecting and stitching images by multi-position monitoring devices to generate an overall monitoring image, laying a foundation for subsequent analysis, and analyzing the image to extract the workshop map and mark obstacles and passages, providing guidance for the robot's movement. The wheeled robot performs a global scan to generate a 3D global image, uses a variety of sensors to update the map in real time, and accurately finds the blind spots for monitoring. Then, a multi-objective optimization algorithm is used to plan the patrol path, taking into account both the patrol time and energy consumption, improving the patrol efficiency and reducing the cost. In this way, by using the wheeled robot in combination with multi-position monitoring devices, efficient, intelligent, and comprehensive monitoring of the production workshop is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic diagram of the steps of the intelligent patrol method of the wheeled robot in an embodiment of the present application;

[0025] Figure 2 It is a schematic diagram of the control device in an embodiment of the present application;

[0026] Figure 3 It is a schematic diagram of the internal architecture of the control system in an embodiment of the present application.

[0027] The realization, functional features, and advantages of the object of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0028] Embodiments of the present application will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are intended to explain the present application, and should not be construed as limiting the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts belong to the scope of protection of the present application.

[0029] In addition, if the description in the present application involves "first", "second", etc., it is only for descriptive purposes (such as for distinguishing the same or similar features), and should not be construed as indicating or implying their relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments may be combined with each other, but it must be based on the ability of those of ordinary skill in the art to implement. When the combination of technical solutions results in contradictions or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present application.

[0030] Referring to Figure 1 , in one embodiment, the intelligent inspection method of the wheeled robot includes:

[0031] Step S10: Based on the monitoring devices distributed at multiple different positions in the production workshop, collect the monitoring images of the production workshop, and perform image stitching on the monitoring images collected by each monitoring device to generate an overall monitoring image;

[0032] Step S20: Analyze the overall monitoring image, extract the workshop map of the production workshop, and mark the obstacles and passages in the workshop map;

[0033] Step S30: Based on the workshop map, control the wheeled robot to perform a global scan of the production workshop to generate a 3D global image; wherein, during the global scan, the wheeled robot supplements and updates the workshop map according to the detection data collected by the detection sensors integrated in the body; the detection sensors at least include an image sensor, a radar sensor, and a collision detection sensor;

[0034] Step S40: Analyze the missing area part of the overall monitoring image compared with the 3D global image, and combine the latest workshop map to identify the missing area part to obtain each monitoring dead corner area in the production workshop;

[0035] Step S50: Based on the charging position of the wheeled robot and the locations of each monitoring blind area, a multi-objective optimization algorithm is used to plan the inspection path of the wheeled robot. Among them, the optimization objectives of the multi-objective optimization algorithm include the shortest inspection time for a single round and the minimum energy consumption for a single round of inspection.

[0036] Step S60: Based on the inspection path, control the wheeled robot to perform intelligent inspection on the production workshop. Among them, based on the detection data collected in real time during the inspection process of the wheeled robot, control the wheeled robot to perform intelligent obstacle avoidance.

[0037] In this embodiment, the execution terminal of the embodiment can be a control system, or other devices or apparatuses (such as a control device) that control the wheeled robot.

[0038] As described in step S10, in order to comprehensively cover the production workshop, the monitoring devices will be reasonably distributed according to factors such as the layout of the workshop, the distribution of equipment, and the technological process. For example, in a large production workshop, monitoring devices are usually installed at the four corners of the workshop, on both sides of the passageways, above key equipment, and in areas prone to danger to ensure that there are no large areas of monitoring blind spots in the workshop.

[0039] Different types of monitoring devices can be selected, such as high-definition cameras, infrared cameras, etc. High-definition cameras can provide clear image details and are suitable for areas with sufficient light; infrared cameras, on the other hand, play a role at night or in dimly lit environments and can be used to monitor the operation of equipment and personnel activities in the workshop.

[0040] Optionally, the monitoring devices will collect images at a set frequency. For some areas with high real-time requirements, such as key processes on the production line, the collection frequency may be set relatively high, for example, collecting 1 - 5 frames per second; while for some relatively stable areas, the collection frequency can be appropriately reduced, such as collecting 1 - 2 frames per minute.

[0041] Optionally, before image stitching, the collected monitoring images are preprocessed first. This includes operations such as removing noise from the images, adjusting the brightness and contrast of the images, etc., to improve the quality of the images and lay a good foundation for subsequent stitching work.

[0042] Optionally, a feature extraction algorithm is used to extract feature points from the preprocessed images. These feature points will be used in subsequent image matching and stitching processes. By comparing the feature points in different monitoring images, the corresponding relationships between them are found. Among them, a matching algorithm based on feature descriptors can be used to quickly and accurately find matching feature point pairs.

[0043] Optionally, based on the matched feature point pairs, a transformation matrix between adjacent images is calculated. This transformation matrix describes how to transform one image into the coordinate system of another image so that they can be accurately stitched together.

[0044] Apply the calculated transformation matrix to adjacent images, perform geometric transformation on the images, and then fuse them. During the fusion process, the overlapping areas between images need to be processed to avoid stitching traces. Methods such as weighted averaging and multi-resolution fusion can be used to achieve smooth image fusion.

[0045] After image stitching and fusion, the individual monitoring images are combined into a complete overall monitoring image. This image can intuitively show the overall picture of the production workshop and provide an important basis for subsequent workshop map extraction and analysis.

[0046] As described in step S20, before conducting an in-depth analysis of the overall surveillance image, a preprocessing operation may be performed to improve the image quality and lay a foundation for subsequent extraction of accurate information.

[0047] Optionally, the color overall monitoring image is converted into a grayscale image. This can reduce the amount of data and the computational complexity while retaining the main feature information of the image for easy subsequent processing.

[0048] Optionally, use Gaussian filtering, median filtering and other methods to remove noise interference in the image. The noise may come from the monitoring device itself, the transmission process, etc. The filtering operation can make the image smoother and enhance the clarity of the image.

[0049] Optionally, edge detection algorithms can be used to highlight the edge information of objects in the image. This helps to more clearly identify the boundaries of different objects in the workshop and provide more accurate data for subsequent feature extraction.

[0050] Optionally, extract key features from the preprocessed images and identify different elements in the workshop. Computer vision techniques, such as feature point-based methods (SIFT, SURF, etc.) or deep learning-based object detection algorithms (such as YOLO, Faster R-CNN, etc.), are used to extract features of various objects in the workshop (such as equipment, shelves, walls, etc.). These features can include the shape, size, color, texture, etc. of the object.

[0051] Based on the extracted features, objects in the workshop are classified and identified. The pre-trained classification model is used to determine the type of each object, such as production equipment, storage shelves, channel signs, etc.

[0052] Establish a suitable coordinate system for the workshop map. A fixed point in the workshop (such as a corner) can be used as the origin to determine the directions and unit lengths of the X and Y axes. This can convert the positions of objects in the image into actual coordinate positions, facilitating subsequent navigation and analysis.

[0053] Based on the coordinate positions and classification information of the objects, draw the corresponding elements on the map. For example, use different shapes and colors to represent different types of equipment, shelves, walls, etc., to form an intuitive workshop map.

[0054] Optionally, mark the identified impassable objects (such as large equipment, cargo piles, etc.) as obstacles. Specific colors (such as red) or symbols (such as crosses) can be used on the map to highlight these obstacles so that the wheeled robot can avoid them during the inspection process.

[0055] Optionally, determine the positions and scopes of the passages according to the actual traffic conditions in the workshop. Use another color (such as green) or lines (such as dotted lines) to mark the passages on the map, providing clear guidance for planning the inspection path of the wheeled robot.

[0056] As described in step S30, import the workshop map extracted and marked in step S20 into the control system of the wheeled robot, enabling the robot to understand the distribution of known obstacles and passages in the workshop.

[0057] Establish the correspondence between the robot's position and the map coordinates, and determine the robot's position in the map in real time through a positioning system (such as visual positioning, inertial navigation, etc.).

[0058] Based on this map, preliminarily plan the path for the robot to conduct a global scan to ensure that all areas of the production workshop can be covered. Path planning takes into account factors such as the layout of the workshop, the distribution of obstacles, and the scanning efficiency, and tries to avoid repeated scans and unnecessary detours.

[0059] Optionally, use the landmark features in the workshop map, such as specific equipment, corners, etc., combined with the robot's own positioning system (such as visual positioning, inertial navigation, etc.), to determine the initial position and orientation of the robot in the workshop map. This is the basis for subsequent accurate scanning and map updating.

[0060] During the global scan, use an image sensor (such as a high-definition camera) to continuously capture the environmental images around the robot. These images contain information such as the appearance, color, and texture of the objects in the workshop. The camera usually takes pictures from multiple angles and directions to obtain more comprehensive visual data. For example, during the movement of the robot, the camera can rotate horizontally and vertically to ensure that the surrounding 360° environment can be photographed.

[0061] Optionally, a radar sensor (such as a lidar) can be used to emit a laser beam and measure the time of reflected light to calculate the distance between the robot and surrounding objects. By continuously rotating the radar antenna, distance information in all directions around the robot can be obtained in real time to construct three-dimensional point cloud data of the surrounding environment.

[0062] The image data collected by the image sensor and the point cloud data obtained by the radar sensor are fused. Using the data fusion algorithm, the two different types of data are matched and aligned in time and space, so that the visual information in the image is combined with the distance information in the point cloud data. For example, feature matching methods can be used to associate feature points in the image with corresponding points in the point cloud data.

[0063] Based on the fused data, a 3D global image of the production workshop is constructed using a 3D reconstruction algorithm. The image presents the overall picture of the workshop in the form of a 3D model, including the true shape and positional relationship of various objects such as equipment, shelves, and walls. By integrating and processing data collected at different positions and angles, the accuracy and details of the 3D model are continuously optimized.

[0064] At the same time, as the robot moves, the collision detection sensor continuously detects the surrounding environment. When the sensor detects that there are no objects in the areas marked as obstacles in the original workshop map, or detects that there are passable spaces in areas that were originally considered impassable, these areas are marked as abnormal areas.

[0065] The abnormal area is then further detected and confirmed by collecting sensor data multiple times to determine whether the area is continuously passable, eliminating misjudgments caused by sensor errors or temporary object movement.

[0066] Based on the detection results of the abnormal area, the possible existence of previously unrepresented passages is identified. The spatial size, connectivity and other factors of the area are analyzed to determine whether it meets the characteristics of a passage. The identified passage can be verified by having the robot try to pass through the area while observing the sensor data and the movement of the robot. If the robot can pass through the area smoothly and the sensor does not detect the risk of collision, the area is confirmed to be a previously unrepresented passage and updated in the workshop map.

[0067] According to the results of the collision detection, the original workshop map is updated. If the obstacles marked in the original map are found to be non-existent, they are removed from the map; if a channel that was not originally represented is identified, the corresponding channel information is added to the map, and the navigation data of the map is updated to provide accurate map support for subsequent inspections.

[0068] When the robot completes the scanning of the entire production workshop according to the planned path and collects sufficient data for generating an accurate 3D global image and updating the workshop map, it is determined that the scanning process ends.

[0069] As described in step S40, unify the coordinate systems of the 3D global image, the overall monitoring image, and the latest workshop map. Map them all to the same coordinate reference system. For example, taking the coordinate system established by the workshop map as the benchmark, ensure that the position information of different data sources can accurately correspond during the comparison and analysis process.

[0070] Extract feature points from the 3D global image and the overall monitoring image respectively. These features have rotation, scale, and illumination invariance and can stably represent the features of objects in different images. Use the feature matching algorithm to match the feature points of the 3D global image and the overall monitoring image to find their corresponding relationships.

[0071] According to the feature matching results, adopt a suitable transformation model (such as affine transformation, perspective transformation) to register the overall monitoring image so that it is spatially aligned with the 3D global image. The registered image is convenient for pixel-by-pixel or regional comparison and analysis.

[0072] Based on the image comparison results and the correlation analysis with the workshop map, determine the blind spots in the production workshop. These areas have obvious information in the 3D global image but are missing or incomplete in the overall monitoring image.

[0073] Optionally, perform a pixel-by-pixel comparison of the registered 3D global image and the overall monitoring image to calculate the difference in pixel values. If there is pixel information in a certain area in the 3D global image while the pixel value in the corresponding area in the overall monitoring image is zero or close to zero, then mark this area as a possible missing area.

[0074] Optionally, divide the image into different regions and compare the features and statistical information (such as color, texture, shape, etc.) of each region. If a certain region has obvious features in the 3D global image but the features are not obvious or missing in the overall monitoring image, then regard this region as a possible missing area.

[0075] Map the position information of the detected missing areas to the latest workshop map to determine the specific positions of these areas in the actual space of the workshop.

[0076] Optionally, check the distribution of obstacles around the missing areas in the workshop map. If the missing areas are blocked by large equipment, shelves, walls, etc., resulting in the monitoring cameras being unable to cover these areas, then these areas are very likely to be blind spots.

[0077] Optionally, consider the relationship between the missing area and the workshop passageway, as well as the accessibility of personnel and equipment. If the missing area is near an important passageway or in an area where personnel and equipment are frequently active, but the monitoring cannot cover it, then this area should be focused on as a blind spot for monitoring.

[0078] Optionally, combined with the functional zoning of the workshop, judge the importance of the functional area where the missing area is located. For example, if the missing area is located in an area involving key production processes, storage of dangerous goods, or a crowded area of personnel, then this area is more likely to be a blind spot for monitoring that needs to be solved.

[0079] Based on the above screening results of the missing areas, comprehensively consider factors such as obstacle occlusion, passageway accessibility, and importance of functional areas to determine the blind spot areas in the production workshop, including determining the central point coordinates or boundary coordinates of each blind spot area.

[0080] Optionally, make an accurate boundary division for each blind spot area to clarify its scope. The boundary of the blind spot area can be represented by drawing a polygon or other geometric figures on the workshop map.

[0081] As described in step S50, clarify the exact coordinates of the charging position of the wheeled robot in the workshop map. The charging position is usually fixed and has a specific identifier in the workshop layout. And collect the position information of each blind spot area in the workshop map, which has been determined in step S40. Integrate the charging position and the position information of the blind spot areas into a data structure for convenient processing by subsequent algorithms.

[0082] Optionally, consider the distribution of obstacles in the workshop map. Obstacle information can be obtained from the latest workshop map, including non-passable areas such as equipment and shelves.

[0083] Optionally, analyze the passageway conditions in the workshop to understand factors such as the width and connectivity of the passageways, which will affect the moving speed and path selection of the robot.

[0084] Select a suitable multi-objective optimization algorithm, such as genetic algorithm, particle swarm algorithm, etc. These algorithms can find a balanced solution among multiple optimization objectives. Taking the genetic algorithm as an example, it continuously iteratively searches for the optimal solution by simulating the biological evolution process.

[0085] Objective optimization for the shortest single-round inspection time: Define a time evaluation function, considering factors such as the moving speed of the robot on different sections and the turning time. The moving speed of the robot in the passageway may be affected by the width of the passageway, road conditions, etc., and the turning time is related to the turning angle and the steering performance of the robot. For example, let the inspection path be P={p1,p2,⋯,p n}, where pi represents the i-th point on the path, t ij represents moving from point p i to point p j The time taken, then the single-round inspection time .

[0086] Objective optimization for the shortest single-round inspection time: Define an energy consumption evaluation function, considering the moving energy consumption, turning energy consumption, etc. of the robot. The moving energy consumption of the robot is related to the moving distance, load weight, etc., and the turning energy consumption is related to the turning angle and the power system of the robot. For example, let e ij represent the energy consumption from point p i to point p j The energy consumption, then the single-round inspection energy consumption .

[0087] Constraint setting: The path must avoid obstacles in the workshop map to ensure the safe movement of the robot. This constraint can be achieved by setting that the path points cannot fall in the area where the obstacles are located. The moving trajectory of the robot must conform to the channel layout of the workshop and cannot cross the impassable area.

[0088] During the path planning process, based on the genetic algorithm, a group of initial inspection paths are randomly generated as the population. Each path is represented as an access order of the blind spot areas, and the starting point and the ending point of the path are both the charging positions. Conduct a feasibility check on each initial path to ensure that it meets the constraints. If there is a situation where the path crosses an obstacle or an impassable area, adjust or regenerate the path.

[0089] For each path in the population, calculate its single-round inspection time T and single-round inspection energy consumption E. According to the objective function, combine these two objective values into a fitness value. The weighted summation method can be used, setting F = w1T + w2E, where w1 and w2 are weight coefficients, and w1 + w2 = 1. The weights can be adjusted according to actual needs to balance the importance of time and energy consumption. Optionally, set w1 to be greater than w2.

[0090] Select the paths in the population according to the fitness value F, and select the paths with higher fitness (such as the fitness is greater than the preset threshold) as the parents for generating the next generation of paths. Among them, the preset threshold can be the average fitness of the current round, that is, select the paths with fitness above the average as the parents.

[0091] Perform crossover operation on the selected parent paths to generate new offspring paths. The crossover operation simulates the gene exchange process in biological inheritance, exchanges some segments of the parent paths, and generates offspring paths with different access orders. Perform mutation operation on the offspring paths, randomly change some access orders in the paths, increase the diversity of the population, and avoid the algorithm falling into a local optimal solution.

[0092] Repeat the fitness evaluation, selection, crossover, and mutation operations until the termination condition is met. The termination condition can be that the fitness value no longer improves significantly, that is, the average fitness difference between two consecutive rounds is less than the preset difference.

[0093] After the iteration ends, select the path with the optimal fitness value from the final population as the inspection path of the wheeled robot. This path achieves a good balance between the single-round inspection time and the single-round inspection energy consumption.

[0094] Output the planned inspection path to the control system of the wheeled robot. The robot conducts inspections according to this path, and at the same time, monitors its own power and position in real time during the inspection to ensure that the inspection task can be completed safely and efficiently.

[0095] As described in step S60, the wheeled robot reads the data of the planned inspection path, which contains a series of coordinate points or path nodes, and clarifies the movement trajectory of the robot in the production workshop.

[0096] When the inspection starts, activate various sensors and actuators of the wheeled robot, including lidar, cameras, motors, etc., for self-inspection to ensure the normal operation of the equipment. Position the robot at the charging position (i.e., the starting point of the inspection path), obtain the initial position information of the robot through the positioning system, and match and calibrate it with the starting point of the inspection path.

[0097] The motion control module of the robot calculates the direction and distance from the current position to the next path node according to the inspection path. Then, control the rotation of the robot's wheels through the motor drive system to make the robot move towards the target node. During the movement, continuously adjust the posture and direction of the robot to ensure that it moves along the planned path.

[0098] Use various sensors installed on the robot to collect detection data of the production workshop in real time. These sensors include but are not limited to:

[0099] Lidar: Used to obtain the three-dimensional point cloud data of the environment around the robot in real time. By analyzing the point cloud data, the position, shape, and distance of obstacles can be detected.

[0100] Camera: Collect images and video information of the workshop for identifying equipment status, personnel activities, etc.

[0101] Temperature sensor: Monitor the temperature changes at different positions in the workshop and promptly detect potential equipment overheating problems.

[0102] Gas sensor: Detect the concentration of harmful gases in the workshop to ensure the safety of the production environment.

[0103] It should be noted that although obstacles have been attempted to be avoided during the path planning stage, obstacle avoidance operations are still required after the inspection starts because the production workshop environment may change during the period from path planning completion to inspection start, resulting in new obstacles. For example, workers temporarily move large equipment and place it on the originally planned path, or new goods are stacked in the passageway. These newly added obstacles were not considered during path planning.

[0104] Moreover, there are a large number of moving objects in the production workshop, such as other robots, forklifts, and staff. Path planning is usually based on static environment information and cannot accurately predict the real-time positions and movement trajectories of these moving objects. When the inspection robot is performing tasks, moving objects may enter its planned path, thus requiring real-time obstacle avoidance.

[0105] Or, due to certain special reasons, such as equipment maintenance, emergency construction, etc., the layout of some areas in the workshop will be temporarily adjusted, and the originally unobstructed path may be temporarily blocked or temporary obstacles may be set up. This requires the inspection robot to have the ability of real-time obstacle avoidance during the inspection process.

[0106] Optionally, perform preprocessing such as filtering and segmentation on the point cloud data collected by the lidar to remove noise and irrelevant information. Then, divide the point cloud data into different object clusters through a clustering algorithm to identify obstacles.

[0107] Optionally, combine the image information collected by the camera and use target detection and recognition algorithms to further confirm the type and position of the obstacles. For example, distinguish different types of obstacles such as equipment, shelves, or personnel.

[0108] Optionally, when a sudden obstacle appears ahead, evaluate the threat level of the obstacle to the robot's inspection task based on factors such as the distance between the obstacle and the robot and the relative speed. For example, if the obstacle is close to the robot and approaching rapidly, the threat level is high; if the obstacle is stationary and far away, the threat level is low.

[0109] Optionally, select a suitable obstacle avoidance strategy according to the threat level of the obstacle and the characteristics of the workshop environment. Common obstacle avoidance strategies include:

[0110] Detour strategy: When the obstacle is small in size and there is enough space around it, the robot can bypass the obstacle and continue to move forward along the inspection path. By re-planning the local path, the area where the obstacle is located is avoided;

[0111] Waiting strategy: If the obstacle is moving (such as a person or other equipment), and it is expected that the obstacle will leave the robot's driving path within a short time, then the robot can pause moving and wait for the obstacle to leave before continuing the inspection.

[0112] According to the selected obstacle avoidance strategy, the motion control module of the robot adjusts the motion parameters of the robot, such as speed, direction, etc., to achieve the obstacle avoidance action. And during the obstacle avoidance process, continuously monitor the state of the obstacle and the changes in the surrounding environment, and adjust the obstacle avoidance strategy in a timely manner to ensure the safety of the robot.

[0113] Optionally, the running state of the wheeled robot is monitored in real time, including parameters such as battery power, speed, and attitude. When the battery power is lower than the set threshold, the robot automatically adjusts the inspection task and gives priority to returning to the charging position for charging.

[0114] The detection data and running state information collected by the robot during the inspection process are uploaded to the host computer or the workshop management system in real time. These data can be used for subsequent analysis and processing, such as equipment status evaluation, production environment monitoring, etc. At the same time, receive the instructions sent by the host computer and adjust the inspection task or perform other operations according to the instructions.

[0115] When the robot reaches the end point of the inspection path (i.e., the charging position), it judges whether the inspection task is completed. If all path nodes have been visited and there are no unprocessed abnormal situations, it is considered that the inspection task is over.

[0116] All the data collected during this inspection process are stored, including sensor data, operation logs, etc., for subsequent query and analysis. Generate an inspection report to summarize the results of this inspection, including information such as problems found and obstacle situations, providing a reference for the management and maintenance of the workshop.

[0117] In one embodiment, based on multi-position monitoring devices, images are collected and stitched to generate an overall monitoring image, laying a foundation for subsequent analysis, and analyzing the image to extract the workshop map and mark obstacles and passages to provide guidance for the robot's movement. The wheeled robot performs a global scan to generate a 3D global image, uses a variety of sensors to update the map in real time, and accurately finds the monitoring blind spots. Then, a multi-objective optimization algorithm is used to plan the inspection path, taking into account the inspection time and energy consumption, improving the inspection efficiency and reducing costs. In this way, by using the wheeled robot in combination with multi-position monitoring devices, efficient, intelligent, and comprehensive monitoring of the production workshop is achieved.

[0118] During the patrol inspection, the robot can avoid obstacles intelligently relying on the real-time collected data to ensure its own safety. In addition, by integrating diverse data, comprehensive monitoring is achieved, potential risks can be discovered in a timely manner, the labor cost is reduced, and the errors of manual inspection are avoided. Moreover, through in-depth data analysis, strong support is provided for production management decision-making, helping the production workshop to operate efficiently, stably and safely.

[0119] In one embodiment, based on the above embodiment, after the step of analyzing the missing area part of the overall monitoring image compared with the 3D global image and combining the latest workshop map to identify the missing area part to obtain each monitoring blind area in the production workshop, it further includes:

[0120] Perform feature recognition on the 3D global image, and use the feature recognition results to generate semantic information of the equipment and objects in each monitoring blind area;

[0121] Among them, when planning the patrol path, based on the semantic information of the equipment and objects in the monitoring blind area, determine the corresponding patrol priority.

[0122] In this embodiment, computer vision technology and deep learning algorithms are used to extract features from the 3D global image. For example, for the equipment and objects in the image, visual features such as their shape, color, texture, and edge can be extracted. If it is a mechanical part, its specific shape contour and surface texture are important features; for equipment with color markings, the color can also provide key information.

[0123] 3D information can also be combined to extract features such as the three-dimensional size, spatial position, and posture of the object. For example, by analyzing the coordinates and angles of the object in 3D space, determine its placement state.

[0124] Match the extracted features with a pre-constructed feature library. The feature library contains standard feature templates of various common equipment and objects in the production workshop. By calculating the similarity between features, determine which category the object in the image belongs to.

[0125] Adopt classification algorithms such as support vector machines and convolutional neural networks to accurately classify the objects. For example, classify the equipment into categories such as production equipment, transportation equipment, and storage equipment, and classify the objects into categories such as raw materials, finished products, and waste materials.

[0126] According to the results of feature recognition, associate the classification information of each equipment and object with their specific positions in the monitoring blind area. For example, determine the specific coordinate position of a certain production equipment in the monitoring blind area.

[0127] Optionally, additional attribute information can be added to each device and object in combination with the business information of the workshop. For example, for production equipment, information such as its production process, operating parameters, and maintenance records can be added; for raw materials and finished products, information such as their names, specifications, and uses can be added.

[0128] Using the associated information, generate a detailed semantic description for each device and object in the blind spot area. For example, "An injection molding machine located at a certain coordinate in the blind spot area, and the current product being produced is a plastic shell."

[0129] Based on the semantic information of the devices and objects, evaluate their importance in the production workshop. For key production equipment that directly affects the production process, a higher priority is given. For example, a large machining center that undertakes the core production process, its failure may cause the entire production line to stop, so it should be inspected first.

[0130] For areas storing important raw materials or finished products, a higher priority should also be given. For example, the safety and inventory situation of a warehouse storing high-value raw materials need to be focused on.

[0131] Optionally, analyze the possible risks of the devices and objects. For devices that are prone to failure and have a relatively high degree of aging, increase their inspection priority. For example, a motor with a relatively long service life has a relatively high probability of failure and needs to be inspected more frequently.

[0132] Consider the impact of environmental factors on the devices and objects. If there are harsh environmental conditions such as high temperature and humidity in a certain blind spot area, the devices and objects in this area may be more likely to be damaged, and the inspection priority should be increased.

[0133] Taking into account the results of importance and risk assessment comprehensively, determine a comprehensive inspection priority for each blind spot area. The method of weighted summation can be used to calculate the weighted scores of the importance and risk assessment results to obtain the final priority score.

[0134] Sort the blind spot areas according to the priority scores so that the inspection order can be reasonably arranged when planning the inspection path.

[0135] Based on the original multi-objective optimization algorithm, incorporate the inspection priority as an important constraint condition. The optimization objectives still include the shortest inspection time in a single round and the minimum energy consumption in a single round, but at the same time, it is necessary to ensure that the high-priority blind spot areas can be inspected first.

[0136] During the search process of the algorithm, adjust the path selection strategy according to the priority of the blind spot areas. For example, preferentially select the path branches that can reach the high-priority areas as soon as possible.

[0137] Using the adjusted multi-objective optimization algorithm, combined with the charging position of the wheeled robot and the location of each monitoring blind spot area, an inspection path that meets the priority requirements is planned.

[0138] Evaluate and optimize the planned path to check whether there are any unreasonable places, such as too long path, frequent return, etc. If problems are found, adjust the path until a path that meets the inspection priority requirements and optimizes the inspection time and energy consumption is obtained.

[0139] In one embodiment, the information in the 3D global image is fully utilized to provide more targeted guidance for the inspection path planning of the wheeled robot, improve the inspection efficiency and quality, and promptly discover potential problems in the production workshop.

[0140] During the inspection process, by focusing on different equipment and areas according to priority, more targeted data can be collected. These data can reflect the actual operating status and potential problems of the equipment, and provide strong support for the formulation of reasonable maintenance plans. For example, by frequently inspecting high-priority equipment, you can understand the laws and trends of its failures, so as to formulate a more scientific preventive maintenance plan.

[0141] In one embodiment, based on the above embodiment, the step of controlling the wheeled robot to perform intelligent obstacle avoidance based on the detection data collected in real time during the inspection process of the wheeled robot includes:

[0142] Based on the detection data collected in real time by the wheeled robot during the inspection process, analyze whether there are obstacles ahead of the wheeled robot's forward path;

[0143] If so, estimate the duration of the obstacle based on visual detection technology;

[0144] The obstacle avoidance strategy is selected based on the duration of the obstacle; the obstacle avoidance strategies include detour obstacle avoidance and wait obstacle avoidance.

[0145] In this embodiment, the wheeled robot is equipped with a variety of detection sensors, such as radar sensors, ultrasonic sensors, image sensors, etc. During the inspection process, these sensors will work in real time to collect relevant data of the surrounding environment.

[0146] Among them, the radar sensor obtains the distance and position information of surrounding objects by emitting laser beams and measuring the time of reflected light. It can generate high-precision three-dimensional point cloud data and clearly depict the outline of the robot's surrounding environment; the ultrasonic sensor uses the propagation characteristics of ultrasonic waves in the air to measure the distance to the object in front. It has the characteristics of low cost and fast response speed, and is suitable for close-range obstacle detection; the image sensor can capture the visual image in front of the robot and provide data support for subsequent visual inspections.

[0147] The control system will process the detected data collected in real time. For example, for the point cloud data of the radar sensor, operations such as filtering and segmentation will be carried out to remove noise and irrelevant information, and extract the point cloud clusters that may represent obstacles; for the data of the ultrasonic sensor, the measured distance will be compared with the preset safe distance.

[0148] Based on the processed data, it is judged whether there are obstacles in front of the forward path of the wheeled robot. If the radar sensor detects a dense point cloud cluster within a certain range in front, or the distance measured by the ultrasonic sensor is less than the safe distance, or possible obstacles are identified in the image captured by the image sensor, it is determined that there are obstacles in front of the forward path.

[0149] When it is determined that there are obstacles in front of the forward path, the robot will mainly analyze the visual images collected by the image sensor. First, preprocess the image, including operations such as adjusting brightness, contrast, and removing noise, to improve the quality of the image and facilitate subsequent feature extraction and analysis.

[0150] Use computer vision algorithms, such as edge detection, corner detection, color feature extraction, etc., to extract the features of the obstacles from the preprocessed image. For example, for an obstacle in the shape of a cuboid, its edge and corner information can be extracted; for an obstacle with a specific color, its color feature can be extracted.

[0151] Match the extracted features with the pre-established obstacle feature library to identify the type of obstacle, such as personnel, vehicles, cargo piles, etc. At the same time, by analyzing consecutive multi-frame images, track the movement trajectory of the obstacle.

[0152] If the identified obstacle is a static object, such as a fixed cargo pile, the robot will estimate the time required to bypass the obstacle based on the current inspection task and path planning, and use this as the obstacle duration.

[0153] For dynamic obstacles, such as moving personnel or vehicles, the robot will predict the time required for them to leave the forward path based on their movement speed, direction, and current position. Algorithms such as Kalman filtering can be used to estimate and predict the movement state of the obstacle, so as to obtain a more accurate obstacle duration.

[0154] When the estimated duration of the obstacle is long and there is a detour path around the robot, the detour obstacle avoidance strategy is selected. For example, the obstacle ahead is a vehicle parked for a long time, and the robot can bypass the vehicle through the adjacent passage and continue the inspection. The control system will re-plan a path around the obstacle based on the workshop map and the current position information. When planning the path, factors such as the length of the path, the difficulty of passage, and whether other obstacles will be encountered are considered to select an optimal detour path. The wheeled robot moves along the re-planned path while continuing to collect detection data in real time to ensure that no new obstacles will be encountered during the detour.

[0155] When the estimated duration of the obstacle is short and the detour path is complex or infeasible, the waiting obstacle avoidance strategy is selected. For example, the obstacle ahead is a person passing by briefly, and the robot can continue to move forward after waiting for the person to pass. The robot will stop moving in place while maintaining monitoring of the obstacle. During the waiting process, the estimated duration of the obstacle will be continuously updated. If it is found that the estimated time has changed, whether to change the obstacle avoidance strategy will be re-evaluated according to the new situation. When the obstacle leaves the forward path, the robot will continue the inspection according to the originally planned inspection path.

[0156] During the inspection work, by estimating the duration of the obstacle, the wheeled robot can flexibly select the obstacle avoidance strategy according to the actual situation. When the duration of the obstacle is short, the waiting obstacle avoidance is selected, which avoids spending time planning and executing the detour path, enabling the robot to continue the inspection along the original path in a short time and reducing unnecessary time waste. On the contrary, if the duration of the obstacle is long, the detour obstacle avoidance is adopted, which can enable the robot to quickly bypass the obstacle and continue to advance the inspection task, avoiding the low efficiency caused by long-term waiting.

[0157] In one embodiment, based on the above embodiment, the step of selecting the obstacle avoidance strategy based on the duration of the obstacle includes:

[0158] Estimate the detour duration required for detour obstacle avoidance;

[0159] If the detour duration is less than the obstacle avoidance duration, select detour obstacle avoidance as the obstacle avoidance strategy; otherwise, select waiting obstacle avoidance as the obstacle avoidance strategy.

[0160] In this embodiment, the workshop map records information about each area in the environment, including passages, obstacle positions, etc. When the detour duration needs to be estimated, the robot will first obtain the current position and the passable path information around it from the map.

[0161] Optionally, the robot uses path planning algorithms such as the A* algorithm, Dijkstra algorithm, etc. to plan the best path around obstacles based on the map information. By calculating the total length of this detour path and combining with the robot's own moving speed, the time required to complete the detour is initially estimated.

[0162] Then, an obstacle avoidance strategy is selected according to the detour duration and the obstacle duration.

[0163] When the estimated detour duration is less than the obstacle duration, the detour obstacle avoidance is selected as the obstacle avoidance strategy. Because in this case, the time spent on detouring is shorter than the time waiting for the obstacle to be removed. For example, if the estimated obstacle duration is 15 seconds and the detour duration is only 10 seconds, choosing to detour at this time can resume normal inspection faster and improve the overall inspection efficiency.

[0164] Once it is determined to choose detour obstacle avoidance, the robot will immediately start moving along the previously planned detour path, while continuing to monitor the surrounding environment in real time to ensure that no new problems will be encountered during the detour.

[0165] If the detour duration is greater than or equal to the obstacle duration, the waiting obstacle avoidance is selected as the obstacle avoidance strategy. In this case, the time spent waiting for the obstacle to be removed by itself is shorter, avoiding wasting more time and energy due to detouring. For example, if the estimated obstacle duration is 8 seconds and the detour duration is 12 seconds, choosing to wait at this time can reduce unnecessary movement and path planning operations. During the waiting process, the robot will keep monitoring the obstacle and continuously update the estimated obstacle duration. If it is found during the waiting process that the estimated obstacle duration has changed and meets the conditions for detouring, the robot will re-evaluate and may switch to the detour obstacle avoidance strategy.

[0166] In one embodiment, when a wheeled robot encounters an obstacle, it can quickly evaluate the time required for detouring and waiting, so as to select the strategy with shorter time consumption, enabling the robot to complete the entire inspection task more quickly. This time comparison-based strategy can enable the robot to more reasonably plan the energy usage. During the inspection process, it avoids excessive energy waste caused by unreasonable obstacle avoidance decisions, ensuring that the robot has sufficient energy to complete the subsequent inspection tasks. Especially in some large and complex inspection environments, the reasonable utilization of energy is particularly important.

[0167] In one embodiment, based on the above embodiment, the intelligent inspection method of the wheeled robot further includes:

[0168] When the wheeled robot starts to perform intelligent inspection on the production workshop, the personnel density of each target area passed by the inspection path is obtained based on the monitoring equipment;

[0169] Before the wheeled robot arrives at the corresponding target area, the moving speed of the wheeled robot is pre-adjusted according to the degree of personnel density; among them, the higher the degree of personnel density, the lower the adjusted moving speed.

[0170] In this embodiment, when the wheeled robot starts the intelligent inspection of the production workshop, the monitoring equipment installed in the workshop is used to obtain the personnel density of each target area passed by the inspection path. The monitoring equipment can be a camera. Using computer vision technology, such as object detection and counting algorithms, the personnel in the monitoring picture are identified and counted, so as to obtain the number of personnel in each target area. According to factors such as the number of personnel and the area of the area, the personnel density is further calculated.

[0171] When the wheeled robot moves towards the corresponding target area, before arriving, it will pre-adjust its moving speed according to the obtained personnel density. The principle it follows is that the higher the personnel density, the lower the adjusted moving speed.

[0172] For example, when it is detected that the personnel density in a certain target area is low, the wheeled robot can move at a relatively fast speed, which can improve the inspection efficiency; while when it is detected that the personnel density in a certain target area is high, such as a large number of workers are carrying out centralized operations, in order to ensure the safety of personnel and avoid accidents such as collisions, the wheeled robot will lower its moving speed and slowly pass through this area.

[0173] In one embodiment, by pre-adjusting the speed, it can better adapt to environments with different personnel densities, reduce safety accidents caused by too fast speed, and create a safer working environment for the production workshop. The personnel distribution in different areas of the production workshop will change with time and production tasks. This way of adjusting the speed according to the personnel density enables the wheeled robot to flexibly adapt to various scenarios and improves its working ability in complex dynamic environments.

[0174] Moreover, it can reduce the interference of the robot to the normal work of personnel, enabling personnel to focus more on their production tasks. At the same time, it also makes it easier for personnel to accept the operation of the robot in the workshop, promoting friendly cooperation between humans and machines. On the premise of ensuring safety, reasonably adjusting the speed will neither lead to too low inspection efficiency due to too slow speed nor affect the work of personnel due to too fast speed, thus improving the work efficiency of the entire production workshop.

[0175] In one embodiment, on the basis of the above embodiment, after the step of analyzing the missing area part of the overall monitoring image compared with the 3D global image and combining the latest workshop map to identify the missing area part to obtain each monitoring blind area in the production workshop, it further includes:

[0176] Based on the layout and field of view of the monitoring devices in the production workshop, using computer simulation technology, virtually simulate the monitoring situation in the production workshop to simulate blind spots in monitoring under different circumstances;

[0177] Based on the simulation results, optimize the analysis results of the blind spot areas for monitoring.

[0178] In this embodiment, based on the known layout of the monitoring devices in the production workshop (i.e., the specific installation locations of each monitoring device) and their fields of view (the spatial areas that each monitoring device can cover), computer simulation technology is used to construct a virtual model of the production workshop.

[0179] In this virtual model, simulate various different scenarios, such as the flow of personnel and changes in the operating states of equipment at different time periods. Because in actual production, these factors may all affect the monitoring effect of the monitoring devices, causing areas that were not originally blind spots for monitoring to become blind spots under certain circumstances. By simulating different situations, potential blind spots for monitoring can be discovered more comprehensively.

[0180] Compare the analysis results of the blind spot areas for monitoring obtained previously by analyzing the overall monitoring images and 3D global images with the blind spots in monitoring under different circumstances obtained through computer simulation. Among them, for areas jointly identified as blind spots for monitoring by both, mark them as status confirmed; for areas where the two identifications are different (i.e., one method considers it a blind spot for monitoring while the other method does not), mark them as status to be confirmed (such differences may be due to limitations in image analysis (such as image resolution, shooting angle, etc.) or inaccurate parameter settings in computer simulation, etc.).

[0181] Then, for areas with status to be confirmed, the wheeled robot can be controlled to re-focus on inspection, or manual intervention can be requested for confirmation, thereby optimizing the analysis results of the blind spot areas for monitoring.

[0182] Optionally, when the wheeled robot reaches the area with status to be confirmed, the robot can collect information on this area from multiple angles and in multiple directions to determine whether there is really a blind spot for monitoring in this area. For example, the robot can adjust the angle of the camera and take pictures of this area from different directions to check whether there are places that cannot be covered by the monitoring devices.

[0183] Optionally, when the detection results of the wheeled robot still cannot clearly determine whether this area is a blind spot for monitoring, or in some special cases (such as when the wheeled robot cannot reach this area), request manual intervention. The manual personnel can go to the area with status to be confirmed for on-site investigation.

[0184] Update the previous analysis results of blind spot areas based on the results rechecked by the wheeled robot or confirmed through manual intervention. Clearly divide the areas to be confirmed into blind spots or non-blind spots for monitoring, and update them to the final analysis report.

[0185] In one embodiment, by comparing the analysis results of different methods and further detecting and confirming the areas to be confirmed, the situations of misjudgment and missed judgment can be effectively reduced, and the accuracy of the analysis results of blind spot areas for monitoring can be improved.

[0186] In one embodiment, on the basis of the above embodiment, the intelligent inspection method of the wheeled robot further includes:

[0187] Conduct risk analysis and early warning for the production workshop based on the inspection images of the wheeled robot and the monitoring images of the monitoring devices.

[0188] In this embodiment, the wheeled robot moves in the production workshop along the pre-planned inspection path, and the image sensor carried by it collects images at certain time intervals or at specific positions. The planning of the inspection path needs to comprehensively cover all key areas of the production workshop, including equipment-intensive areas, passages, storage areas, etc.

[0189] The monitoring devices fixedly installed in the workshop continuously take images of the areas they are responsible for. These monitoring devices are distributed at different positions in the workshop and have different perspectives to achieve full-round monitoring of the workshop.

[0190] Perform preprocessing operations on the collected inspection images of the wheeled robot and the monitoring images of the monitoring devices, including image denoising, grayscale conversion, histogram equalization, etc. For denoising, methods such as Gaussian filtering can be used to remove noise interference in the images; grayscale conversion converts color images into grayscale images to reduce the amount of data; histogram equalization enhances the contrast of the images to improve the accuracy of subsequent analysis.

[0191] Optionally, identify various production equipment from the images and analyze its appearance and operating status. For example, check whether the shell of the equipment is damaged or deformed, whether the indicator light status of the equipment is normal, and whether there is abnormal jitter in the moving parts of the equipment. Through feature extraction algorithms such as edge detection and contour extraction, accurately extract the key features of the equipment.

[0192] Optionally, detect and track the personnel in the images and analyze whether their behaviors comply with safety specifications. For example, detect whether the personnel wear necessary safety protection equipment (safety helmets, goggles, etc.), whether they stay or operate illegally in dangerous areas, and whether there are dangerous behaviors such as running and pushing. Use technologies such as human pose estimation and target tracking to achieve the extraction of personnel behavior characteristics.

[0193] Optionally, observe the environmental information in the image, such as whether there is water accumulation or oil stain on the ground, whether the passage is unobstructed, and whether there are abnormal situations such as smoke or flame. Extract the characteristics of the environmental conditions through methods such as color analysis and texture analysis.

[0194] Optionally, collect a large amount of image data in advance and label the risk characteristics therein, such as equipment failure, personnel violation, environmental hazard, etc. Divide the labeled data into a training set, a validation set, and a test set for constructing and evaluating a risk analysis model.

[0195] Optionally, select a suitable machine learning or deep learning model, such as a convolutional neural network, a recurrent neural network, etc. Train the model with the training set data, and by continuously adjusting the parameters of the model, enable the model to accurately identify the risk characteristics in the image and classify the risk levels.

[0196] According to the severity of the risk and the possible impacts, divide the risks into different levels, such as low risk, medium risk, and high risk. For example, minor wear of equipment can be classified as low risk, situations where personnel's illegal operations may lead to accidents can be classified as medium risk, and potential risks of serious accidents such as fires and explosions are classified as high risk.

[0197] Set corresponding warning rules for different levels of risks. When the model detects the risk characteristics in the image and determines the risk level, trigger the corresponding warning signal according to the warning rules. For example, low risk can be warned through the prompt information on the monitoring system interface, medium risk can send text messages or emails to notify relevant personnel, and high risk immediately triggers an audible and visual alarm and notifies the emergency rescue department.

[0198] Perform real-time analysis on the inspection images of the wheeled robot and the monitoring images of the monitoring equipment. Once risk characteristics are found, immediately give warnings according to the warning rules. At the same time, record the warning information for subsequent query and analysis.

[0199] In one embodiment, the wheeled robot can flexibly move to every corner of the production workshop for inspection and collect images of areas that are difficult to cover due to the fixed position of the monitoring equipment. The monitoring equipment can continuously and stably monitor the key areas of the workshop. The combination of the two realizes the all-round and dead-angle-free monitoring of the production workshop, greatly increasing the probability of risk discovery.

[0200] The wheeled robot can collect images from different angles and distances, complementing the fixed-view images provided by the monitoring equipment. This enables richer information to be obtained for the same area or object, helping to more accurately identify potential risks and reducing the situations of misjudgment and missed judgment.

[0201] The wheeled robot collects images in real time during the inspection process, and the monitoring equipment also works continuously, which can capture the dynamic changes in the production workshop in a timely manner. Whether it is a sudden failure of equipment, illegal operation of personnel or abnormal environment, it can be discovered and analyzed and warned in the first time.

[0202] In addition, refer to Figure 2 In an embodiment of the present application, a control device Z10 is further provided, comprising:

[0203] The monitoring module Z11 is used to collect monitoring images of the production workshop based on monitoring devices distributed in multiple different locations in the production workshop, and to stitch the monitoring images collected by each monitoring device to generate an overall monitoring image;

[0204] The extraction module Z12 is used to analyze the overall monitoring image, extract the workshop map of the production workshop, and mark the obstacles and passages in the workshop map;

[0205] The scanning module Z13 is used to control the wheeled robot to perform a global scan of the production workshop based on the workshop map to generate a 3D global image; wherein, during the global scanning process, the wheeled robot supplements and updates the workshop map according to the detection data collected by the detection sensor integrated in the body; the detection sensor includes at least an image sensor, a radar sensor and a collision detection sensor;

[0206] The analysis module Z14 is used to analyze the missing area of the overall monitoring image compared with the 3D global image, and identify the missing area in combination with the latest workshop map to obtain the monitoring blind spots in the production workshop;

[0207] The planning module Z15 is used to plan the inspection path of the wheeled robot using a multi-objective optimization algorithm based on the charging position of the wheeled robot and the location of each monitoring blind spot area; the optimization goals of the multi-objective optimization algorithm planning include the shortest inspection time for a single round and the minimum energy consumption for a single round of inspection;

[0208] The inspection module Z16 is used to control the wheeled robot to perform intelligent inspection of the production workshop based on the inspection path; wherein, based on the detection data collected in real time by the wheeled robot during the inspection process, the wheeled robot is controlled to perform intelligent obstacle avoidance.

[0209] Optionally, the control device Z10 may be a virtual control device (such as a virtual machine) or a physical device (such as a physical device other than a control system that can execute a corresponding method).

[0210] In addition, a control system is also provided in an embodiment of the present application. The internal architecture of the control system can be as follows: Figure 3As shown in the figure, it includes a processor, a memory, a communication interface, and an input interface connected through a system bus. Among them, the processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database is used to store data called by the computer program. The communication interface is used to communicate with external terminals. The input interface is used to receive signals input by external devices. When the computer program is executed by the processor, it implements an intelligent inspection method for a wheeled robot as described in the above embodiments.

[0211] Those skilled in the art can understand that Figure 3 the structure shown in the figure is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the control system to which the solution of this application is applied. For example, in some alternative embodiments, the control system may further include an output interface (not shown in the figure), and the output interface is also connected to the system bus and is used to output corresponding signals to external devices.

[0212] In addition, this application also proposes a computer-readable storage medium, the computer-readable storage medium includes a computer program, and when the computer program is executed by the processor, it implements the steps of the intelligent inspection method for the wheeled robot as described in the above embodiments. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0213] In summary, for the intelligent inspection method, control device, control system, and computer-readable storage medium of the wheeled robot provided in the embodiments of this application, images are collected and stitched by multi-position monitoring devices to generate an overall monitoring image, laying a foundation for subsequent analysis, and analyzing the images to extract the workshop map and mark obstacles and passages to provide guidance for the robot's movement. The wheeled robot performs a global scan to generate a 3D global image, uses a variety of sensors to update the map in real time, and accurately finds monitoring blind spots. Then, a multi-objective optimization algorithm is used to plan the inspection path, taking into account both inspection time and energy consumption, improving the inspection efficiency and reducing costs. In this way, the combination of the wheeled robot and multi-position monitoring devices is used to achieve efficient, intelligent, and comprehensive monitoring of the production workshop.

[0214] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium provided in this application and used in the embodiments can include non-volatile and / or volatile memories. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0215] It should be noted that in this article, the terms "including", "comprising", or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, device, article, or method including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such a process, device, article, or method. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, device, article, or method including that element.

[0216] The above are only the preferred embodiments of this application, and do not limit the patent scope of this application. Any equivalent structural or equivalent process transformation made by using the specification and drawings of this application, or directly or indirectly applied in other related technical fields, are equally included in the patent protection scope of this application.

Claims

1. An intelligent inspection method for a wheeled robot, characterized in that, Including: Based on monitoring devices distributed at multiple different locations in the production workshop, collect monitoring images of the production workshop, and perform image stitching on the monitoring images collected by each monitoring device to generate an overall monitoring image; Analyze the overall monitoring image, extract the workshop map of the production workshop, and mark the obstacles and passages in the workshop map; Based on the workshop map, control the wheeled robot to perform a global scan of the production workshop to generate a 3D global image; wherein, during the global scan, the wheeled robot supplements and updates the workshop map according to the detection data collected by the detection sensors integrated in the body; the detection sensors at least include an image sensor, a radar sensor, and a collision detection sensor; Analyze the missing area part of the overall monitoring image compared with the 3D global image, and combine it with the latest workshop map to identify the missing area part to obtain each monitoring blind spot area in the production workshop; According to the layout and field of view of the monitoring devices in the production workshop, use computer simulation technology to virtually simulate the monitoring situation of the production workshop to simulate monitoring blind spots in different situations; Based on the simulation results, optimize the analysis results of the monitoring blind spot areas; wherein, compare the simulation results with the analysis results of the monitoring blind spot areas; if the areas jointly identified as monitoring blind spots by both are marked as status confirmed; if the areas identified differently by both are marked as status to be confirmed; for the areas with status to be confirmed, control the wheeled robot to re-inspect; when the wheeled robot reaches the area with status to be confirmed, the wheeled robot collects information about this area from multiple angles and in multiple directions to determine whether there is really a monitoring blind spot in this area; Based on the charging position of the wheeled robot and the locations of various blind spots for monitoring, a multi-objective optimization algorithm is used to plan the inspection path of the wheeled robot; and, feature recognition is performed on the 3D global image, and using the feature recognition results, semantic information of the devices and objects in each blind spot area for monitoring is generated; based on the semantic information of the devices and objects in the blind spot area for monitoring, the corresponding inspection priorities are determined; among them, the optimization objectives planned by the multi-objective optimization algorithm include the shortest inspection time for a single round and the minimum energy consumption for a single round of inspection; during the path planning process, based on the genetic algorithm, a group of initial inspection paths is randomly generated as the population, and each path is represented as an access order of the blind spot areas for monitoring, and the starting point and the ending point of the path are both the charging position; for each path in the population, the inspection time T for a single round and the energy consumption E for a single round of inspection are calculated; the two objective values are combined into a fitness value F using weighted summation, and it is set that F = w1T + w2E, where w1 and w2 are weight coefficients, and w1 + w2 = 1, w1 > w2; according to the fitness value F, the paths in the population are selected, and the paths with a fitness greater than the preset threshold are selected as the parents for generating the next generation of paths; crossover operations are performed on the selected parent paths to generate new offspring paths, and then mutation operations are performed on the offspring paths; the operations of fitness value evaluation, population path selection, parent path crossover, and offspring path mutation are repeated until the termination condition is met; after the iteration ends, the path with the optimal fitness value F is selected from the final population as the inspection path of the wheeled robot; among them, during the search process of the algorithm, according to the inspection priorities of the blind spot areas for monitoring, the path selection strategy is adjusted, and the blind spot areas for monitoring with higher inspection priorities are preferentially selected; Based on the inspection path, control the wheeled robot to perform intelligent inspection on the production workshop; among them, based on the detection data collected in real time during the inspection process by the wheeled robot, control the wheeled robot to perform intelligent obstacle avoidance.

2. The intelligent inspection method of the wheeled robot according to claim 1, wherein, The steps of controlling the wheeled robot to perform intelligent obstacle avoidance based on the detection data collected in real time during the inspection process by the wheeled robot include: Based on the detection data collected in real time during the inspection process by the wheeled robot, analyze whether there are obstacles in front of the forward path of the wheeled robot; If so, based on the visual detection technology, estimate the duration of the obstacle; Select an obstacle avoidance strategy based on the duration of the obstacle; among them, the obstacle avoidance strategies include bypassing obstacle avoidance and waiting for obstacle avoidance.

3. The intelligent inspection method of the wheeled robot according to claim 2, characterized in that, The steps of selecting an obstacle avoidance strategy based on the duration of the obstacle include: Estimate the bypassing duration required for bypassing obstacle avoidance; If the bypassing duration is less than the obstacle avoidance duration, select bypassing obstacle avoidance as the obstacle avoidance strategy; otherwise, select waiting for obstacle avoidance as the obstacle avoidance strategy.

4. The intelligent inspection method of the wheeled robot according to claim 1, characterized in that, The intelligent inspection method of the wheeled robot further includes: When the wheeled robot starts to perform intelligent inspection on the production workshop, based on the monitoring equipment, obtain the personnel density of each target area passed by the inspection path; Before the wheeled robot arrives at the corresponding target area, adjust the moving speed of the wheeled robot in advance according to the personnel density; among them, the higher the personnel density, the lower the adjusted moving speed.

5. The intelligent inspection method of the wheeled robot according to any one of claims 1-4, characterized in that, The intelligent inspection method of the wheeled robot further includes: Based on the inspection images of the wheeled robot and the monitoring images of the monitoring equipment, perform risk analysis and early warning on the production workshop.

6. A control device, characterized in that, It includes: A monitoring module, which is used to collect the monitoring images of the production workshop based on the monitoring equipment distributed at multiple different positions in the production workshop, and splice the monitoring images collected by each monitoring equipment to generate an overall monitoring image; An extraction module, which is used to analyze the overall monitoring image, extract the workshop map of the production workshop, and mark the obstacles and passages in the workshop map; A scanning module, which is used to control the wheeled robot to perform a global scan of the production workshop based on the workshop map to generate a 3D global image; wherein, during the global scan, the wheeled robot supplements and updates the workshop map according to the detection data collected by the detection sensors integrated in the body; the detection sensors at least include an image sensor, a radar sensor, and a collision detection sensor; An analysis module, which is used to analyze the missing area part of the overall monitoring image compared with the 3D global image, and combine the latest workshop map to identify the missing area part to obtain each monitoring dead corner area in the production workshop; according to the layout and field of view of the monitoring equipment in the production workshop, use computer simulation technology to virtually simulate the monitoring situation of the production workshop to simulate the monitoring dead corners in different situations; based on the simulation results, optimize the analysis results of the monitoring dead corner area; wherein, compare the simulation results with the analysis results of the monitoring dead corner area; if the areas jointly identified as monitoring dead corners by both are marked as status confirmed; if the areas identified differently by both are marked as status to be confirmed; for the areas with status to be confirmed, control the wheeled robot to re-inspect; when the wheeled robot reaches the area with status to be confirmed, the wheeled robot collects information on this area from multiple angles and in multiple directions to determine whether there is really a monitoring dead corner in this area; A planning module, which is used to plan the inspection path of the wheeled robot based on the charging position of the wheeled robot and the positions of each blind spot area, by using a multi-objective optimization algorithm; and, perform feature recognition on the 3D global image, generate semantic information of the devices and objects in each blind spot area by using the feature recognition results; determine the corresponding inspection priorities based on the semantic information of the devices and objects in the blind spot area; wherein, the optimization objectives of the multi-objective optimization algorithm planning include the shortest inspection time for a single round and the minimum inspection energy consumption for a single round; during the path planning process, based on the genetic algorithm, a group of initial inspection paths are randomly generated as a population, each path is represented as an access order of a blind spot area, and the starting point and the ending point of the path are both the charging position; for each path in the population, calculate the inspection time T for a single round and the inspection energy consumption E for a single round; use weighted summation to combine these two objective values into a fitness value F, and set F = w1T + w2E, where w1 and w2 are weight coefficients, and w1 + w2 = 1, w1 is greater than w2; select the paths in the population according to the fitness value F, select the paths with a fitness greater than a preset threshold as the parents for generating the next generation of paths; perform crossover operations on the selected parent paths to generate new offspring paths, and then perform mutation operations on the offspring paths; repeat the fitness value evaluation, population path selection, parent path crossover, and offspring path mutation operations until the termination condition is met; after the iteration ends, select the path with the optimal fitness value F from the final population as the inspection path of the wheeled robot; wherein, during the search process of the algorithm, according to the inspection priorities of the blind spot areas, adjust the path selection strategy, and preferentially select the blind spot areas with high inspection priorities; An inspection module, which is used to control the wheeled robot to perform intelligent inspection on the production workshop based on the inspection path; wherein, based on the detection data collected in real time by the wheeled robot during the inspection, control the wheeled robot to perform intelligent obstacle avoidance.

7. A control system, characterized in that, The control system includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the intelligent inspection method of the wheeled robot according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium. When the computer program is executed by the processor, it implements the steps of the intelligent inspection method of the wheeled robot according to any one of claims 1 to 5.

Images