Roller monitoring method and device based on three-light fusion and inspection robot

Abstract

The embodiment of the application provides a kind of based on three light fusion's roller monitoring method, device and inspection robot, the method comprises: obtaining the monitoring image containing roller support number and rotating roller;Obtain the optical image and infrared thermal imaging of the position where roller is at corresponding moment;Real-time identification of each roller in optical image is carried out using roller identification model, and the actual temperature of each roller is determined according to infrared thermal imaging;When detecting that the actual temperature of the roller exceeds the preset temperature threshold, trigger the roller overheating alarm, and extract the video clip corresponding to the alarm time from the monitoring image, and the video clip is used to obtain the support number information where the overheating roller is located.The scheme can realize the real-time monitoring of the temperature of the roller and automatically alarm and locate when high temperature occurs, thereby effectively preventing the damage of the equipment caused by the overheating of the roller, and greatly reducing the labor cost.

Description

Technical Field

[0001] This application relates to the field of idler roller monitoring technology, and in particular to an idler roller monitoring method, device and inspection robot based on three-light fusion. Background Technology

[0002] In long-distance industrial belt conveyors (hereinafter referred to as belt conveyors), the high-speed transmission of the belt and the high-speed rotation of the idler rollers in contact with it lead to overheating. Traditional methods for detecting idler roller overheating primarily rely on single-lens optical cameras for monitoring. For example, high-resolution cameras can be used to monitor the wear condition of the idler rollers to determine if there are any abnormalities. The advantages of single-lens optical cameras are their intuitiveness and low hardware cost. However, several problems exist in practical applications. For instance, in high-speed belt conveyors, the idler rollers are constantly rotating at high speed, making idler roller wear detection very difficult. This necessitates a certain amount of manpower for monitoring each belt conveyor, resulting in high labor costs. Moreover, when wear occurs, the actual overheating temperature of the idler roller is already very severe, potentially threatening normal production, making preventative measures impossible. Summary of the Invention

[0003] In view of this, embodiments of this application provide a method, device, and inspection robot for monitoring idler rollers based on three-light fusion.

[0004] In a first aspect, embodiments of this application provide a method for monitoring idler rollers based on three-light fusion, including:

[0005] Acquire monitoring images containing the idler bracket number and the rotating idler;

[0006] Acquire optical and infrared thermal images of the location of the idler roller at the corresponding time.

[0007] The idler roller identification model is used to identify each idler roller in the optical image in real time, and the actual temperature of each idler roller is determined based on the infrared thermal imaging.

[0008] When the actual temperature of an idler roller exceeds a preset temperature threshold, an overheating alarm is triggered, and a video clip corresponding to the alarm time is extracted from the monitoring image. The video clip is used to obtain the bracket number information of the overheated idler roller.

[0009] Secondly, embodiments of this application provide a roller monitoring device based on three-light fusion, comprising:

[0010] The data acquisition module is used to acquire monitoring images containing the idler bracket number and the rotating idler.

[0011] The data acquisition module is also used to acquire optical images and infrared thermal images of the location of the idler roller at the corresponding time.

[0012] A roller identification module is used to identify each roller in the optical image in real time using a roller identification model;

[0013] A temperature calculation module is used to determine the actual temperature of each idler roller based on the infrared thermal imaging.

[0014] The alarm control module is used to trigger an overheat alarm for idlers when the actual temperature of an idler exceeds a preset temperature threshold, and to extract the video segment corresponding to the alarm time from the monitoring image. The video segment is used to obtain the bracket number information of the overheated idler.

[0015] Thirdly, embodiments of this application provide an inspection robot, which is equipped with a global optical camera, a local optical camera, an infrared thermal imaging camera, a processor, and a memory; the memory stores a computer program, and the processor is used to execute the computer program to implement the above-mentioned roller monitoring method based on three-light fusion;

[0016] The global optical camera is used to acquire monitoring images that capture the idler support number and the rotating idler; the local optical camera is used to acquire optical images of the location of the idler at a corresponding moment; and the infrared thermal imaging camera is used to acquire infrared thermal images of the same location at the same moment.

[0017] Fourthly, embodiments of this application provide a readable storage medium storing a computer program, which, when executed on a processor, implements the aforementioned roller monitoring method based on three-light fusion.

[0018] The embodiments of this application have the following beneficial effects:

[0019] This application presents a three-light fusion-based idler monitoring method for monitoring overheating of idlers on conveyor belts. It uses a global optical camera to acquire monitoring images containing idler bracket numbers and rotating idlers, achieving traditional monitoring. Simultaneously, a local optical camera and an infrared thermal imaging camera acquire optical and infrared thermal images of the idler's location, respectively. Then, an idler recognition model identifies each idler in the optical images in real time and determines the actual temperature of each idler based on the infrared thermal imaging. When the actual temperature of an idler exceeds a preset temperature threshold, an overheating alarm is triggered, and a video clip corresponding to the alarm time is extracted from the monitoring images. This video clip is used to obtain the bracket number information of the overheated idler. This solution utilizes three-light data (data from two optical cameras plus data from an infrared camera) to achieve real-time monitoring of idler temperature, automatic alarm in case of high temperatures, and extraction of overheated idler information. This not only effectively prevents equipment damage caused by idler overheating and significantly reduces labor costs, but also allows maintenance personnel to quickly locate the overheated idler by simply viewing the video clips captured by the global camera, improving maintenance and production efficiency. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A schematic diagram of the inspection robot according to an embodiment of this application is shown;

[0022] Figure 2 Another structural schematic diagram of the inspection robot according to an embodiment of this application is shown;

[0023] Figure 3 A flowchart of the roller monitoring method based on three-light fusion according to an embodiment of this application is shown;

[0024] Figure 4 The image shown is an image of the idler roller bracket number captured by a global optical camera according to an embodiment of this application;

[0025] Figure 5 A flowchart illustrating the acquisition process of the idler roller recognition model according to an embodiment of this application is shown;

[0026] Figure 6 A schematic diagram of a first structure of the idler roller recognition model according to an embodiment of this application is shown;

[0027] Figure 7A schematic diagram of the intermediate optimized residual structure according to an embodiment of this application is shown;

[0028] Figure 8 This paper shows a schematic diagram of the spatial pyramid pooling layer in the idler roller recognition model of this application embodiment;

[0029] Figure 9 A second structural schematic diagram of the idler roller recognition model according to an embodiment of this application is shown;

[0030] Figure 10 A flowchart of roller temperature detection according to an embodiment of this application is shown;

[0031] Figure 11 A schematic diagram of actual idler roller temperature identification is shown;

[0032] Figure 12 A schematic diagram of the structure of the idler roller monitoring device based on three-light fusion according to an embodiment of this application is shown. Detailed Implementation

[0033] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0034] The components of the embodiments of this application described and illustrated in the accompanying drawings can be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of this application provided in the drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0035] In the following text, the terms "comprising," "having," and their cognates, which may be used in various embodiments of this application, are intended only to indicate a particular feature, number, step, operation, element, component, or combination thereof, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations thereof, or adding the possibility of one or more combinations thereof. Furthermore, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0036] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of this application pertain. Terms (such as those defined in commonly used dictionaries) shall be interpreted as having the same meaning as in their contextual meaning in the relevant technical field and shall not be construed as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of this application.

[0037] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0038] Please refer to Figure 1 This is a schematic diagram of the inspection robot proposed in this application embodiment. The inspection robot is mainly used to monitor whether the idlers on the conveyor belt or belt conveyor are experiencing overheating or other operational issues. Exemplarily, the inspection robot includes a memory 11, a processor 12, and an information acquisition unit 13. The information acquisition unit 13 is used to acquire information about the external environment or the robot itself. The memory 11 stores a computer program, and the processor 12 is used to execute the computer program to implement the three-light fusion-based idler monitoring method of this application embodiment, thereby effectively preventing idler overheating and equipment damage.

[0039] The memory 11 may be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory 11 stores computer programs, and the processor 12 can execute the computer programs accordingly after receiving execution instructions.

[0040] The processor 12 can be an integrated circuit chip with signal processing capabilities. The processor 12 can be a general-purpose processor, including at least one of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Network Processor (NP), Digital Signal Processor (DSP), Application-Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0041] The information acquisition unit 13 may include, but is not limited to, an optical camera for acquiring global and / or local information, and an infrared thermal imaging camera (also known as an infrared thermal imaging sensor) for acquiring infrared temperature information, etc., which can be specifically set according to the actual application scenario of the robot. Taking the roller monitoring scenario of this embodiment as an example, such as Figure 2 As shown, the inspection robot can be equipped with a global optical camera 131, a local optical camera 132, and an infrared thermal imaging camera 133. For example, the local optical camera 132 and the infrared thermal imaging camera 133 can be installed on one side of the robot body, such as by being adjacent left and right or top and bottom, or these two structures can be installed separately. The global optical camera 131 can be installed on the other side of the body. The placement of these three structures can be set according to actual needs and is not limited here. It should be understood that the advantage of placing the local optical camera 132 and the infrared thermal imaging camera 133 on one side of the robot body, compared to separating them on both sides, is that, in the case of the same side, the accuracy of converting the coordinates of the roller center in the optical coordinate system to the coordinates in the infrared coordinate system using the perspective transformation algorithm is higher. In addition, each of these three structures can be configured with a corresponding rotation drive unit to achieve shooting at different rotation angles.

[0042] As a preferred embodiment, the aforementioned global optical camera 131, local optical camera 132, and infrared thermal imaging camera 133 can all be directly connected to the robot's processor 12 (also known as the edge computing unit) via a network cable to enable real-time processing of the data collected by these three structures. In this embodiment, the global optical camera 131 can employ a panoramic optical lens, primarily used for traditional security monitoring and perimeter defense identification; the local optical camera 132 can be used to monitor the location of local targets, such as the location of the idler roller; and the infrared thermal imaging camera 133 can be used to collect the infrared temperature of the target area, such as for temperature detection of the idler roller.

[0043] Based on the inspection robot with the above structure, this application proposes a three-light fusion-based idler monitoring method, mainly applied to the inspection scenario of conveyor belts. Specifically, it utilizes the inspection robot to monitor whether the idlers on the conveyor belt are overheating. This solution uses three-light data (i.e., data from two optical cameras plus data from an infrared thermal imaging camera) to achieve real-time monitoring of the idler temperature, automatic alarm and location information extraction when the idler temperature is detected to be too high. This not only effectively prevents equipment damage caused by idler overheating, but also allows for quick location of the overheated idler simply by analyzing recorded video clips, improving maintenance efficiency. Specific embodiments are described below.

[0044] Please refer to Figure 3 The roller monitoring method based on three-light fusion includes steps S110 to S140:

[0045] S110, acquire a monitoring image containing the idler bracket number and the rotating idler.

[0046] Typically, each set of idlers has its own belt support number for quick location of the corresponding idler. When the belt conveyor is in operation, the idlers rotate at high speed, allowing the inspection robot to monitor the conveyor via a patrol method. For example, when monitoring a section of the belt conveyor, a global monitoring image can be obtained using its installed global optical camera, such as... Figure 4 As shown, the monitoring image mainly refers to the image that captures the bracket number where the idler roller is located and multiple idler rollers.

[0047] It is understandable that this monitoring image can be used to achieve traditional security monitoring, namely monitoring the conveyor belt and the surrounding external environment within the entire field of view; on the other hand, since it is set in a position that can capture information about the idler support, if an idler overheats, it can also save a short video that provides information about the location of the overheated idler, so that workers can use the idler support number recorded in the video to carry out equipment maintenance, thereby improving maintenance efficiency.

[0048] S120: Acquire the optical image and infrared thermal image of the location of the idler roller at the corresponding time.

[0049] Optical images primarily refer to images captured using an optical camera; while infrared thermal imaging refers to infrared images acquired using an infrared thermal imaging camera. Exemplarily, the inspection robot can use an installed local optical camera to capture images of a local area facing the location of the idler roller, and then identify the roller to determine its specific position. Simultaneously, an installed infrared thermal imaging camera also captures images of the same location (i.e., the same area) of the idler roller at the same time to obtain infrared thermal images of the corresponding area.

[0050] It is understandable that since the idler rollers are within the captured image, their infrared temperature information is also reflected in the infrared thermal image. Therefore, by identifying the position of each idler roller, the temperature information of the corresponding roller can be determined. Of course, in this embodiment, the local optical camera and the infrared thermal imaging camera can be calibrated in advance to ensure that the images captured by both correspond.

[0051] As a preferred solution, the local optical camera can be directly connected to the edge computing unit of the inspection robot to directly transmit the raw image data, thus eliminating the need for video stream compression and decompression. This ensures that the robot can still provide real-time images of the idler rollers during its inspection process, which can be used in conjunction with infrared thermal imaging for accurate idler roller identification and temperature detection.

[0052] S130 uses a roller recognition model to identify each roller in the optical image in real time and determines the actual temperature of each roller based on the infrared thermal imaging.

[0053] The idler roller recognition model is mainly used to identify the idler roller as a target object in the image, and can be obtained by pre-training a constructed convolutional neural network model. In this embodiment, the idler roller information identified by the idler roller recognition model may include, but is not limited to, the coordinates of the center of each idler roller in the optical image, which can be denoted as the first coordinate to distinguish it from the idler roller center coordinates in the infrared thermal imaging (denoted as the second coordinate) mentioned later.

[0054] Furthermore, after identifying the idler rollers, the temperature information of each idler roller can be further determined from the infrared thermal imaging. In this embodiment, the temperature of the center of the idler roller is used as the temperature of the entire idler roller being measured. Of course, the temperature of the entire idler roller can also be calculated from the temperature of the entire identified idler roller area, etc. The specific method can be adjusted according to actual needs, and is not limited here.

[0055] In one implementation, such as Figure 5 As shown, the above-mentioned idler roller recognition model is obtained through pre-training, including sub-steps S210 to S240:

[0056] S210, collects image datasets of idlers on the conveyor belt.

[0057] For example, a large number of images containing idlers can be pre-collected, and the idlers in these images can be labeled with rectangular boxes to serve as sample images for model training.

[0058] S220, use a clustering algorithm to calculate the roller anchor frame parameters in the image dataset. These roller anchor frame parameters are used to determine the anchor frame size of the model training input.

[0059] For example, clustering algorithms can employ K-means and similar methods. An anchor box, in object detection algorithms, refers to multiple predefined bounding boxes with different aspect ratios, centered at an anchor point. Clustering algorithms are used to calculate the aspect ratios of the anchor boxes to configure the anchor box size for the rollers used as input for model training.

[0060] S230, perform image size unification and image enhancement processing on the image dataset to obtain a training sample set with the same image size.

[0061] Image size standardization refers to setting the size of each roller image in the image dataset to the same preset size, such as 480x288, which can be determined according to the size requirements of the model input image, so as to facilitate the input of sample images into the convolutional neural network model for direct processing.

[0062] Image enhancement processing can increase sample diversity, thereby improving the robustness of the trained model. For example, Mosaic and CutMix data augmentation algorithms can be used for image enhancement. Mosaic data augmentation mainly combines multiple images into one image according to a certain ratio, typically using four images stitched together, enabling the model to recognize targets within a smaller area. It's understandable that each set of idler rollers is wrapped around the conveyor belt, and there is a problem of partial occlusion. Therefore, image enhancement processing can ensure that the trained idler roller recognition model can adapt to more complex scenarios.

[0063] S240, the constructed convolutional neural network model is iteratively trained using the training sample set until the training loss value meets the preset conditions, and then the training is stopped to obtain the trained idler roller recognition model.

[0064] In one implementation, the constructed convolutional neural network model (denoted as the HDRollerDet network) mainly consists of three parts, such as... Figure 6As shown, the network consists of a feature extraction network (P1), a pooling transition network (P2), and a feature fusion network (P3). The feature extraction network primarily extracts feature maps from the input image containing the idler roller from low to high levels, obtaining the corresponding feature maps. The pooling transition network performs pooling and fusion processing on the feature maps output from a specified layer in the feature extraction network to obtain an expanded feature map. Residual optimization is then performed on the expanded feature map to obtain an optimized feature map. The feature fusion network primarily upsamples the obtained optimized feature map and fuses it with a feature map of the same size output from the feature extraction network to output the idler roller prediction result.

[0065] like Figure 6 As shown, this feature extraction network can include sequentially connected input convolutional layers, multiple optimized residual structures (denoted as C3 structure), and an output convolutional layer, used for extracting features from shallow detail features (i.e., low-level) to high-level semantic features (i.e., high-level) at scales from 1 / 2 to 1 / 32. Specifically, both the input and output convolutional layers can be constructed from Conv modules, mainly differing in the number of channels; the Conv module can consist of a 3x3 convolutional kernel, a batch normalization (BN) module, and a ReLU activation function. The intermediate structure between the input and output convolutional layers is an optimized residual structure, such as... Figure 6 As shown, the feature extraction network includes three C3*n structures, where n is 1, 3, and 3 respectively. In this embodiment, the optimized residual structure can significantly reduce the number of parameters and accelerate inference speed while maintaining better accuracy and performance. Compared with the residual network structure, its speed can be improved by up to 20%.

[0066] In one implementation, such as Figure 7 As shown, the optimized residual structure includes two branches equally divided according to the input channels of the input convolutional layer, such as each branch having N / 2 channels. Each branch includes one convolutional module (i.e., Figure 7 In the Conv, N / 2), one branch also includes a bottleneck structure (i.e., BottleNeck*k) connected to the output of the convolutional module; then, the outputs of these two branches are connected to a splicing unit (i.e., Figure 7 The concat function (N) is then connected to the convolution output module (Conv, N), which concatenates the channels and then performs convolution. At this point, the output of the optimized residual unit is still N channels.

[0067] like Figure 6 As shown, the pooling transition network includes a spatial pyramid pooling layer and an optimized residual structure. In one implementation, the spatial pyramid pooling layer (also known as the SPP network) is as follows: Figure 8As shown, it includes a convolutional module (i.e. Figure 8 The convolutional module (Conv structure) and three pooling units with different kernel sizes are connected to a stitching unit, which performs the feature fusion operation. For example, the three pooling units can use kernel sizes of 5*5, 9*9, and 13*13, but this is just one example and not the only limitation. It can be understood that when the roller is close to the camera lens, using a spatial pyramid structure can improve the detection performance of large objects.

[0068] In one implementation, the feature fusion network may include a single-scale feature fusion structure, such as... Figure 6 As shown, the feature fusion network includes an upsampling unit, a stitching unit, and a 1*1 convolutional module. For example, the upsampling unit is used to upsample from a 1 / 32 scale to a 1 / 16 scale, and then connects it with the 1 / 16 scale feature map output by the feature extraction network according to the channels. Finally, after 1*1 convolution processing, the result of roller recognition is output.

[0069] Furthermore, when the inspection robot's gimbal rotates to other angles to detect rollers in distant images, the distant rollers may appear small in the image, making them difficult to distinguish. To improve the model's detection performance for small objects in the distance, as an option, a multi-scale feature fusion structure can be adopted, outputting prediction results at multiple different scales, depending on the actual needs. For example... Figure 9 As shown, the feature fusion network in this idler roller recognition model may include splicing units that splice multiple scales, such as outputting at three scales of 1 / 8, 1 / 16, and 1 / 32.

[0070] It's understandable that using multi-scale output increases data processing time and reduces real-time performance. However, for scenarios where real-time performance isn't critical, multi-scale fusion ensures the model can effectively identify idlers even when they are far from the camera. Therefore, users can choose between single-scale and multi-scale fusion structures based on their specific needs.

[0071] At this point, after constructing the convolutional neural network model, the corresponding loss function is designed to determine the training conditions for the model. In one implementation, this loss function can be based on classification loss (denoted as L). obj ) and regression loss (denoted as L) reg The model's total loss function L can be constructed using various methods, such as cross-entropy loss and DIoU function. For example, the classification loss can be constructed using the cross-entropy loss function, and the regression loss can be constructed using the DIoU function. In this case, the model's total loss function L is: L = L obj +L regIt is understood that the classification loss and regression loss described above are merely examples. In practical applications, users can make adaptive adjustments based on these examples, which are not limited here.

[0072] Next, the constructed convolutional neural network model is trained using the training sample set. During each training iteration, the network parameters are adjusted using the loss value calculated from the output prediction results, and then the next training iteration begins. This iterative training continues until a training stopping condition is met, resulting in a trained roller recognition model. The training stopping condition can be that the calculated loss value is less than a preset threshold (i.e., sufficiently small), or that a preset number of training iterations has been reached; these conditions are not specified here.

[0073] Therefore, the aforementioned idler recognition model is used to identify each idler in the obtained optical image, for example, outputting the coordinates of the center of each idler in the image and the size of each idler in the image. Then, the actual temperature of each idler is determined.

[0074] like Figure 10 As shown, the process of obtaining the actual temperature of each idler roller, exemplarily, includes:

[0075] S310, using a perspective transformation algorithm, the first coordinate of the center of each roller is transformed to a second coordinate in the infrared coordinate system constructed based on the infrared thermal imaging.

[0076] S320, the temperature value corresponding to each of the second coordinates in the infrared thermal image is taken as the infrared temperature of the center of the roller at the corresponding position, and the infrared temperature of the center of the roller is defined as the actual temperature of the roller.

[0077] The perspective transformation algorithm primarily utilizes a perspective transformation matrix to achieve coordinate transformation between the optical coordinate system and the infrared coordinate system. In this embodiment, the two coordinate systems need to be pre-calibrated. The perspective transformation matrix can be uniquely determined using the internal parameters of the local optical camera and the infrared thermal imaging camera, along with the installation device and monitoring distance.

[0078] Assuming the perspective transformation matrix is ​​denoted by H, the above perspective transformation can be described by the following formula:

[0079]

[0080] In the formula, [XY 1] T Let [x, y, 1] be the first coordinate, which is the coordinate of the center of the roller in the optical image; T Let H be the second coordinate, which is the coordinate of the center of the idler roller in the infrared coordinate system; H is a 3*3 matrix.

[0081] S330 determines the actual temperature of the corresponding idler roller based on the infrared temperature at the center of the idler roller.

[0082] Therefore, after determining the coordinates of the center of the corresponding idler roller in the infrared thermal image, the temperature value of the center of the corresponding idler roller can be obtained, and then the actual temperature of the corresponding idler roller can be determined.

[0083] In one implementation, the temperature at the center of the idler roller can be directly used as the actual temperature of the corresponding idler roller, that is, the temperature at the center of the idler roller can represent the temperature of the entire idler roller.

[0084] In another embodiment, after obtaining the infrared temperature at the center of each idler roller, multiple temperature values ​​in the neighborhood containing the center of the idler roller can be statistically analyzed based on the infrared thermal imaging, and the average of these multiple temperature values ​​can be used as the actual temperature of the corresponding idler roller.

[0085] For example, one can select three to five neighborhoods around the center of the idler roller, then calculate the temperature value at each coordinate point within that area, and average the results. This average temperature is then taken as the actual temperature of the entire idler roller. It's understandable that calculating the temperature at more coordinate points improves the accuracy of the temperature readings. Figure 11 As shown, combined with the infrared thermal imaging on the right side of the figure, each idler roller identified in the optical image on the left will display its corresponding actual temperature.

[0086] S140: When it is detected that the actual temperature of an idler roller exceeds the preset temperature threshold, an idler roller overheating alarm is triggered, and the video segment corresponding to the alarm time is extracted from the monitoring image. This video segment is used to obtain the bracket number information where the overheated idler roller is located.

[0087] The set temperature threshold is used to determine whether each idler roller is overheating. For example, if the actual temperature of one or more idler rollers exceeds the threshold, an overheating alarm is triggered. Correspondingly, the inspection robot will extract the video clip at the moment the overheating alarm is triggered from the monitoring images captured by the global optical camera and upload it to the monitoring platform. Since the global optical camera is pre-positioned to capture the idler roller bracket number, it can provide information on the location of the overheated roller, allowing maintenance workers to quickly locate the problem and perform equipment repairs.

[0088] It is understood that the three-light fusion described in this embodiment refers to combining the three-light data obtained from a global optical camera, a local optical camera, and an infrared thermal imaging camera to jointly achieve real-time temperature monitoring of the idler rollers and quickly determine the location of the overheated idler rollers when overheating occurs. This three-light fusion-based idler roller monitoring method significantly reduces labor costs, eliminating the need for manual monitoring at each conveyor belt. Because it enables intelligent real-time temperature monitoring of the idler rollers, it effectively prevents overheating and damage to idler rollers or conveyor belts, thereby improving enterprise production efficiency.

[0089] Please refer to Figure 12 Based on the methods of the above embodiments, this embodiment proposes a roller monitoring device 200 based on three-light fusion. Exemplarily, the roller monitoring device 200 includes:

[0090] The data acquisition module 210 is used to acquire monitoring images containing the idler bracket number and rotating idler, as well as optical images and infrared thermal images of the idler's position at the corresponding time.

[0091] The idler roller recognition module 220 is used to identify each idler roller in the optical image in real time using an idler roller recognition model.

[0092] Temperature calculation module 230 is used to determine the actual temperature of each idler roller based on the infrared thermal imaging.

[0093] The alarm control module 240 is used to trigger an overheat alarm for the idler roller when the actual temperature of the idler roller exceeds a preset temperature threshold, and to extract the video segment corresponding to the alarm time from the monitoring image. The video segment is used to obtain the bracket number information of the overheated idler roller.

[0094] It is understood that the apparatus of this embodiment corresponds to the method of the above embodiments, and the options in the above embodiments are also applicable to this embodiment, so they will not be described again here.

[0095] This application also provides a terminal device, such as a server for background monitoring, exemplary of which the terminal device includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to enable the terminal device to perform the functions of the various modules in the above-described three-light fusion-based idler monitoring method or the above-described three-light fusion-based idler monitoring device.

[0096] This application also provides a readable storage medium for storing the computer program used in the aforementioned terminal device.

[0097] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that, in alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0098] In addition, the functional modules or units in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0099] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a smartphone, personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0100] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for monitoring idler rollers based on three-light fusion, characterized in that, The method is applied to an inspection robot, which is equipped with a global optical camera, a local optical camera, and an infrared thermal imaging camera; the method includes: Acquire monitoring images containing the idler bracket number and rotating idler, captured by the global optical camera; Acquire the optical image of the roller's location at the corresponding moment captured by the local optical camera and the infrared thermal image of the same location at the same moment captured by the infrared thermal imaging camera; Each idler in the optical image is identified in real time using an idler identification model. The identification information of the idler includes the first coordinate of the center of each idler in the optical image coordinate system. The first coordinate of the center of each idler is transformed to the second coordinate in the infrared thermal imaging coordinate system using a perspective transformation algorithm. The temperature value corresponding to each of the second coordinates in the infrared thermal imaging is used as the infrared temperature of the center of the idler at the corresponding position. The actual temperature of the corresponding idler is determined based on the infrared temperature of the center of the idler. When the actual temperature of an idler roller exceeds a preset temperature threshold, an overheating alarm is triggered, and a video clip corresponding to the alarm time is extracted from the monitoring image. The video clip is used to obtain the bracket number information where the overheated idler roller is located. The idler roller recognition model includes a feature extraction network, a pooling transition network, and a feature fusion network. The feature extraction network extracts feature maps from low-level to high-level from the input image containing the idler roller to obtain corresponding feature maps. The pooling transition network performs pooling and fusion processing on the feature maps output by a specified layer in the feature extraction network to obtain an expanded feature map. Then, residual optimization is performed on the expanded feature map to obtain an optimized feature map. The feature fusion network upsamples the optimized feature map and fuses the upsampled feature map with a feature map of the same size output by the feature extraction network to output the idler roller prediction result. The feature extraction network comprises an input convolutional layer, multiple optimized residual structures, and an output convolutional layer connected in sequence. The optimized residual structure includes two branches evenly divided according to the input channels of the input convolutional layer. The outputs of the two branches are connected to a concatenation unit, which is connected to a convolutional output module. Each branch includes a convolutional module, and one branch also includes a bottleneck module connected to the output of the convolutional module. The pooling transition network comprises a spatial pyramid pooling layer and optimized residual structures. The spatial pyramid pooling layer includes a convolutional module and three pooling units with different kernel sizes. The outputs of the convolutional module and the three pooling units are concatenated according to channels. The spatial pyramid pooling layer is used to output an expanded feature map. The feature fusion network includes single-scale or multi-scale feature fusion structures.

2. The roller monitoring method based on three-light fusion according to claim 1, characterized in that, Determining the actual temperature of the corresponding idler roller based on the infrared temperature at the center of the idler roller includes: The infrared temperature at the center of the idler roller is defined as the actual temperature of the corresponding idler roller. Alternatively, multiple temperature values ​​within the neighborhood of the center of the idler roller can be statistically analyzed based on the infrared thermal imaging, and the average of the multiple temperature values ​​can be taken as the actual temperature of the corresponding idler roller.

3. The roller monitoring method based on three-light fusion according to claim 1, characterized in that, The idler roller recognition model is obtained through pre-training and includes: Collect an image dataset containing idler rollers from the conveyor belt system; Clustering algorithms are used to calculate the roller anchor frame parameters in the image dataset. These parameters are used to determine the anchor frame size for model training input. The image dataset is subjected to image size unification and image enhancement processing to obtain a training sample set with the same image size; The constructed convolutional neural network model is iteratively trained using the training sample set until the training loss value meets the preset conditions, at which point training stops, and a trained idler roller recognition model is obtained.

4. A roller monitoring device based on three-light fusion, characterized in that, Applied to inspection robots, the inspection robot is equipped with a global optical camera, a local optical camera, and an infrared thermal imaging camera; the device includes: The data acquisition module is used to acquire monitoring images containing the idler support number and rotating idler captured by the global optical camera; The data acquisition module is also used to acquire the optical image of the location of the roller at the corresponding moment captured by the local optical camera and the infrared thermal image of the same location at the same moment captured by the infrared thermal imaging camera. The idler roller recognition module is used to identify each idler roller in the optical image in real time using an idler roller recognition model; the recognition information of the idler roller includes the first coordinate of the center of each idler roller in the optical image coordinate system; The temperature calculation module is used to transform the first coordinate of the center of each idler roller to the second coordinate in the infrared thermal imaging coordinate system using a perspective transformation algorithm; to take the temperature value corresponding to each second coordinate in the infrared thermal imaging as the infrared temperature of the center of the idler roller at the corresponding position; and to determine the actual temperature of the corresponding idler roller based on the infrared temperature of the center of the idler roller. An alarm control module is used to trigger an overheat alarm for idlers when the actual temperature of an idler exceeds a preset temperature threshold, and to extract a video clip corresponding to the alarm time from the monitoring image. The video clip is used to obtain the bracket number information where the overheated idler is located. The idler roller recognition model includes a feature extraction network, a pooling transition network, and a feature fusion network. The feature extraction network extracts feature maps from low-level to high-level from the input image containing the idler roller to obtain corresponding feature maps. The pooling transition network performs pooling and fusion processing on the feature maps output by a specified layer in the feature extraction network to obtain an expanded feature map. Then, residual optimization is performed on the expanded feature map to obtain an optimized feature map. The feature fusion network upsamples the optimized feature map and fuses the upsampled feature map with a feature map of the same size output by the feature extraction network to output the idler roller prediction result. The feature extraction network comprises an input convolutional layer, multiple optimized residual structures, and an output convolutional layer connected in sequence. The optimized residual structure includes two branches evenly divided according to the input channels of the input convolutional layer. The outputs of the two branches are connected to a concatenation unit, which is connected to a convolutional output module. Each branch includes a convolutional module, and one branch also includes a bottleneck module connected to the output of the convolutional module. The pooling transition network comprises a spatial pyramid pooling layer and optimized residual structures. The spatial pyramid pooling layer includes a convolutional module and three pooling units with different kernel sizes. The outputs of the convolutional module and the three pooling units are concatenated according to channels. The spatial pyramid pooling layer is used to output an expanded feature map. The feature fusion network includes single-scale or multi-scale feature fusion structures.

5. An inspection robot, characterized in that, The inspection robot is equipped with a global optical camera, a local optical camera, an infrared thermal imaging camera, a processor, and a memory; the memory stores a computer program, and the processor is used to execute the computer program to implement the roller monitoring method based on three-light fusion as described in any one of claims 1-3; The global optical camera is used to acquire monitoring images that capture the idler support number and the rotating idler; the local optical camera is used to acquire optical images of the location of the idler at a corresponding moment; and the infrared thermal imaging camera is used to acquire infrared thermal images of the same location at the same moment.

6. A readable storage medium, characterized in that, It stores a computer program, which, when executed on a processor, implements the roller monitoring method based on three-light fusion according to any one of claims 1-3.

Images