Carrier roller jamming detection method and electronic equipment

Abstract

The embodiment of the invention provides a carrier roller jamming detection method and electronic equipment, and relates to the technical field of carrier roller fault detection. A visible light video, a thermal infrared video and a diagnosis demand instruction of a to-be-detected carrier roller are obtained, the thermal infrared video is converted into a thermal infrared pseudo-color video according to a preset temperature interval range, and the visible light video, the thermal infrared pseudo-color video and the diagnosis demand instruction are input into a carrier roller jamming detection model; and obtaining a jamming fault detection conclusion and a maintenance suggestion of the to-be-detected carrier roller. The visible light video and the thermal infrared video are combined, and the rotating state and the high-temperature state of the carrier roller are nursed at the same time, so that the carrier roller jamming detection precision is high, and the false alarm rate and the missing report rate are low.

Description

Technical Field

[0001] The present invention relates to the technical field of roller fault detection, and in particular to a roller jam detection method and electronic equipment. Background Art

[0002] Rollers are important components of belt conveyors. As the length of the belt increases, the number of rollers will also increase proportionally, and the possibility of abnormal failures in a large number of rollers will also increase. Roller jam is a serious component failure. Due to the sliding friction between the roller surface and the belt after jamming, the heat generated increases significantly, and may even cause a fire. Therefore, the detection of roller jam failure is very important.

[0003] In the related art, the detection method combining roller rotation and high temperature coordination mainly obtains the rotation state and temperature state of the current roller, and determines the roller jam fault when the roller rotation abnormality and high temperature abnormality are satisfied at the same time. However, the rotation abnormality and the high temperature abnormality are calculated separately. However, this method has certain limitations in practical applications, because the rotation abnormality and the high temperature abnormality are processed independently, and the correlation between the two is not fully considered, which will lead to inaccurate judgment results. Summary of the invention

[0004] In view of this, an object of an embodiment of the present invention is to provide a roller jam detection method and an electronic device to at least partially improve the above-mentioned problem.

[0005] In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:

[0006] In a first aspect, an embodiment of the present invention provides a method for detecting roller jamming, the method comprising:

[0007] Obtain visible light video, thermal infrared video and diagnostic requirement instructions of the roller to be inspected;

[0008] Converting the thermal infrared video into a thermal infrared pseudo-color video according to a preset temperature range;

[0009] The visible light video, the thermal infrared pseudo-color video and the diagnostic requirement instruction are input into a roller jam detection model to obtain a jam fault detection conclusion and maintenance suggestion for the roller to be detected.

[0010] Optionally, converting the thermal infrared video into a thermal infrared pseudo-color video according to a preset temperature range includes:

[0011] Each frame of the thermal infrared temperature image of the thermal infrared video is truncated according to a preset temperature range to obtain a truncated temperature image;

[0012] Normalizing each of the truncated temperature images to obtain a normalized grayscale image;

[0013] Each of the normalized grayscale images is mapped to a pseudo-color image to obtain a thermal infrared pseudo-color video.

[0014] Optionally, the roller blocking detection model includes a visual encoder, a linear projection layer and a large language model, and the visible light video, the thermal infrared pseudo-color video and the diagnostic requirement instruction are input into the roller blocking detection model to obtain a blocking fault detection conclusion and maintenance suggestion of the roller to be detected, including:

[0015] Inputting the visible light video and the thermal infrared pseudo-color video into the visual encoder respectively to obtain a visible light feature map and a thermal infrared feature map;

[0016] Fusing the visible light feature map and the thermal infrared feature map to obtain a fused feature map;

[0017] Inputting the fused feature map into the linear projection layer to obtain a visual feature vector;

[0018] Input the diagnostic requirement instruction into a word embedding layer to obtain an instruction feature vector;

[0019] After the visual feature vector and the instruction feature vector are fused, they are input into the large language model to obtain the jamming fault diagnosis conclusion and inspection and maintenance suggestions of the roller to be detected.

[0020] Optionally, the roller jam detection model further includes a temporal pooling layer and a spatial pooling layer, and the visible light feature map and the thermal infrared feature map are fused to obtain a fused feature map, including:

[0021] Inputting the visible light feature map into the temporal pooling layer and the spatial pooling layer respectively, performing temporal pooling and spatial pooling respectively, to obtain a visible light temporal feature map and a visible light spatial feature map;

[0022] Inputting the thermal infrared feature map into the temporal pooling layer and the spatial pooling layer respectively, performing temporal pooling and spatial pooling respectively, to obtain a thermal infrared temporal feature map and a thermal infrared spatial feature map;

[0023] The visible light time series feature map, the visible light space feature map, the thermal infrared time series feature map and the thermal infrared space feature map are fused to obtain a fused feature map.

[0024] Optionally, the time series pooling process includes:

[0025] For each time step, the features of all spatial positions of the time step are summed to obtain a total spatial feature vector;

[0026] Dividing the total spatial feature vector by the total number of all spatial positions to obtain a temporal pooling feature map;

[0027] The spatial pooling process includes:

[0028] For each spatial position, sum the features of all time steps of the spatial position to obtain a total time series feature vector;

[0029] The total spatial feature vector is divided by the total number of all time steps to obtain a spatial pooling feature map.

[0030] Optionally, the linear projection layer is a fully connected layer, and the calculation formula of the visual feature vector is:

[0031] z=W·v+b

[0032] Among them, v is the fused feature map, W is the weight matrix of the linear projection layer, b is the bias vector of the linear projection layer, and z is the visual feature vector after projection.

[0033] Optionally, the method further comprises the step of training the roller jam detection model, comprising:

[0034] Obtain visual data sets and instruction data sets;

[0035] Using the visual dataset, based on CLIP pre-trained weights, performing full parameter multi-task fine-tuning learning on the visual encoder;

[0036] Freeze the weights of the visual encoder, use the instruction dataset, fine-tune all parameters of the linear projection layer, and fine-tune the large language model using LoRA;

[0037] Using the instruction dataset, the weights of the visual encoder, the linear projection layer, and the large language model are fine-tuned end-to-end simultaneously; wherein the visual encoder and the linear projection layer perform full parameter fine-tuning, and the large language model performs LoRA fine-tuning.

[0038] Optionally, using the visual dataset to perform full parameter multi-task fine-tuning learning on the visual encoder based on CLIP pre-trained weights includes:

[0039] Pre-training the visual encoder using the CLIP paradigm;

[0040] The pre-trained visual encoder is used as the backbone network, and the task heads of object detection and semantic segmentation are added, and the visual dataset is used for full-parameter multi-task fine-tuning learning.

[0041] Optionally, the loss function of the multi-task fine-tuning learning is:

[0042] L total =γ det (L cls +λ1·L bbox )+γ seg (α·L seg_ce +β·L seg_dice )

[0043] Among them, L total is the total loss function, which consists of target detection loss and segmentation loss. The target detection loss includes classification loss and bounding box loss, and the segmentation loss includes cross entropy loss and Dice loss. cls is the classification loss, used to evaluate the accuracy of target category prediction, L bbox is the bounding box loss, which is used to evaluate the gap between the predicted box and the true box, L seg_ce is the cross entropy loss, used to evaluate the classification accuracy of each pixel, L seg_dice is the Dice loss, which is used to evaluate the overlap between the predicted segmentation area and the true segmentation area, γ det , γ seg is the weight coefficient of the target detection loss and the segmentation loss, λ1, α, β are the weight coefficients of each loss;

[0044]

[0045] Among them, N is the total number of samples, C is the total number of categories, is the true label of the i-th sample belonging to the c-th category, with a value of 0 or 1. is the probability that the i-th sample is predicted to be the c-th class;

[0046]

[0047] Among them, N pos is the number of positive samples, that is, the positive samples corresponding to the real box, is the predicted value of the bounding box, is the true value of the bounding box, It is a smooth L1 loss function, which is used to reduce the adverse effects of large errors on training;

[0048]

[0049] Among them, H is the height of the image, W is the width of the image, C is the total number of categories, and y ijc is the true label of the (i, j)th pixel belonging to the cth class, with a value of 0 or 1, p ijc is the probability that the (i, j)th pixel is predicted to be of the cth class;

[0050]

[0051] Among them, H is the height of the image, W is the width of the image, C is the total number of categories, and y ijc is the true label of the (i, j)th pixel belonging to the cth class, with a value of 0 or 1, p ijc is the probability that the (i, j)th pixel is predicted to be of the cth class.

[0052] In a second aspect, an embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements any of the above-described methods when executing the program.

[0053] A roller jam detection method and electronic device provided in an embodiment of the present invention can achieve high accuracy, low false alarm and low missed alarm rate in roller jam detection by combining visible light video and thermal infrared video and monitoring the rotation state and high temperature state of the roller at the same time. Combined with diagnostic requirement instructions dedicated to roller jam faults and sent into a multimodal large model, the fault detection conclusion and maintenance suggestions of the current roller can be directly output. The input and feature fusion of the visible light / thermal infrared dual-channel video streams can further enhance the fault diagnosis capability. Through the multimodal large model training and fine-tuning strategy, accurate identification of roller jam faults can be achieved and corresponding inspection and maintenance suggestions can be output.

[0054] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0056] Figure 1 A schematic structural block diagram of an electronic device provided by an embodiment of the present invention;

[0057] Figure 2 A schematic flow chart of a method for detecting roller jamming provided by an embodiment of the present invention;

[0058] Figure 3 A schematic structural block diagram of a roller jam detection model provided in an embodiment of the present invention;

[0059] Figure 4Another schematic diagram of a flow chart of a roller jam detection method provided by an embodiment of the present invention;

[0060] Figure 5 Another schematic structural block diagram of a roller jam detection model provided by an embodiment of the present invention;

[0061] Figure 6 A flowchart illustrating the execution of a method for detecting roller jamming provided by an embodiment of the present invention;

[0062] Figure 7 A schematic diagram of a flow chart of training a roller jam detection model provided in an embodiment of the present invention.

[0063] Icons: 100-electronic device; 101-memory; 102-communication interface; 103-processor; 104-bus; 300-roller jam detection model; 310-visual encoder; 320-linear projection layer; 330-large language model; 340-temporal pooling layer; 350-spatial pooling layer. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0065] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0067] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0068] As described in the background technology, in industrial production, the detection of roller jamming failure is very important. The relevant technologies mainly include three methods: roller rotation state care, roller high temperature care, and roller rotation + high temperature care.

[0069] Roller rotation status monitoring: when the belt conveyor is lightly loaded or unloaded, the lack of contact between the belt and the roller will also cause the roller to not rotate. Moreover, when the belt conveyor stops, all rollers are in a state of stopped rotation, resulting in abnormal roller rotation. Therefore, false alarms of jamming are likely to occur if the analysis is based solely on the roller rotation status.

[0070] High temperature care plan for rollers. Since rolling friction turns into sliding friction after the roller is stuck, a large amount of heat will be generated in the contact area between the belt and the roller body. The heat will be transferred to the entire roller area, causing high temperature of the roller. However, there are many non-roller jam faults that may cause high temperature in the bearing area of ​​the roller, such as bearing damage, etc. Moreover, when the belt conveyor transports high-temperature materials, the heat will be transferred to the roller, resulting in abnormal roller temperature. Therefore, simply analyzing the high temperature state of the roller can easily cause false alarms of jams.

[0071] The roller rotation + high temperature care solution mainly obtains the current rotation status and temperature status of the roller, and determines the roller jam failure when the roller rotation abnormality and high temperature abnormality are met. However, the rotation abnormality and high temperature abnormality are calculated separately, and the correlation between the two is not fully considered, which will lead to inaccurate judgment results.

[0072] Based on the above situation, an embodiment of the present invention provides a roller jam detection method and electronic device. By combining visible light video and thermal infrared video and using a roller jam detection model to simultaneously monitor the roller rotation state and the roller high temperature state, the roller jam detection accuracy can be high, with low false alarms and low missed alarm rates.

[0073] To implement the process steps and functions of each example of the present invention, please refer to Figure 1 , Figure 1A schematic structural block diagram of an electronic device provided by an embodiment of the present invention. The electronic device 100 includes a memory 101 and a processor 103, and the memory 101 and the processor 103 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses 104 or signal lines. The memory 101 can be used to store software programs and modules, and the processor 103 executes the software programs and modules stored in the memory 101 to perform various functional applications and data processing.

[0074] The electronic device 100 may be, but is not limited to, a personal computer (PC), a server, a distributed computer, etc. It is understandable that the electronic device 100 is not limited to a physical server, but may also be a virtual machine on a physical server, a virtual machine built on a cloud platform, etc., which can provide the same functions as the server or virtual machine. The operating system of the electronic device 100 may be, but is not limited to, a Windows system, a Linux system, etc.

[0075] Among them, the memory 101 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc.

[0076] The communication connection between the electronic device 100 and an external device is achieved through at least one communication interface 102 (which can be wired or wireless).

[0077] The processor 103 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the embodiment of the present invention can be completed by the hardware integrated logic circuit in the processor 103 or the instructions in the form of software. The processor 103 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0078] Understandably, Figure 1 The structure shown is for illustration only. The electronic device 100 may also include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0079] The following is an exemplary description of the roller jam detection method provided by the present invention. Specifically, Figure 2 For, see Figure 2 , the execution subject of this method can be the above Figure 1 The electronic device 100 shown in FIG. 1 includes: Figure 2 The following steps are described:

[0080] S210: Obtaining visible light video, thermal infrared video and diagnostic requirement instructions of the roller to be inspected.

[0081] The visible light video and thermal infrared video can be collected by an inspection robot. The inspection robot finds the roller according to the serial number of the roller that needs to be inspected, and collects the visible light video and thermal infrared video of the roller. The inspection robot includes a visible light camera and a thermal infrared camera. The visible light camera is used to capture the appearance image of the roller, and the thermal infrared camera is used to detect the temperature distribution of the roller.

[0082] The diagnostic requirement instruction refers to the need to diagnose the roller, for example, to detect whether the temperature of the roller is abnormal, to check the physical damage of the roller, to diagnose the alignment of the roller, to detect whether the roller is stuck, etc.

[0083] S220: Convert the thermal infrared video into a thermal infrared pseudo-color video according to a preset temperature range.

[0084] The process of converting thermal infrared video into pseudo-color video is to intuitively display the temperature information in the thermal infrared image through color coding, where each temperature value corresponds to a color. Pseudo-color video can provide richer information for the visual encoder, helping the model to better understand and analyze thermal infrared data.

[0085] S230: Input the visible light video, the thermal infrared pseudo-color video and the diagnostic requirement instruction into the roller jam detection model to obtain the jam fault detection conclusion and maintenance suggestion of the roller to be detected.

[0086] The roller jam detection model combines visible light video and thermal infrared pseudo-color video to extract features and conduct comprehensive analysis to identify whether the roller is jammed. The jam fault detection conclusion of the roller to be detected can be obtained through the rotation state (such as stalling, slow rotation) in the visible light video and the temperature anomaly (such as local overheating) in the thermal infrared pseudo-color video. The conclusion can be whether the roller is jammed and the fault type (such as slight jamming, severe jamming) is classified. According to the fault type and severity, detailed maintenance suggestions are automatically generated, such as: slight jamming: lubrication treatment and regular inspection are recommended; severe jamming: stop the relevant equipment immediately and arrange emergency maintenance or replace the roller.

[0087] This method can obtain roller status information from multiple dimensions by combining visible light video and thermal infrared pseudo-color video. Visible light video provides appearance characteristics and rotation status, while thermal infrared pseudo-color video displays temperature distribution to accurately determine the jamming phenomenon of the roller.

[0088] There are many ways to convert thermal infrared video into thermal infrared pseudo-color video. In one possible implementation, the conversion can be achieved by using a lookup table. The above step S220 may include the following steps:

[0089] S221: truncating each frame of the thermal infrared temperature image of the thermal infrared video according to a preset temperature range to obtain a truncated temperature image.

[0090] First, each frame of the input thermal infrared temperature image is truncated according to the preset temperature range. Assuming the pixel value of the original temperature image is T(x, y), the pixel value of the truncated temperature image is:

[0091] T clip (x,y)=min(max(T(x,y),T min ),T max )

[0092] Among them, T min and T max They are the lower and upper limits of the preset temperature range respectively.

[0093] S222: normalize each truncated temperature image to obtain a normalized grayscale image.

[0094] The truncated temperature image can be normalized to the range of 0 to 255, and the normalized image pixel value is obtained as follows:

[0095]

[0096] Among them, T min and T max are respectively the lower limit and upper limit of the above preset temperature range.

[0097] S223: Map each normalized grayscale image to a pseudo-color image to obtain a thermal infrared pseudo-color video.

[0098] Mapping the normalized grayscale image to a pseudo-color image can be achieved through a lookup table (LUT), which can be done by following the steps below:

[0099] S2231: Create a LUT. A LUT is a data structure that maps input values ​​(usually grayscale values ​​in image processing) to output values ​​(color values, usually RGB values). For grayscale images, the LUT usually contains 256 entries (if the grayscale value range is 0 to 255). Each entry stores an RGB color value. This table can be pre-calculated and stored to speed up the mapping process.

[0100] S2232: Define mapping rules, in the LUT, define the color corresponding to each gray value. For example, you can define that low gray values ​​are mapped to blue, medium gray values ​​are mapped to green, and high gray values ​​are mapped to red, forming a simple rainbow mapping scheme.

[0101] S2233: Apply LUT, for each pixel in the grayscale image, use its grayscale value as an index to find the corresponding RGB color value in the LUT. This process is very efficient because the lookup is a direct index operation.

[0102] S2234: Generate a pseudo color image, assign the found RGB color values ​​to the corresponding pixels, and generate a new color image. In this way, the original grayscale image is converted into a pseudo color image, which enhances the visibility of certain features in the image.

[0103] LUT provides a fast and effective way to convert grayscale images into pseudo-color images, making different areas or features in the image more obvious and easier to analyze and identify.

[0104] In one possible implementation, see Figure 3 , Figure 4The roller blocking detection model 300 may include a visual encoder 310, a linear projection layer 320, and a large language model 330. The above step S230 may include the following steps:

[0105] S231: Input the visible light video and the thermal infrared pseudo-color video into the visual encoder respectively to obtain a visible light feature map and a thermal infrared feature map.

[0106] Optionally, the vision encoder can use the ViT-L / 14 image large model, which is a variant of the VisionTransformer model, where "L" stands for the Large model and "14" refers to the size of each image patch being 14x14 pixels.

[0107] The collected visible light video is input into the visual encoder to extract appearance features, rotation state, etc., and the thermal infrared pseudo-color video is input into the visual encoder to extract temperature distribution features. The temperature distribution features may include local overheating areas and temperature gradient changes.

[0108] S232: Fusing the visible light feature map and the thermal infrared feature map to obtain a fused feature map.

[0109] The fusion method can use Concat or Add. Concat directly connects the visible light feature map and the thermal infrared feature map, and Add adds the visible light feature map and the thermal infrared feature map.

[0110] Through fusion technology, the visible light feature map and the thermal infrared feature map are merged into a fused feature map. By combining the advantages of the two modalities, the accuracy of subsequent analysis can be improved.

[0111] S233: Input the fused feature map into the linear projection layer to obtain a visual feature vector.

[0112] The linear projection layer maps the high-dimensional space in the fused feature map to a low-dimensional space to generate a compact visual feature vector, which not only reduces the computational complexity but also retains the key feature information.

[0113] Optionally, the linear projection layer may be a fully connected layer, and the calculation formula of the visual feature vector is:

[0114] z=W·v+b

[0115] Among them, v is the fused feature map, W is the weight matrix of the linear projection layer, b is the bias vector of the linear projection layer, and z is the projected visual feature vector.

[0116] S234: Input the diagnostic requirement instruction into the word embedding layer to obtain an instruction feature vector.

[0117] The diagnostic requirement instructions are converted into numerical representations, namely instruction feature vectors, through the word embedding layer. The word embedding layer can capture the semantic information in the instructions and ensure that the model understands the specific diagnostic tasks.

[0118] Optionally, the tokenizer of the word embedding layer can be the transformer library of Hugging Face, which contains a vocabulary and a tokenization algorithm that can convert text into a corresponding token id sequence.

[0119] S235: After fusing the visual feature vector and the instruction feature vector, the vector is input into the large language model to obtain the jamming fault diagnosis conclusion and inspection and maintenance suggestions of the roller to be inspected.

[0120] The visual feature vector and the instruction feature vector are further integrated to form a comprehensive feature vector, which ensures that the model fully considers the dual information of visual data and diagnostic tasks when performing the final analysis. By inputting the large language model, the jamming fault diagnosis conclusion and inspection and maintenance suggestions of the roller to be detected can be output.

[0121] Optionally, the large language model may be Llama 3.1-8B. LLaMA (Large Language Model MetaAI) is a large language model series launched by Meta, and Llama 3.1-8B is the third generation version of the LLaMA series model, based on the classic Transformer architecture, with a parameter volume of 8B (8 billion parameters).

[0122] In order to accelerate the model inference process, capture higher-level abstract features, and effectively capture dynamic changes between video frames, in a possible implementation, see Figure 5 The roller blocking detection model 300 may further include a temporal pooling layer 340 and a spatial pooling layer 350. The above step S232 may include the following steps:

[0123] S2321: Input the visible light feature map into the temporal pooling layer and the spatial pooling layer respectively, perform temporal pooling and spatial pooling respectively, and obtain a visible light temporal feature map and a visible light spatial feature map.

[0124] S2322: Input the thermal infrared feature map into the temporal pooling layer and the spatial pooling layer respectively, perform temporal pooling and spatial pooling respectively, and obtain a thermal infrared temporal feature map and a thermal infrared spatial feature map.

[0125] S2323: Fusing the visible light time series feature map, the visible light space feature map, the thermal infrared time series feature map, and the thermal infrared space feature map to obtain a fused feature map.

[0126] The visible light feature map is input into the temporal pooling layer and the spatial pooling layer respectively. After the temporal pooling operation, the visible light temporal feature map is obtained. After the spatial pooling operation, the visible light spatial feature map is obtained.

[0127] The thermal infrared feature map is also input into the temporal pooling layer and the spatial pooling layer. After the temporal pooling operation, the thermal infrared temporal feature map is obtained. After the spatial pooling operation, the thermal infrared spatial feature map is obtained.

[0128] The above four feature maps (visible light temporal feature map, visible light spatial feature map, thermal infrared temporal feature map, thermal infrared spatial feature map) are fused. The fusion method can also be Concat or Add to finally generate a fused feature map.

[0129] In a possible implementation, the above-mentioned timing pooling process may include:

[0130] For each time step, the features of all spatial positions of the time step are summed to obtain the total spatial feature vector. The total spatial feature vector is divided by the total number of all spatial positions to obtain the temporal pooling feature map.

[0131] The process can be expressed by the following formula:

[0132]

[0133] For each time step t, sum the features X(t, s, :) of all spatial positions s from 1 to S, that is, sum X(t, s, :) for s from 1 to S. Divide the sum by the total number of all spatial positions S to get the average value. The final T(t) is a vector that represents the average value of all spatial position features at time step t, that is, the temporal pooling feature map after temporal pooling.

[0134] In a possible implementation, the spatial pooling process may include:

[0135] For each spatial position, the features of all time steps of the spatial position are summed to obtain the total temporal feature vector; the total spatial feature vector is divided by the total number of all time steps to obtain the spatial pooling feature map.

[0136] The process can be expressed by the following formula:

[0137]

[0138] For each spatial position s, sum the features X(t, s, :) of all time steps t from 1 to T, where X(t, s, :) represents the feature vector at time step t and spatial position s. Divide the sum by the total number of time steps T to get the average value. The final S(s) is a vector that represents the average value of all time step features at spatial position s, that is, the spatial pooling feature map after spatial pooling.

[0139] See also Figure 6 , a possible implementation method of the roller blocking detection method is described. First, the inspection robot pan / tilt collects the visible light video and thermal infrared video of the current serial number roller according to the registration information, and converts the thermal infrared video into a pseudo-color video according to the preset temperature range. Subsequently, the visible light video and thermal infrared pseudo-color video are respectively sent to the visual encoder ViT-L to output the visible light feature map and thermal infrared feature map; the visible light feature map and thermal infrared feature map are respectively sent to the spatial pooling layer and the temporal pooling layer to output the respective temporal feature maps and spatial feature maps corresponding to the visible light and thermal infrared. The temporal feature map and spatial feature map corresponding to the visible light and thermal infrared are concat-fused and sent to the linear projection layer to output the visual token vector; finally, the roller blocking diagnosis requirement instruction (prompt) is output through the word embedding layer to output the instruction token vector, which is concat-fused with the visual token vector and sent to the language large model LLM to output the blocking fault diagnosis conclusion and maintenance suggestions.

[0140] Furthermore, the method also includes the step of training a roller jam detection model, see Figure 7 , the steps include:

[0141] S240: Acquire a visual data set and an instruction data set.

[0142] Among them, the visual data set includes visible light video and infrared pseudo-color video of roller jam scene, and the labels are visible light roller detection frame, visible light roller segmentation mask, thermal infrared roller detection frame, thermal infrared roller segmentation mask, and thermal infrared belt segmentation mask.

[0143] The instruction dataset includes diagnostic requirement instructions, visible light video, and infrared pseudo-color video. The labels are diagnostic conclusions and inspection and maintenance suggestions.

[0144] S250: Use visual datasets to perform full-parameter multi-task fine-tuning of visual encoders based on CLIP pre-trained weights.

[0145] Multi-task fine-tuning learning based on CLIP pre-trained weights can better extract the visual features of the belt conveyor scene.

[0146] Optionally, step S250 may include the following steps:

[0147] S251: Pre-training visual encoders using the CLIP paradigm.

[0148] S252: Use the pre-trained visual encoder as the backbone network, add object detection and semantic segmentation task heads, and use the visual dataset for full parameter multi-task fine-tuning learning.

[0149] The visual encoder is first pre-trained based on the CLIP paradigm. Then the pre-trained visual encoder is used as the backbone network, and the task heads of target detection and semantic segmentation are added. The visible light / thermal infrared data of the belt conveyor scene are fine-tuned with full parameters and multi-task learning, so that the visual encoder can extract more robust and effective features for key targets such as rollers and belts.

[0150] Among them, Faster R-CNN can be selected as the target detection head, the output feature map of the visual encoder is combined with the RPN network to generate candidate regions, and then target detection is performed through ROIPooling and classification regression network; SegFormer can be selected as the semantic segmentation head to aggregate the multi-level features extracted by the visual encoder and predict the category of each pixel.

[0151] Optionally, the loss function of the multi-task fine-tuning learning is:

[0152] L total =γ det (L cls +λ1·L bbox )+γ seg (α·L seg_ce +β·L seg_dice )

[0153] Among them, L total is the total loss function, which consists of target detection loss and segmentation loss. The target detection loss includes classification loss and bounding box loss, and the segmentation loss includes cross entropy loss and Dice loss. cls is the classification loss, used to evaluate the accuracy of target category prediction, L bbox is the bounding box loss, which is used to evaluate the gap between the predicted box and the true box, L seg_ce is the cross entropy loss, used to evaluate the classification accuracy of each pixel, L seg_dice is the Dice loss, which is used to evaluate the overlap between the predicted segmentation area and the true segmentation area, γ det , γ seg is the weight coefficient of the target detection loss and the segmentation loss, and λ1, α, β are the weight coefficients of each loss.

[0154]

[0155] Among them, N is the total number of samples, C is the total number of categories, is the true label of the i-th sample belonging to the c-th category, with a value of 0 or 1. is the probability that the i-th sample is predicted to be of the c-th class.

[0156]

[0157] Among them, N pos is the number of positive samples, that is, the positive samples corresponding to the real box, is the predicted value of the bounding box, is the true value of the bounding box, It is a smooth L1 loss function, which is used to reduce the adverse effects of large errors on training.

[0158]

[0159] Among them, H is the height of the image, W is the width of the image, C is the total number of categories, and y ijc is the true label of the (i, j)th pixel belonging to the cth class, with a value of 0 or 1, p ijc is the probability that the (i, j)th pixel is predicted to be of the cth class.

[0160]

[0161] Among them, H is the height of the image, W is the width of the image, C is the total number of categories, and y ijc is the true label of the (i, j)th pixel belonging to the cth class, with a value of 0 or 1, p ijc is the probability that the (i, j)th pixel is predicted to be of the cth class.

[0162] S260: Freeze the weights of the visual encoder, use the instruction dataset, fine-tune all parameters of the linear projection layer, and fine-tune the large language model with LoRA.

[0163] After freezing the weights of the visual encoder, the LoRA method is used to improve the efficiency of training fine-tuning. Based on the constructed "diagnostic requirement instructions" and "diagnostic conclusion + maintenance suggestions" question-answer data sets, some weights of the large language model are fine-tuned. Among them, LoRA is an efficient method for fine-tuning large language models. By adding low-rank matrices to some weights of the model, the number of parameters and the amount of calculation required for fine-tuning are reduced.

[0164] In LoRA fine-tuning, cross entropy loss is used to measure the performance of the model. For an input sequence x = (x1, x2, ..., x T), that is, the diagnostic requirement instruction token vector after the word embedding layer, and the corresponding target output is y = (y1, y2, ..., y T ), that is, the vector of "diagnosis conclusion + maintenance suggestion", the cross entropy loss function is:

[0165]

[0166] Where T is the total length of the input sequence (the number of time steps), V is the vocabulary size of the output, which represents the total number of all possible words, and y t,i is the true distribution of the target output, one-hot encoded, indicating that the true label at the t-th time step is the i-th word in the vocabulary. t,i |x1,x2,...,x t-1 ; θ, A, B) is the large language model that predicts the output y at the t-th time step given the input sequence x1, x2, …, xt-1 t,i The probability of x1,x2,...,x t-1 is the input sequence of the large language model, which represents the input before the current time step. θ is the frozen model weight, which comes from the pre-trained large language model. In LoRA fine-tuning, these weights are fixed and will not be updated. A and B are low-rank matrices introduced by LoRA.

[0167] Through LoRA fine-tuning, we focused on improving the large language model's recognition and understanding of the characteristics of roller jam failures.

[0168] S270: Use the instruction dataset to fine-tune the weights of the visual encoder, linear projection layer, and large language model end-to-end simultaneously; among them, the visual encoder and linear projection layer perform full parameter fine-tuning, and the large language model performs LoRA fine-tuning.

[0169] After fine-tuning the visual encoder, linear projection layer, and large language model, the instruction dataset is used to train the various parts of the roller jam detection model together. The visual encoder and linear projection layer are still fine-tuned in all parameters, and the large language model is still fine-tuned using LoRA.

[0170] In summary, a roller jam detection method and electronic device provided in an embodiment of the present invention can achieve high accuracy, low false alarm, and low missed alarm rate in roller jam detection by combining visible light video and thermal infrared video and monitoring the rotation state and high temperature state of the roller at the same time. Combined with diagnostic requirement instructions dedicated to roller jam faults, they are sent into a multimodal large model to directly output the fault detection conclusion and maintenance suggestions for the current roller. The input and feature fusion of the visible light / thermal infrared dual-channel video streams can further enhance the fault diagnosis capability. Through the multimodal large model training and fine-tuning strategy, accurate identification of roller jam faults can be achieved and corresponding inspection and maintenance suggestions can be output.

[0171] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and a module, a program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0172] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0173] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a computer-readable storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0174] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0175] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A method for detecting roller jamming, characterized in that: The method comprises: Obtain visible light video, thermal infrared video and diagnostic requirement instructions of the roller to be inspected; Converting the thermal infrared video into a thermal infrared pseudo-color video according to a preset temperature range; The visible light video, the thermal infrared pseudo-color video and the diagnostic requirement instruction are input into a roller jam detection model to obtain a jam fault detection conclusion and maintenance suggestion for the roller to be detected.

2. The method according to claim 1, characterized in that The step of converting the thermal infrared video into a thermal infrared pseudo-color video according to a preset temperature range includes: Each frame of the thermal infrared temperature image of the thermal infrared video is truncated according to a preset temperature range to obtain a truncated temperature image; Normalizing each of the truncated temperature images to obtain a normalized grayscale image; Each of the normalized grayscale images is mapped to a pseudo-color image to obtain a thermal infrared pseudo-color video.

3. The method according to claim 1, characterized in that The roller blocking detection model includes a visual encoder, a linear projection layer and a large language model. The visible light video, the thermal infrared pseudo-color video and the diagnostic requirement instruction are input into the roller blocking detection model to obtain the blocking fault detection conclusion and maintenance suggestion of the roller to be detected, including: Inputting the visible light video and the thermal infrared pseudo-color video into the visual encoder respectively to obtain a visible light feature map and a thermal infrared feature map; Fusing the visible light feature map and the thermal infrared feature map to obtain a fused feature map; Inputting the fused feature map into the linear projection layer to obtain a visual feature vector; Input the diagnostic requirement instruction into a word embedding layer to obtain an instruction feature vector; After the visual feature vector and the instruction feature vector are fused, they are input into the large language model to obtain the jamming fault diagnosis conclusion and inspection and maintenance suggestions of the roller to be detected.

4. The method according to claim 3, characterized in that The roller blocking detection model further includes a temporal pooling layer and a spatial pooling layer, and the visible light feature map and the thermal infrared feature map are fused to obtain a fused feature map, including: Inputting the visible light feature map into the temporal pooling layer and the spatial pooling layer respectively, performing temporal pooling and spatial pooling respectively, to obtain a visible light temporal feature map and a visible light spatial feature map; Inputting the thermal infrared feature map into the temporal pooling layer and the spatial pooling layer respectively, performing temporal pooling and spatial pooling respectively, to obtain a thermal infrared temporal feature map and a thermal infrared spatial feature map; The visible light time series feature map, the visible light space feature map, the thermal infrared time series feature map and the thermal infrared space feature map are fused to obtain a fused feature map.

5. The method according to claim 4, characterized in that The process of timing pooling includes: For each time step, the features of all spatial positions of the time step are summed to obtain a total spatial feature vector; Dividing the total spatial feature vector by the total number of all spatial positions to obtain a temporal pooling feature map; The spatial pooling process includes: For each spatial position, sum the features of all time steps of the spatial position to obtain a total time series feature vector; The total spatial feature vector is divided by the total number of all time steps to obtain a spatial pooling feature map.

6. The method according to claim 3, characterized in that The linear projection layer is a fully connected layer, and the calculation formula of the visual feature vector is: z=W·v+b Among them, v is the fused feature map, W is the weight matrix of the linear projection layer, b is the bias vector of the linear projection layer, and z is the projected visual feature vector.

7. The method according to claim 3, characterized in that The method further includes the step of training the roller jam detection model, including: Obtain visual data sets and instruction data sets; Using the visual dataset, based on CLIP pre-trained weights, performing full parameter multi-task fine-tuning learning on the visual encoder; Freeze the weights of the visual encoder, use the instruction dataset, fine-tune all parameters of the linear projection layer, and fine-tune the large language model using LoRA; Using the instruction dataset, the weights of the visual encoder, the linear projection layer, and the large language model are fine-tuned end-to-end simultaneously; wherein the visual encoder and the linear projection layer perform full parameter fine-tuning, and the large language model performs LoRA fine-tuning.

8. The method according to claim 7, characterized in that The method of using the visual dataset to perform full parameter multi-task fine-tuning learning on the visual encoder based on CLIP pre-trained weights includes: Pre-training the visual encoder using the CLIP paradigm; The pre-trained visual encoder is used as the backbone network, and the task heads of object detection and semantic segmentation are added, and the visual dataset is used for full-parameter multi-task fine-tuning learning.

9. The method according to claim 8, characterized in that The loss function of the multi-task fine-tuning learning is: L total =c det (L cls +λ1·L bbox )+c seg (a·L seg_ce +β·L seg_dice ) Among them, L total is the total loss function, which consists of target detection loss and segmentation loss. The target detection loss includes classification loss and bounding box loss, and the segmentation loss includes cross entropy loss and Dice loss. cls is the classification loss, used to evaluate the accuracy of target category prediction, L bbox is the bounding box loss, which is used to evaluate the gap between the predicted box and the true box, L seg_ce is the cross entropy loss, used to evaluate the classification accuracy of each pixel, L seg_dice is the Dice loss, which is used to evaluate the overlap between the predicted segmentation area and the true segmentation area, γ det , γ seg is the weight coefficient of the target detection loss and the segmentation loss, λ1, α, β are the weight coefficients of each loss; Among them, N is the total number of samples, C is the total number of categories, is the true label of the i-th sample belonging to the c-th category, with a value of 0 or 1. is the probability that the i-th sample is predicted to be the c-th class; Among them, N pos is the number of positive samples, that is, the positive samples corresponding to the real box, is the predicted value of the bounding box, is the true value of the bounding box, It is a smooth L1 loss function, which is used to reduce the adverse effects of large errors on training; Among them, H is the height of the image, W is the width of the image, C is the total number of categories, and y ijc is the true label of the (i, j)th pixel belonging to the cth class, with a value of 0 or 1, p ijc is the probability that the (i, j)th pixel is predicted to be of the cth class; Among them, H is the height of the image, W is the width of the image, C is the total number of categories, and y ijc is the true label of the (i, j)th pixel belonging to the cth class, with a value of 0 or 1, p ijc is the probability that the (i, j)th pixel is predicted to be of the cth class.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 9 is implemented.

Images