Start-stop machine monitoring method and system based on multi-modal data fusion

Abstract

The application provides a start-stop machine monitoring method and system based on multi-modal data fusion. A gray weighted straight line fitting method is used to obtain a first image start-stop machine result, and a video classification model is used for processing to obtain a second image start-stop machine result. An audio classification model is used to process audio information to obtain an audio start-stop machine result. A pulse signal is used to obtain speed information, and the speed information is used to obtain a speed start-stop machine result. The multiple start-stop machine results are fused according to weights to obtain a target device start-stop machine result. In the scheme, the start-stop machine result of the device is determined by using multi-modal data fusion, the defect that the accuracy of a single algorithm is not high is reduced, the monitoring is more timely and accurate, and predictive maintenance is facilitated.

Description

Technical Field

[0001] This invention relates to the field of automatic monitoring technology, and more specifically, to a start-up and shutdown monitoring method and system based on multimodal data fusion. Background Technology

[0002] Currently, monitoring solutions for the start-up and shutdown status of belt conveyors mainly rely on sensors such as temperature, vibration, and photoelectric sensors. By installing specific sensors at designated locations, the sensor signals are collected and analyzed to determine the operating status of the belt conveyor.

[0003] In one solution, temperature sensors are installed on key components of the belt conveyor (motor, rollers, transmission system, etc.) to monitor temperature. When the temperature of a key component exceeds a set threshold, the conveyor is considered to be running. In another solution, vibration sensors are installed on key components (rollers, motor, etc.) to collect effective acceleration values ​​over a period of time. These values ​​are clustered into two groups, and the average of the two groups is used as the start-up threshold. When the effective acceleration value exceeds the start-up threshold, the equipment is considered to be running; otherwise, it is stopped. Finally, current sensors are installed at the motor to monitor current fluctuations. If the current change exceeds a set threshold, the conveyor is considered to be running.

[0004] However, all the existing monitoring solutions mentioned above have certain drawbacks. Specifically, in temperature sensor-based monitoring solutions, when the conveyor belt is first started, the temperature takes some time to rise and reach the specified threshold; similarly, when it stops, the temperature does not immediately drop below the threshold. Therefore, judging the start-up and shutdown of the equipment by temperature has a certain delay. Moreover, temperature sensors are also affected by the ambient temperature; different seasons and regions will affect the sensitivity of the temperature sensors. If the conveyor belt runs at a relatively slow speed, the temperature change may not be significant, making it difficult to capture the equipment's operating status in a timely and accurate manner.

[0005] In vibration sensor-based monitoring solutions, vibration sensors are easily affected by other equipment and the environment. For belt conveyors, the corridor usually contains more than one belt conveyor. When one belt conveyor starts and another stops, the transmission of vibration can interfere with the vibration sensor signal, resulting in false signals. Moreover, the threshold value of the effective value of the vibration signal is difficult to determine due to different operating conditions, and the operating status of equipment at the threshold boundary cannot be accurately identified.

[0006] In current sensor-based monitoring solutions, the current sensor only monitors the current of the motor equipment and cannot directly reflect the actual movement of the belt conveyor. When the belt slips, the belt will not move significantly, but the motor will still run normally. Moreover, when the corresponding equipment is damaged, the motor may run idle, causing the current sensor to fail to detect significant changes in current, thus providing an incorrect equipment operating status.

[0007] In conclusion, current mainstream methods for monitoring the operating status of belt conveyors cannot simultaneously guarantee timeliness and accuracy, and still have some objective limitations and defects. Summary of the Invention

[0008] The purpose of this invention is to provide a start-up and shutdown monitoring method and system based on multimodal data fusion, so as to improve the timeliness and accuracy of monitoring and facilitate predictive maintenance.

[0009] In a first aspect, the present invention provides a start-up and shutdown monitoring method based on multimodal data fusion, the method comprising: Image acquisition is performed on the target device. The acquired image is fitted with a gray-scale weighted straight line to obtain the first image start-up and shutdown result. The acquired image is then processed using a pre-trained video classification model to obtain the second image start-up and shutdown result. The audio information of the target device is collected, and the audio information is processed using a pre-trained audio classification model to obtain the audio start / stop result; The pulse signal of the target device is collected, the rotation speed information is obtained based on the pulse signal, and the rotation speed start-up and shutdown result is obtained based on the rotation speed information; The first image start-up / stop result, the second image start-up / stop result, the audio start-up / stop result, and the speed start-up / stop result are fused according to their weights to obtain the start-up / stop result of the target device.

[0010] In an optional implementation, the image comprises a series of consecutive frames; The step of obtaining the start / stop result of the first image by using gray-level weighted linear fitting on the acquired image includes: For the current frame image, convert the current frame image and the previous frame image into grayscale images, and take the absolute value of the difference between the grayscale values ​​at the corresponding positions to obtain the difference grayscale image; For each pixel coordinate point in the differential grayscale image, determine the polar coordinate system parameters corresponding to the straight line passing through that pixel coordinate point; For each set of polar coordinate system parameters, the voting result and the number of votes are obtained based on the number of times determined by the polar coordinate system parameters; Determine the target polar coordinate system parameter with the largest voting result, and obtain the number of votes for the target polar coordinate system parameter; Based on the voting results and the number of votes for the target polar coordinate system parameters, the first image start / stop result is obtained.

[0011] In an optional implementation, the step of obtaining the voting result and the number of votes for each set of polar coordinate system parameters based on the number determined by the polar coordinate system parameters includes: For each set of polar coordinate system parameters, once the polar coordinate system parameters are determined, the gray value of the pixel coordinate point is superimposed on its voting result, and 1 is added to its vote count, until all pixel coordinate points are traversed, and the final voting result and vote count of the polar coordinate system parameters are obtained.

[0012] In an optional implementation, the step of obtaining the first image start / stop result based on the voting results and the number of votes according to the target polar coordinate system parameters includes: The average gray value of the participants in the line fitting vote is calculated based on the voting results and the number of votes for the target polar coordinate system parameters. The average gray value is compared with a preset threshold to obtain the start / stop result; The start / stop result of the first image is obtained based on the start / stop results corresponding to multiple frames of images within the sliding window.

[0013] In an optional implementation, the image comprises a series of consecutive frames; The step of processing the acquired images using a pre-trained video classification model to obtain the second image start / stop result includes: The current frame image and its previous multiple frames are cropped separately, and the cropped multiple frames are stitched together to form a video clip. The video segment is imported into a pre-trained video classification model for processing to obtain the start / stop result corresponding to the current frame image. By combining the start-up and shutdown results corresponding to multiple frames within the sliding window, the start-up and shutdown results of the second image are obtained.

[0014] In an optional implementation, the step of processing the audio information using a pre-trained audio classification model to obtain the audio start / stop result includes: The audio information is sliced ​​to obtain multiple audio segments, and Mel-spectral features are extracted from each audio segment. The Mel-spectral features of each audio segment are imported into a pre-trained audio classification model for processing to obtain the start / stop results corresponding to the audio information. The audio start / stop result is obtained by combining the start / stop results corresponding to multiple audio segments within the sliding window.

[0015] In an optional implementation, the steps of acquiring the pulse signal of the target device, obtaining rotational speed information based on the pulse signal, and obtaining the rotational speed start-up and shutdown result based on the rotational speed information include: Acquire the pulse signal of the target device, and determine the number of sampling clock cycles that pass within a single pulse cycle based on the pulse signal; The rotational speed is calculated based on the number of cycles and the sampling frequency, assuming the target device is rotating at a constant speed. When the number of cycles indicates that the target device is at a non-constant rotation speed, the cycle of the current pulse signal is obtained based on the time difference between two consecutive pulse signals, and the average value of the cycles of multiple pulse signals within a set window is calculated. The rotation speed is then calculated based on the average value of the cycles. The rotational speed is compared with the preset rotational speed to determine the start-stop result corresponding to the rotational speed information; By combining the start-stop results corresponding to multiple speed information within the sliding window, the speed start-stop result is obtained.

[0016] In an optional implementation, the step of fusing the first image start-up / stop result, the second image start-up / stop result, the audio start-up / stop result, and the speed start-up / stop result according to weights to obtain the start-up / stop result of the target device includes: Reinforcement learning is used to dynamically adjust the weights corresponding to each modality of data; The first image start-stop result, the second image start-stop result, the audio start-stop result, and the speed start-stop result are fused according to the weights of the corresponding modal data to obtain the start-stop result of the target device.

[0017] In an optional implementation, the step of dynamically adjusting the weights corresponding to each modality data using reinforcement learning includes: Construct multiple intelligent agents, each of which is responsible for deciding the weights corresponding to each modal data. In each iteration, each agent learns a policy based on a reinforcement learning network, adjusts its actions based on the Q-value function and the current state to determine the state of the next iteration, wherein the current state is the start-up and shutdown result corresponding to each modality data, and the action is the adjustment action of the weight corresponding to each modality data. Repeat the iterations multiple times until the reward function is maximized under the weights corresponding to each modality data. The reward function is determined based on the difference between the start-up / shutdown predicted state and the actual state.

[0018] Secondly, the present invention provides a start-up and shutdown monitoring system based on multimodal data fusion, the system comprising: The image processing module is used to acquire images of the target device, obtain the first image start-up and shutdown result by using gray-scale weighted linear fitting on the acquired image, and process the acquired image using a pre-trained video classification model to obtain the second image start-up and shutdown result. The audio processing module is used to collect audio information from the target device, process the audio information using a pre-trained audio classification model, and obtain audio start / stop results. The speed processing module is used to acquire the pulse signal of the target device, obtain speed information based on the pulse signal, and obtain the speed start-up and shutdown result based on the speed information; The fusion monitoring module is used to fuse the first image start-up and shutdown results, the second image start-up and shutdown results, the audio start-up and shutdown results, and the speed start-up and shutdown results according to weights to obtain the start-up and shutdown results of the target device.

[0019] This invention provides a start-stop monitoring method and system based on multimodal data fusion. The method involves acquiring image, audio, and pulse signals from a target device. The acquired images are fitted using a grayscale weighted linear model to obtain a first image start-stop result. A pre-trained video classification model is then used to process the acquired images to obtain a second image start-stop result. A pre-trained audio classification model is used to process the audio information to obtain an audio start-stop result. Rotational speed information is obtained based on the pulse signals, and a rotational speed start-stop result is derived from this information. Finally, the first image start-stop result, the second image start-stop result, the audio start-stop result, and the rotational speed start-stop result are fused according to their respective weights to obtain the final start-stop result for the target device.

[0020] This solution integrates multimodal data from images, rotation speed, and audio to determine the start-up and shutdown results of the equipment. This reduces the inaccuracy of a single algorithm, making monitoring more timely and accurate, and facilitating predictive maintenance. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart of a start-up and shutdown monitoring method based on multimodal data fusion provided in an embodiment of the present invention; Figure 2 A schematic diagram of the overall logic of the start-up and shutdown monitoring method based on multimodal data fusion provided in an embodiment of the present invention; Figure 3 This is an image of the conveyor belt in two frames before and after the actual material is present in an embodiment of the present invention, along with a differential grayscale image. Figure 4 The images and differential grayscale images of the belt in the absence of material in this embodiment of the invention are two frames before and after the belt. Figure 5 This is a schematic diagram of the pulse signal at a constant rotation speed in an embodiment of the present invention; Figure 6 This is a schematic diagram of the pulse signal under non-constant rotation speed in an embodiment of the present invention; Figure 7 This is a waveform diagram of a fixed-frequency sampling clock in an embodiment of the present invention; Figure 8 This is a functional block diagram of the start-up and shutdown monitoring system based on multimodal data fusion provided in an embodiment of the present invention; Figure 9 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention.

[0024] Please see Figure 1 The flowchart below shows the start-up and shutdown monitoring method based on multimodal data fusion provided in this embodiment of the invention. The detailed steps of the start-up and shutdown monitoring method based on multimodal data fusion are described below.

[0025] S11, Image acquisition is performed on the target device, and the acquired image is obtained by gray-scale weighted linear fitting to obtain the first image start / stop result; S12, The acquired images are processed using a pre-trained video classification model to obtain the second image start / stop result; S13, Collect audio information from the target device, process the audio information using a pre-trained audio classification model, and obtain audio start / stop results; S14: Acquire the pulse signal of the target device, obtain the speed information based on the pulse signal, and obtain the speed start-up and shutdown results based on the speed information; S15, the first image start-up and shutdown results, the second image start-up and shutdown results, the audio start-up and shutdown results, and the speed start-up and shutdown results are fused according to their weights to obtain the start-up and shutdown results of the target device.

[0026] The start-stop monitoring method provided in this embodiment fuses multimodal data of images, rotation speed and audio to determine the start-stop results of the equipment, reducing the shortcomings of low accuracy of a single algorithm, making monitoring more timely and accurate, and facilitating predictive maintenance.

[0027] The specific implementation methods of each of the above steps will be explained in detail below.

[0028] Please refer to the following: Figure 2 The monitoring and processing are mainly based on image, audio, and speed modules. In this embodiment, the target device can be a belt conveyor. When the belt conveyor is started, the pixel values ​​of the images collected at different times show a significant difference compared to when it is stopped. Therefore, in this embodiment, an image acquisition device (such as an industrial camera) is used to acquire images containing the target device, and then the start-up and shutdown status of the belt conveyor is obtained based on image analysis.

[0029] In this embodiment, the acquired image is processed using two branches: one branch processes the image based on gray-scale weighted linear fitting, and the other branch processes the image using a video classification model, such as the deep learning method of the Transformer video classification model.

[0030] The step of obtaining the start / stop result of the first image by using gray-level weighted linear fitting to obtain the acquired image can be implemented in the following way: For the current frame image, convert the current frame image and the previous frame image into grayscale images, and take the absolute value of the difference between the grayscale values ​​at corresponding positions to obtain a difference grayscale image; for each pixel coordinate point in the difference grayscale image, determine the polar coordinate system parameters corresponding to the straight line passing through the pixel coordinate point; for each set of polar coordinate system parameters, obtain its voting result and voting quantity based on the number of times the polar coordinate system parameters are determined; determine the target polar coordinate system parameter with the largest voting result, and obtain the voting quantity of the target polar coordinate system parameter; based on the voting result and voting quantity of the target polar coordinate system parameter, obtain the start / stop result of the first image.

[0031] In this embodiment, the inter-frame difference method is used to compare the pixel differences of images collected from a specific area of ​​the belt conveyor at different times to determine the operating status of the belt conveyor.

[0032] Images can be acquired at a frequency of N frames per second using an image acquisition device. For multiple consecutive acquired images, if the current frame is frame1 (e.g., ... Figure 3 If the image is on the left side of the current frame, then the previous frame of the current frame is denoted as frame2 (e.g., the image on the left side of the current frame). Figure 3 If the image is in the middle frame, then the difference grayscale image between the current frame image and the previous frame image can be obtained (e.g., ...). Figure 3 (Image on the right side of the middle section).

[0033] Specifically, the current frame image and the previous frame image can first be converted from color images to grayscale images, converting the three-channel color information into a single channel. Then, the grayscale values ​​at corresponding positions in the current frame image and the previous frame image are subtracted, and the absolute value is taken to obtain a difference grayscale image. The difference grayscale image can be cropped according to a pre-set ROI region to focus on obtaining images of the areas where the belt and idler rollers are located.

[0034] Based on this, the existing conventional processing method is to directly binarize the differential grayscale image, setting pixel values ​​below the grayscale threshold to 0 and pixel values ​​above the grayscale threshold to 255. Then, morphological processing is performed on the binary image, and Hough line detection is used to fit the line in the binary image. However, this method has two drawbacks. One is that the selection of the grayscale threshold is difficult, as the types of materials and operating speeds vary greatly from one conveyor belt to another. The other is noise interference. For example, daytime and nighttime lighting conditions directly affect the distribution of grayscale values ​​after differential processing, and noise disturbances around the conveyor belt area can also interfere with the fitting results.

[0035] Based on the above considerations, in this embodiment, instead of using a hard threshold to obtain a binary image and then perform line fitting, the Hough line detection is performed directly based on the grayscale image after difference. In the process of fitting the line, the magnitude of the grayscale value is taken into account as a weight. The larger the grayscale value, the more obvious the change before and after, and the more likely it is to be the edge of the belt or an area with large material changes, which is more likely to characterize the possibility of starting the belt conveyor.

[0036] The Hough line detection based on grayscale weighting involves the following processes: Parametric lines: In a Cartesian coordinate system, a line can be represented using... Let be the symbol, where k and b represent the slope and intercept, respectively. In polar coordinates, a straight line is represented by . cos To indicate, among which It is the perpendicular distance from the straight line to the origin. It is the angle between the line and the x-axis, and the two together constitute the unique parameter representing the line.

[0037] Image space to Hough space transformation: For each pixel coordinate point (x, y) in the difference grayscale image, calculate the polar coordinate system parameters corresponding to the line passing through that pixel coordinate point. , All pixels on a straight line share the same ( , ).

[0038] Based on this, grayscale weighted voting is performed. For each group of polar coordinate parameters, the voting result and the number of votes are obtained according to the frequency determined by the polar coordinate parameters. Specifically, this can be achieved in the following way: For each set of polar coordinate system parameters, once the polar coordinate system parameters are determined, the gray value of the pixel coordinate point is superimposed on its voting result, and 1 is added to its vote count, until all pixel coordinate points are traversed, and the final voting result and vote count of the polar coordinate system parameters are obtained.

[0039] In this embodiment, for each pixel coordinate (x, y), the following is listed: The corresponding range from -180° to 180° The values ​​are recorded using an accumulator array H to store the parameters of each polar coordinate system. , The voting results are used to determine the grayscale value corresponding to the pixel coordinates. Another accumulator array HC records the polar coordinate system parameters for each group. , The number of votes.

[0040]

[0041]

[0042] in, N For a straight line ( , The total number of points on the grid. Let be the grayscale value corresponding to the coordinates of the i-th pixel.

[0043] Based on the above, the voting results are summarized. Specifically, the polar coordinate system parameter with the largest voting result is determined as the target polar coordinate system parameter, and the number of votes for the target polar coordinate system parameter is obtained. Then, based on the voting result and the number of votes for the target polar coordinate system parameter, the start / stop result of the first image is obtained. Specifically, this step can be implemented in the following way: The average gray value of the voting participants in the line fitting vote is calculated based on the voting results and the number of votes in the target polar coordinate system parameters; the average gray value is compared with a preset threshold to obtain the start / stop result; the start / stop result of the first image is obtained based on the start / stop results corresponding to multiple frames of images within the sliding window.

[0044] In this embodiment, based on the accumulator result, we obtain... The maximum value h corresponds to ( , The array represents the line parameter with the most grayscale votes, and also yields the value corresponding to the maximum value. The numerical value hc is calculated using the following formula to obtain the average gray value of the votes participating in the linear fitting. :

[0045] When the belt conveyor stops The value is generally between 2 and 5, during startup. Since the values ​​are relatively large, using 10 as the threshold for judging start-up and shutdown can accommodate most situations. Therefore, the average grayscale value is compared with the preset threshold to determine whether the belt conveyor is stopped or started.

[0046] like Figure 4 As shown, when there is no material on the belt, the edge of the belt can still be fitted with a straight line in the manner described above.

[0047] Considering that there may be occasional interference from noise such as light and shadow in a few frames, the result of linear fitting is not used directly to represent the start-up and stop status of the belt conveyor at the current moment. Instead, a sliding window is used to maintain the results of multiple past frames (such as S frames). Finally, the contents of the sliding window are voted on to give the confidence result 'a' of the equipment start-up, which is used as the final start-up and stop result of the first image.

[0048] in, S The length of the sliding window. st This represents the number of devices started in the window.

[0049] Furthermore, in the method of performing image analysis and processing using a video classification model, the acquired images are processed using a pre-trained video classification model to obtain the second image start / stop result. Specifically, this step can be implemented in the following way: The current frame image and its preceding multiple frames are cropped separately, and the cropped multiple frames are stitched together to form a video segment. The video segment is then imported into a pre-trained video classification model for processing to obtain the start / stop result corresponding to the current frame image. The start / stop results corresponding to the multiple frames within the sliding window are combined to obtain the start / stop result of the second image.

[0050] In this embodiment, the video classification model can be trained in advance using the collected training samples. Specifically, multiple frames of images from the target device can be pre-collected, and firstly, they are cropped according to a pre-defined ROI region to focus on the designated belt area. The cropped images from each frame are then stitched together to form a small video clip, which serves as a training sample.

[0051] The constructed deep learning network is trained using multiple training samples until a preset iteration stopping condition is met, resulting in a well-trained video classification model.

[0052] In this embodiment, VideoMAE is used as the video classification model. The Video-MAE model employs an asymmetric encoder-decoder architecture for self-supervised pre-training tasks of masking and reconstruction. Each input image... Images of resolution are divided into The input image uses non-overlapping pixel blocks (tokens), each of which is transformed into high-dimensional features through block embedding (token embedding). Then, a portion of these pixel blocks (tokens) are randomly masked at a high masking rate (75%), and the remaining pixel blocks are sent to the encoder for feature extraction. The extracted features are then concatenated with another set of pre-defined learnable pixel blocks (learnable tokens) to form a feature set with the same size as the input image. Finally, a lightweight decoder reconstructs the original image based on these features.

[0053] ViodeMAE utilizes video segments composed of N frames, employs an extremely high masking rate, and retains very few tokens as input to the encoder. To better extract the spatiotemporal features of these unmasked tokens, ViT is introduced as the backbone. At the same time, spatiotemporal joint self-attention is adopted in the attention layer, and the mean squared error loss L1 is used to train the model.

[0054]

[0055] in, p It is the token's ID. It is a set of masking tokens, and I is the input image. It is a reconstructed image.

[0056] Based on the above design, the VideoMAE model can accurately capture the timing information in video clips, thereby determining whether the belt in the video clip is in motion or stationary.

[0057] The trained video classification model is used to infer the current frame image, providing the start / stop result of the target device corresponding to the current frame image. Similarly, to avoid noise interference, a sliding window is used to maintain the results of past multiple frames. Finally, the results obtained within the sliding window are statistically analyzed to obtain the confidence result b of the target device start-up, which serves as the start / stop result for the second image.

[0058] in, S The length of the sliding window. st This represents the number of devices that can be started in the sliding window.

[0059] The above describes the method for analyzing and processing the acquired images to obtain the start-up and shutdown results of the two branches. Considering that belt conveyor equipment is usually used in industrial environments to transport coal, stones, etc., the sound at the start-up point is significantly different from that at the shutdown point, and can be distinguished by audio.

[0060] Based on this, in this embodiment, the audio acquisition device in the industrial camera can also be used to acquire the environmental audio at various times in real time, and then the audio can be analyzed and processed to obtain the start and stop results of the belt conveyor.

[0061] Specifically, the audio information can be processed using a pre-trained audio classification model to obtain the audio start / stop results. This step can be achieved in the following way: The audio information is sliced ​​to obtain multiple audio segments, and Mel-spectral features are extracted from each audio segment. The Mel-spectral features of each audio segment are then imported into a pre-trained audio classification model for processing to obtain the start / stop results corresponding to the audio information. The start / stop results corresponding to multiple audio segments within a sliding window are combined to obtain the audio start / stop result.

[0062] In this embodiment, the audio classification model can be trained in advance. Audio segments can be collected in advance, and their Mel-spectral features can be extracted to form audio training samples. The audio classification model is obtained by training the model using multiple audio training samples.

[0063] In this embodiment, the ResNet34 model is used as the audio classification model, and the cross-entropy loss function L2 is used to train the model during the training process.

[0064]

[0065] in, These are the labels for the audio training samples. It is the class probability value output by the network.

[0066] The trained audio classification model is used to analyze and process the currently acquired audio information. Similarly, the Mel spectrum features of the current audio information are extracted and imported into the audio classification model to obtain the corresponding start-up and shutdown results.

[0067] Considering the impact of noise, a sliding window is used to maintain the start / stop results of multiple past audio information. Finally, the start / stop results within the sliding window are statistically analyzed to obtain the confidence result c of the target device's start-up, which is used as the audio start / stop result.

[0068] in, S The length of the sliding window. stThis represents the number of devices that can be started in the sliding window.

[0069] In addition, in this embodiment, a speed sensor can be installed at the roller component of the belt conveyor. The speed sensor collects signals, analyzes and obtains the speed of the belt conveyor, and then determines the start-up and shutdown status.

[0070] Specifically, by acquiring the pulse signal of the target device, obtaining the rotational speed information based on the pulse signal, and obtaining the rotational speed start-up and shutdown results based on the rotational speed information, this step can be achieved in the following way: The system acquires pulse signals from the target device and determines the number of sampling clock cycles that elapse within a single pulse period based on the pulse signal. When the number of cycles indicates the target device is operating at a constant speed, the speed is calculated based on the number of cycles and the sampling frequency. When the number of cycles indicates the target device is operating at a non-constant speed, the period of the current pulse signal is obtained based on the time difference between two consecutive pulse signals, and the average period of multiple pulse signals within a set window is calculated. The speed is then calculated based on this average period. The speed is compared with a preset speed to determine the start / stop result corresponding to the speed information. Finally, the start / stop result corresponding to multiple speed information within a sliding window is combined to obtain the speed start / stop result.

[0071] Please refer to the following: Figure 5 and Figure 6 These are the pulse signals generated by rotation. Figure 5 A pulse signal at a constant rotational speed. Figure 6 This is a pulse signal operating at a non-constant rotational speed. Figure 7 The waveform diagram is for a sampling clock with a fixed frequency.

[0072] A fixed-frequency sampling clock is counted starting from the rising or falling edge of the acquired pulse signal and ending at the next rising or falling edge of the pulse signal. Assume the sampling frequency of the sampling clock is... F The sampling clock period can then be calculated using the following formula. T c :

[0073] Then calculate the number of sampling clock cycles that elapse within a single pulse period. N Assuming N If it is a fixed value, it indicates that the target equipment is at a constant speed. N If the speed keeps changing, it indicates that the target device is operating at a non-constant speed.

[0074] When the target device is at a constant rotational speed, the rotational speed can be calculated based on the number of cycles and the sampling frequency using the following formula. (Transfer / Distribution):

[0075] When the target device is operating at a non-constant speed, the period T of the current pulse signal is calculated based on the time difference between two consecutive pulses using the following formula:

[0076] in, and These are the number of sampling clock cycles for the (i+1)th and i-th pulses, respectively.

[0077] Set a defined window L, and calculate the average period of multiple pulse signals within this defined window. :

[0078] The rotational speed (RPM) is calculated based on the obtained average cycle value using the following formula:

[0079] After obtaining the rotational speed under different conditions through the above methods, the rotational speed is compared with the preset rotational speed. For belt conveyors, the belt conveying speed is usually not very slow. Therefore, the preset rotational speed can be set between 1 and 1.5. When the rotational speed is greater than the preset rotational speed, the equipment operating status obtained by the rotational speed can be considered as the start-up state, and vice versa as the stop state.

[0080] Similarly, to avoid the influence of noise, a sliding window is used to record the rotational speed at multiple moments, and based on the recording results within the sliding window, the confidence level result d for the start-up and shutdown of the belt conveyor speed is given as the start-up and shutdown result:

[0081] Where S is the length of the sliding window, and st is the number of devices started in the sliding window.

[0082] After obtaining the first image start / stop result a, the second image start / stop result b, the audio start / stop result c, and the speed start / stop result d through the above methods, the four start / stop results can be fused according to weights to obtain the final start / stop result, which can be represented as follows:

[0083] in, For the weights corresponding to multimodal data x, different weights are assigned to the start and stop of each mode based on its accuracy, reliability, or importance in the actual scenario. For example, in some cases where the field noise is very severe, or there are many field devices and poor image quality, but the speed sensor is operating stably, the following weights can be used as a reference: =0.1, =0.1, =0.1, =0.7. Simple weighted fusion can be achieved using static weights, but this approach relies on the expert experience of on-site personnel and lacks interpretability.

[0084] Considering the limitations of static weights, this embodiment uses reinforcement learning to dynamically adjust the weights of different modalities to optimize the start-up and shutdown status monitoring of the belt conveyor. Specifically, the step of fusing the first image start-up and shutdown result, the second image start-up and shutdown result, the audio start-up and shutdown result, and the speed start-up and shutdown result according to their weights to obtain the start-up and shutdown result of the target device can be achieved in the following way: The weights of each modality data are dynamically adjusted using reinforcement learning. The start-stop results of the first image, the start-stop results of the second image, the start-stop results of the audio, and the start-stop results of the speed are fused according to the weights of the corresponding modal data to obtain the start-stop results of the target device.

[0085] The steps described above, which use reinforcement learning to dynamically adjust the weights corresponding to each modality of data, can be implemented in the following way: Multiple agents are constructed, each responsible for deciding the weights corresponding to each modality of data. In each iteration, each agent learns a policy based on a reinforcement learning network, adjusts its actions based on the Q-value function and the current state to determine the state for the next iteration. The current state is the start / stop result corresponding to each modality of data, and the action is the adjustment action of the weights corresponding to each modality of data. This iteration is repeated for multiple rounds until the reward function is maximized under the weights corresponding to each modality of data. The reward function is determined based on the difference between the start / stop predicted state and the actual state.

[0086] In this embodiment, the problem of dynamically adjusting the weights of each modality of data is transformed into the problem of finding the optimal weight combination for the agent in reinforcement learning. The basic elements involved in reinforcement learning include the following categories: Agent: In this invention, the agent is the part responsible for deciding the weight of each modality of data.

[0087] Environment: The environment includes the status of the belt conveyor and data collected through various sensors (speed, image, audio).

[0088] State: Represents the observed data of the system at a certain moment, including the start and stop results corresponding to the current modal data.

[0089] Action: An action is the operation that the agent needs to take, that is, to adjust the weights of different modal data.

[0090] Reward: The reward is given based on the impact of the agent's decision (adjusting weights) on the final start-up and shutdown state prediction. The goal is to maximize the reward to ensure the accuracy of the final prediction.

[0091] In reinforcement learning, the main considerations are state space, action space, reward function, and policy.

[0092] The state space comprises all the information that the agent can perceive at each time step. In this invention, the state mainly includes the following three types of information: Confidence of each module: Start-up and shutdown confidence of output by methods such as rotational speed, inter-frame difference, image and video classification, and audio classification (i.e., four start-up and shutdown results); Historical prediction error: The prediction error for each mode, used to guide the importance of modes during weighted fusion; Environmental changes: The operating conditions of the equipment, such as whether the equipment is in a stable state, whether the vibration is severe, and whether the belt is unloaded, can affect the performance of different modes.

[0093] In this invention, the state can be represented as:

[0094] in, This represents the start / stop confidence level for a specific mode. This represents the prediction error for a certain mode.

[0095] In reinforcement learning, the action space represents the actions that an agent can choose. In this invention, an action refers to adjusting the confidence of the start-stop results of different modalities. A new weight is assigned to each modality. Based on the four states obtained above, the weight update method is defined as follows:

[0096]

[0097] in, The weights for each modality range from [0,1].

[0098] The reward function is the most crucial part of reinforcement learning. The goal of optimization is to maximize the final prediction accuracy. Therefore, the reward function is defined based on the accuracy of the final prediction result. In this invention, the reward is defined based on the prediction error. For example, when the system predicts the start-up and shutdown state of a belt conveyor, it compares the difference between the predicted state and the actual state and gives a corresponding reward. Cross-entropy loss is used to measure the accuracy of the prediction. The reward function is defined as follows:

[0099] in, It is the actual status of the target device (start / stop label). It is the predicted state of start-up and shutdown after weighted fusion. If the prediction result is close to the actual result, the reward is larger; if the prediction error is large, the reward is smaller.

[0100] In reinforcement learning, the policy is the rule used by the agent to choose actions. This invention uses a reinforcement learning network (DQN) to learn a policy function based on the current state. To select the optimal weight adjustment action The strategy selects actions by learning a Q-value function, which can be represented as follows:

[0101] in, This represents the Q-value for choosing a certain action given a certain state. It is a discount factor.

[0102] In reinforcement learning, the weight parameters and the reinforcement learning network are initialized first to determine the initial state S0. In each round, the weight parameters and the reinforcement learning network are determined according to the current state S0. t Select Action and get rewards and the next state The policy is updated using the Q-learning algorithm, which updates the Q-value function by minimizing the error. Through the optimization process of reinforcement learning, the weights of each mode are automatically adjusted to maximize the final prediction accuracy.

[0103] The performance of various modes may differ under different operating environments. For example, when the conveyor belt is running stably, the speed mode usually provides the most accurate information, so the speed mode has a higher weight. When the equipment experiences vibration or noise, audio classification or image / video classification may become more important, in which case the weight of the audio or image mode needs to be increased.

[0104] Through reinforcement learning, the system can adaptively adjust the weights of these modalities according to different scenarios. For example, in some situations, the audio modality performs poorly, and the system automatically reduces the weight of audio. In other environments, image classification may be very reliable, and the system automatically increases the weight of image classification.

[0105] By dynamically adjusting modality weights through reinforcement learning, the system can adapt to changes in different environments and maximize the contribution of each modality. The goal of reinforcement learning is to maximize the accuracy of model predictions through a reward function and automatically optimize the weights of each modality.

[0106] This method is highly adaptable and can dynamically adjust the influence of different modalities in the final decision based on real-time data. It is an innovative and promising solution.

[0107] In this embodiment, after determining the weights of each modal data in the above manner, the first image start-stop result, the second image start-stop result, the audio start-stop result, and the speed start-stop result are fused according to their respective weights to obtain the final start-stop result.

[0108] In this embodiment, the first image start-up / stop result, the second image start-up / stop result, the audio start-up / stop result, and the speed start-up / stop result can be compared to analyze and obtain the status of the target equipment. For example, if the speed start-up / stop result indicates that the current equipment is started with a very high confidence level, but the image start-up / stop result indicates that it is stopped with a very high confidence level, then the current belt conveyor equipment is likely in a slipping state. If both audio and image results indicate that the equipment is started, but the speed result consistently indicates that it is stopped, then the motor of the current belt conveyor equipment may be faulty, thus achieving early warning.

[0109] In this way, by comparing and analyzing the start-up and shutdown results corresponding to multimodal data, we can better understand the equipment and analyze the possible abnormal situations of the target equipment.

[0110] The start-stop monitoring method based on multimodal data fusion provided in this embodiment combines images, rotational speed, and audio to determine the operating status of the belt conveyor. Furthermore, the weights of the multimodal data are dynamically adjusted using reinforcement learning, which can adaptively adjust according to on-site conditions, making it more consistent with actual scenarios.

[0111] In addition, by comparing start-up and shutdown results from different sources, it is possible to determine whether the equipment is slipping, whether the motor is worn, etc., thus providing predictive maintenance.

[0112] In video processing, attention is focused on the belt area of ​​interest to prevent interference from areas outside the belt. Furthermore, using grayscale values ​​as an important basis for fitting a straight line enables efficient and accurate extraction of straight lines from the difference grayscale image without the need for binarization and morphological processing, thus avoiding interference and errors caused by these processes.

[0113] Based on the same inventive concept, please refer to Figure 8 This invention also provides a functional module diagram of a start-up and shutdown monitoring system based on multimodal data fusion. This embodiment can divide the multimodal data fusion-based start-up and shutdown monitoring system into functional modules according to the above method embodiments. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware or as software functional modules. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0114] For example, when dividing functional modules according to their respective functions, Figure 8 The start-stop monitoring system based on multimodal data fusion shown is only a schematic diagram. This system may include an image processing module, an audio processing module, a speed processing module, and a fusion monitoring module. The functions of each module in this system will be described in detail below.

[0115] The image processing module is used to acquire images of the target device, obtain the first image start-up and shutdown result by using gray-scale weighted linear fitting on the acquired image, and process the acquired image using a pre-trained video classification model to obtain the second image start-up and shutdown result. The audio processing module is used to collect audio information from the target device, process the audio information using a pre-trained audio classification model, and obtain audio start / stop results. The speed processing module is used to acquire pulse signals from the target device, obtain speed information based on the pulse signals, and obtain speed start / stop results based on the speed information; The fusion monitoring module is used to fuse the first image start-up and shutdown results, the second image start-up and shutdown results, the audio start-up and shutdown results, and the speed start-up and shutdown results according to their weights to obtain the start-up and shutdown results of the target equipment.

[0116] The start-up and shutdown monitoring system based on multimodal data fusion provided in this embodiment can be used to execute the start-up and shutdown monitoring method based on multimodal data fusion under any of the above embodiments. For details not covered in this embodiment, please refer to the corresponding descriptions in the above embodiments. This embodiment will not elaborate further here.

[0117] Please see Figure 9 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. The electronic device can be a computer, server, or other device in a back-end analysis platform. It can communicate with industrial cameras, speed sensors, etc. The electronic device includes a memory, a processor, and a communication module. The memory, processor, and communication module are electrically connected directly or indirectly to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0118] The memory is used to store computer programs or data. Memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc.

[0119] The processor is used to read / write data or programs stored in the memory and execute the start-up and shutdown monitoring method based on multimodal data fusion provided in any embodiment of the present invention.

[0120] The communication module is used to establish communication connections between electronic devices and other communication terminals via a network, and to send and receive data via the network.

[0121] It should be understood that, Figure 9 The structure shown is only a schematic diagram of an electronic device; the electronic device may also include components that are larger than those shown. Figure 9 The more or fewer components shown, or having the same Figure 9 The different configurations shown.

[0122] Furthermore, embodiments of the present invention also provide a computer-readable storage medium storing machine-executable instructions, which, when executed, implement the start-up and shutdown monitoring method based on multimodal data fusion provided in the above embodiments.

[0123] Specifically, the computer-readable storage medium can be a general-purpose storage medium, such as a removable disk or hard disk. When the computer program on the computer-readable storage medium is executed, it can perform the aforementioned start-up and shutdown monitoring method based on multimodal data fusion. The processes involved in the execution of the executable instructions on the computer-readable storage medium can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0124] In the embodiments provided by this invention, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interface; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0125] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0126] Furthermore, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0127] It should be noted that if the functionality is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0128] The above are merely embodiments of the present invention and are not intended to limit the scope of protection of the present invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A start-up and shutdown monitoring method based on multimodal data fusion, characterized in that, The method includes: Image acquisition is performed on the target device. The acquired image is fitted with a gray-scale weighted straight line to obtain the first image start-up and shutdown result. The acquired image is then processed using a pre-trained video classification model to obtain the second image start-up and shutdown result. The audio information of the target device is collected, and the audio information is processed using a pre-trained audio classification model to obtain the audio start / stop result; The pulse signal of the target device is collected, the rotation speed information is obtained based on the pulse signal, and the rotation speed start-up and shutdown result is obtained based on the rotation speed information; The first image start-up and shutdown result, the second image start-up and shutdown result, the audio start-up and shutdown result, and the speed start-up and shutdown result are fused according to weights to obtain the start-up and shutdown result of the target device; The image includes multiple consecutive frames; the step of obtaining the first image start / stop result by using grayscale weighted linear fitting on the acquired image includes: For the current frame image, convert the current frame image and the previous frame image into grayscale images, and take the absolute value of the difference between the grayscale values ​​at corresponding positions to obtain a difference grayscale image; for each pixel coordinate point in the difference grayscale image, determine the polar coordinate system parameters corresponding to the straight line passing through the pixel coordinate point; for each set of polar coordinate system parameters, obtain its voting result and voting quantity based on the number of times the polar coordinate system parameters are determined; determine the target polar coordinate system parameter with the largest voting result, and obtain the voting quantity of the target polar coordinate system parameter; based on the voting result and voting quantity of the target polar coordinate system parameter, obtain the first image start / stop result; The step of processing the acquired images using a pre-trained video classification model to obtain the second image start / stop result includes: The current frame image and its preceding multiple frames are cropped separately, and the cropped multiple frames are stitched together to form a video segment; the video segment is imported into a pre-trained video classification model for processing to obtain the start / stop result corresponding to the current frame image; the start / stop result of the second image is obtained by combining the start / stop results corresponding to the multiple frames in the sliding window. The steps of acquiring the pulse signal of the target device, obtaining speed information based on the pulse signal, and obtaining the speed start / stop result based on the speed information include: The process involves acquiring pulse signals from the target device, determining the number of sampling clock cycles elapsed within a single pulse cycle based on the pulse signals, calculating the rotational speed based on the number of cycles and the sampling frequency when the number of cycles indicates that the target device is operating at a constant speed, and calculating the rotational speed based on the average of the cycles of multiple pulse signals within a set window when the number of cycles indicates that the target device is operating at a non-constant speed. The process also involves comparing the rotational speed with a preset rotational speed to determine the start / stop result corresponding to the rotational speed information, and finally, combining the start / stop results corresponding to multiple rotational speed information within a sliding window to obtain the rotational speed start / stop result.

2. The start-up and shutdown monitoring method based on multimodal data fusion according to claim 1, characterized in that, The step of obtaining the voting result and the number of votes for each set of polar coordinate system parameters based on the number determined by the polar coordinate system parameters includes: For each set of polar coordinate system parameters, once the polar coordinate system parameters are determined, the gray value of the pixel coordinate point is superimposed on its voting result, and 1 is added to its vote count, until all pixel coordinate points are traversed, and the final voting result and vote count of the polar coordinate system parameters are obtained.

3. The start-up and shutdown monitoring method based on multimodal data fusion according to claim 1, characterized in that, The step of obtaining the first image start / stop result based on the voting results and the number of votes according to the target polar coordinate system parameters includes: The average gray value of the participants in the line fitting vote is calculated based on the voting results and the number of votes for the target polar coordinate system parameters. The average gray value is compared with a preset threshold to obtain the start / stop result; The start / stop result of the first image is obtained based on the start / stop results corresponding to multiple frames of images within the sliding window.

4. The start-up and shutdown monitoring method based on multimodal data fusion according to claim 1, characterized in that, The step of processing the audio information using a pre-trained audio classification model to obtain the audio start / stop result includes: The audio information is sliced ​​to obtain multiple audio segments, and Mel-spectral features are extracted from each audio segment. The Mel-spectral features of each audio segment are imported into a pre-trained audio classification model for processing to obtain the start / stop results corresponding to the audio information. The audio start / stop result is obtained by combining the start / stop results corresponding to multiple audio segments within the sliding window.

5. The start-up and shutdown monitoring method based on multimodal data fusion according to claim 1, characterized in that, The step of fusing the first image start-up / stop result, the second image start-up / stop result, the audio start-up / stop result, and the speed start-up / stop result according to weights to obtain the start-up / stop result of the target device includes: Reinforcement learning is used to dynamically adjust the weights corresponding to each modality of data; The first image start-stop result, the second image start-stop result, the audio start-stop result, and the speed start-stop result are fused according to the weights of the corresponding modal data to obtain the start-stop result of the target device.

6. The start-up and shutdown monitoring method based on multimodal data fusion according to claim 5, characterized in that, The step of dynamically adjusting the weights corresponding to each modality data using reinforcement learning includes: Construct multiple intelligent agents, each of which is responsible for deciding the weights corresponding to each modal data. In each iteration, each agent learns a policy based on a reinforcement learning network, adjusts its actions based on the Q-value function and the current state to determine the state of the next iteration, wherein the current state is the start-up and shutdown result corresponding to each modality data, and the action is the adjustment action of the weight corresponding to each modality data. Repeat the iterations multiple times until the reward function is maximized under the weights corresponding to each modality data. The reward function is determined based on the difference between the start-up / shutdown predicted state and the actual state.

7. A start-up and shutdown monitoring system based on multimodal data fusion, characterized in that, The system is used to implement the start-up and shutdown monitoring method based on multimodal data fusion as described in any one of claims 1-6, the system comprising: The image processing module is used to acquire images of the target device, obtain the first image start-up and shutdown result by using gray-scale weighted linear fitting on the acquired image, and process the acquired image using a pre-trained video classification model to obtain the second image start-up and shutdown result. The audio processing module is used to collect audio information from the target device, process the audio information using a pre-trained audio classification model, and obtain audio start / stop results. The speed processing module is used to acquire the pulse signal of the target device, obtain speed information based on the pulse signal, and obtain the speed start-up and shutdown result based on the speed information; The fusion monitoring module is used to fuse the first image start-up and shutdown results, the second image start-up and shutdown results, the audio start-up and shutdown results, and the speed start-up and shutdown results according to weights to obtain the start-up and shutdown results of the target device.

Images