Equipment abnormal behavior identification method and system based on audio and video streams

Abstract

The invention relates to the technical field of equipment anomaly recognition, and particularly discloses an equipment anomaly behavior recognition method and system based on audio and video streams, and the method comprises the steps: collecting equipment operation audio decibel values, preprocessing the audio decibel values to obtain digital audio streams, converting the digital audio streams to obtain a detection spectrogram, calculating a wave crest offset ratio and a wave crest number ratio, and obtaining section anomaly values; the method comprises the following steps: acquiring a video image of an equipment connection part, calculating a position and an angle offset ratio by using an algorithm to obtain a stable value of a part, judging and generating an abnormal analysis signal, further calculating an abnormal difference ratio, a stable difference ratio and an equipment abnormal ratio based on the abnormal analysis signal, drawing a change curve, and calculating an angle offset ratio, variance and the like to obtain linear values. The system comprises an audio acquisition and analysis module, a video analysis module and an abnormity judgment module. The device abnormal behavior can be effectively identified, and the device maintenance efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of device abnormality identification, and in particular to a method and system for identifying abnormal behavior of a device based on audio and video streams. Background Art

[0002] In the power industry, due to the large number of personnel and equipment, complex situations, and limited production supervisors, it is difficult to achieve all-round and full-process safety inspections and management. Therefore, the use of equipment abnormal behavior recognition technology based on audio and video streams, combined with power equipment, can effectively improve the safety and stability of power facilities and optimize power production and efficiency.

[0003] Traditional methods for identifying abnormal device behavior mainly focus on single-mode data monitoring and analysis, and lack multi-dimensional device status monitoring based on audio and video information. That is, audio signals can reflect changes in sound characteristics during device operation, which can serve as early warning signals of device abnormalities; video images can intuitively display information such as the appearance of the device, the status of the connection parts, and the movement trajectory of the moving parts. By organically combining the two, the abnormal behavior characteristics of the device can be captured more comprehensively and accurately, avoiding the problem of missing information caused by single-mode monitoring.

[0004] To this end, the present invention provides a method and system for identifying abnormal behavior of a device based on audio and video streams. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for identifying abnormal behavior of a device based on audio and video streams to solve the above-mentioned background problems.

[0006] The purpose of the present invention can be achieved through the following technical solutions: A method for identifying abnormal behavior of a device based on audio and video streams, comprising: Collecting the audio decibel value emitted by the device during operation, pre-processing the audio decibel value, and obtaining a digital audio stream; Based on the digital audio stream, the detection spectrum is transformed to obtain a detection spectrum diagram, the detection spectrum diagram and the target spectrum diagram are divided into multiple target segments and compared to obtain the peak shift ratio Py and the peak number ratio Bf, based on the peak shift ratio Py and the peak number ratio Bf, the segment abnormality value Yc is calculated, if the segment abnormality value Yc ≥ the segment abnormality threshold value Ycz, if the segment abnormality value Yc ≥ the segment abnormality threshold value Ycz, an image analysis signal is generated; Based on the image analysis signal, the video image of the equipment connection part is obtained, and the position offset ratio Py and the angle offset ratio Jd of the equipment connection part are calculated through the algorithm. Based on the position offset ratio Py and the angle offset ratio Jd, the component stability value Wd is calculated, and the component abnormal value Bj is less than the component stability threshold Wdz, and the abnormal analysis signal is generated; Based on the abnormal analysis signal, the abnormal difference ratio Ycz and the stable difference ratio Wdc are obtained. The abnormal difference ratio Ycz is ratioed with the stable difference ratio Wdc to obtain the equipment abnormality ratio Ycb, which is plotted as a cycle-equipment abnormality ratio Ycb change curve. The two ends of the curve are connected to obtain a reference line. The peak points and trough points of the change curve are obtained and connected to the coordinate origin with a straight line. The angle to be measured formed by the straight line and the reference line is calculated, and the angle to be measured is calculated with the threshold of the angle to be measured to obtain the angle offset ratio Jdb and the angle offset variance Fc. The linear value Xz is calculated by the formula. If the linear value Xz ≥ the linear threshold Xzy, an early warning is sent to the system.

[0007] As a further solution of the present invention: the segment abnormal value Yc is obtained in the following manner: Based on the peak offset ratio Py and the peak number ratio Bf, the segment outlier value Yc is calculated using a weighted formula.

[0008] As a further solution of the present invention: the peak shift ratio Py is obtained as follows: Perform fast Fourier transform on the digital audio stream to convert the time domain signal of the digital audio stream into a frequency domain signal. Based on the frequency domain signal, draw a detection spectrum with frequency as the horizontal axis and amplitude as the vertical axis; Dividing the horizontal axes of the detection spectrum graph and the target spectrum graph into a plurality of segments to obtain a target segment; Obtain the peak position difference between the detection spectrum graph and the target spectrum graph in the target section to obtain the peak offset difference; The peak shift difference is ratioed to the shift difference threshold to obtain a peak shift ratio, which is marked as Py.

[0009] As a further solution of the present invention: the peak number ratio Bf is obtained as follows: Obtain the number of peaks of the detection spectrum graph and the target spectrum graph in the target section, perform difference processing on the number of peaks of the detection spectrum graph and the target spectrum graph, and take the absolute value to obtain the difference in the number of peaks; The peak number difference is processed by ratio processing with the peak number of the target spectrum graph to obtain the peak number ratio, and the peak number ratio is marked as Bf.

[0010] As a further solution of the present invention: the component stability value Wd is obtained in the following manner: Based on the position deviation ratio Py and the angle deviation ratio Jd, the component stability value Wd is calculated; By formula: , the component stability value Wd is calculated, where ln(b1*Py+b2*Jd+1.011) is a logarithmic function with base e, and b1 and b2 are preset proportional coefficients.

[0011] As a further solution of the present invention: the position offset ratio Py is obtained as follows: During the monitoring period, video images of the equipment connection parts are obtained; Get the total number of video image frames, and mark the total number of video frames as n; The positional relationship of the device connection parts of each frame of the video is used to calculate the position change difference of the center coordinates of the connection parts, and the position change difference is marked as Bh i , i=1,2,.....,n-1, n is the total number of video image frames; The YOLOv8 algorithm is used to calculate the center coordinates of the connection parts of each frame of the video (x i ,y i ), the image position coordinates of adjacent frames (x i+1 ,y i+1 ); Calculate the offset in the X-axis direction: , the offset in the Y-axis direction: , through the distance formula: , calculate the position offset Bh1; The position change difference Bh i Performing summation and averaging to obtain a position difference mean, calculating the difference between the position difference mean and the position difference threshold to obtain a position offset difference; The offset difference is processed by ratio with the position difference threshold to obtain the position offset ratio, which is marked as Py; The angle deviation ratio Jd is obtained as follows: Based on the X-axis and Y-axis offsets, the position offset angle θ is calculated using the inverse tangent function. i , i=1,2,.....,n-1, n is the total number of video image frames; The position is offset by an angle θ i Perform summation and mean processing to obtain the mean of the angle difference; The angle difference mean is compared with the angle difference threshold to obtain an angle deviation ratio, and the calculated angle deviation ratio is marked as Jd.

[0012] As a further solution of the present invention: the linear value Xz is obtained as follows: Based on the angle deviation ratio Jdb and the angle deviation variance Fc, the linear value Xz is calculated; By formula: , the linear value Xz is calculated, where c1 and c2 are preset proportional coefficients.

[0013] As a further solution of the present invention: the angle deviation ratio Jdb is obtained as follows: Perform difference processing on the angle to be measured and the angle threshold to be measured to obtain the angle offset difference; Obtain all angle offset differences on the cycle-equipment abnormality ratio Ycb change curve, sum and average all angle differences to obtain the average offset difference; The mean value of the offset difference is compared with the angle threshold to be measured to obtain the angle offset ratio, which is marked as Jdb; The angle offset variance is obtained as follows: Put all the angle offset differences on the cycle-equipment abnormality ratio Ycb change curve into the angle offset difference data group, calculate the variance of all the angle offset differences in the data group, obtain the angle offset variance, and mark the angle offset variance as Fc.

[0014] As a further solution of the present invention: the method for obtaining the angle to be measured is: Marking the target segment generating the image analysis signal as an abnormal segment; Obtain the difference between the segment abnormal value Yc and the segment abnormal threshold Ycz in the abnormal segment to obtain the segment abnormal difference; Obtain the section abnormal differences of all abnormal sections, sum and average the abnormal differences of all sections to obtain the mean of the abnormal differences; The mean of the abnormal difference is processed by ratio with the segment abnormal threshold value Ycz to obtain the abnormal difference ratio, and the abnormal difference ratio is marked as Ycz; The component stability value Wd is processed with the component stability threshold value Wdz to obtain the component stability difference; Calculate the ratio of the component stability difference to the component stability threshold Wdz to obtain the stability difference ratio, and mark the stability difference ratio as Wdc; The stable difference ratio Wdc is processed with the abnormal difference ratio Ycz to obtain the equipment abnormality ratio Ycb; Obtain the equipment abnormality ratio Ycb of multiple monitoring cycles, and draw the cycle-equipment abnormality ratio Ycb change curve in a two-dimensional rectangular coordinate system; Connect the two ends of the cycle-equipment abnormality ratio Ycb change curve with a straight line to obtain a reference line; Get the peak and trough points of the cycle-equipment abnormality ratio Ycb change curve; Connect the peak point, trough point and coordinate origin with a straight line, obtain the angle enclosed by the straight line and the reference line, and get the angle to be measured.

[0015] A device abnormal behavior recognition system based on audio and video streams, comprising: Audio acquisition module: collects the audio decibel value emitted by the device during operation, pre-processes the audio decibel value, and obtains a digital audio stream; Audio analysis module: based on the digital audio stream, the detection spectrum is transformed to obtain the detection spectrum and the target spectrum, and the detection spectrum and the target spectrum are divided into multiple target segments and compared to obtain the peak shift ratio Py and the peak number ratio Bf. Based on the peak shift ratio Py and the peak number ratio Bf, the segment abnormal value Yc is calculated. If the segment abnormal value Yc ≥ the segment abnormal threshold Ycz, if the segment abnormal value Yc ≥ the segment abnormal threshold Ycz, an image analysis signal is generated; Video analysis module: Based on the image analysis signal, the video image of the equipment connection part is obtained, and the position offset ratio Py and the angle offset ratio Jd of the equipment connection part are calculated through the algorithm. Based on the position offset ratio Py and the angle offset ratio Jd, the component stability value Wd is calculated, and the component abnormal value Bj is less than the component stability threshold Wdz, and an abnormal analysis signal is generated; Abnormal judgment module: Based on the abnormal analysis signal, the section abnormal value Yc and the section abnormal threshold Ycz are calculated to obtain the abnormal difference ratio Ycz. Similarly, the stable difference ratio Wdc is obtained. The abnormal difference ratio Ycz and the stable difference ratio Wdc are ratio-processed to obtain the equipment abnormality ratio Ycb, which is plotted as a cycle-equipment abnormality ratio Ycb change curve. The two ends of the curve are connected to obtain a reference line, and the peak point and trough point of the change curve are obtained and connected to the coordinate origin. The angle to be measured enclosed by the straight line and the reference line and the angle threshold to be measured are calculated to obtain the angle offset ratio Jdb and the angle offset variance Fc. The linear value Xz is calculated by the formula. If the linear value Xz ≥ the linear threshold Xzy, an early warning is sent to the system.

[0016] Beneficial effects of the present invention: By combining audio and video stream data, the operating status of the equipment can be monitored from multiple dimensions. Audio spectrum analysis can capture the frequency and amplitude changes of the sound of the equipment operation. At the same time, video image analysis can intuitively present the visual status of the equipment appearance, connection parts and moving parts. The fusion of the two realizes information complementarity and avoids the limitations of single-mode monitoring.

[0017] The use of fast Fourier transform in the audio processing stage can quickly and accurately convert time-domain audio signals into frequency-domain signals, clearly present the audio spectrum characteristics, and provide a reliable basis for audio anomaly analysis. In video analysis, important parameters such as position offset ratio and angle offset ratio are calculated, which effectively improves the accuracy and speed of video analysis. Based on these advanced algorithms, the system can process audio and video data in real time and efficiently, promptly detect abnormal equipment behavior and accurately locate the cause of the fault. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The present invention will be further described below in conjunction with the accompanying drawings.

[0019] Figure 1 It is a flow chart of a method for identifying abnormal behavior of a device based on audio and video streams of the present invention; Figure 2 It is a module diagram of a device abnormal behavior recognition system based on audio and video streams of the present invention; Figure 3 It is a detection spectrum diagram of the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] Example 1, please refer to Figure 1 As shown, the present invention is a method for identifying abnormal behavior of a device based on audio and video streams, comprising: Step 1: collecting audio decibel values ​​emitted by the device during operation, pre-processing the audio decibel values, and obtaining a digital audio stream; During the monitoring period, the audio sensor is used to collect the audio decibel value of the equipment during operation, and a digital audio stream is obtained after preprocessing; In some embodiments, preprocessing the audio decibel value includes: converting the collected audio decibel value into a digital audio value through an analog signal-to-digital signal converter, and performing noise reduction processing on the digital audio value to remove background noise of the digital audio, thereby obtaining a digital audio stream; Step 2: Perform transformation based on the digital audio stream to obtain a detection spectrum diagram, divide the horizontal axes of the detection spectrum diagram and the target spectrum diagram into multiple sections to obtain a target section, obtain the difference in the positions of the peaks of the detection spectrum diagram and the target spectrum diagram in the target section, further calculate to obtain the peak shift ratio Py, obtain the difference in the number of peaks of the detection spectrum diagram and the target spectrum diagram, calculate the difference in the number of peaks and the number of peaks of the target spectrum diagram to obtain the peak number ratio Bf, calculate the section abnormal value Yc based on the peak shift ratio Py and the peak number ratio Bf, and generate an image analysis signal if the section abnormal value Yc ≥ the section abnormal threshold Ycz; like Figure 3 As shown, the digital audio stream is subjected to a fast Fourier transform (FFT), and the time domain signal of the digital audio stream is converted into a frequency domain signal. Based on the frequency domain signal, a detection spectrum is drawn with the frequency as the horizontal axis and the amplitude as the vertical axis; Dividing the horizontal axes of the detection spectrum graph and the target spectrum graph into a plurality of segments to obtain a target segment; It should be noted that the target segment division of the detection spectrum graph and the target spectrum graph is consistent; comparing the detection spectrum with the target spectrum; Obtain the peak position difference between the detection spectrum graph and the target spectrum graph in the target section to obtain the peak offset difference; For example, there are three significant peaks in the target spectrum, which are located at the frequency values ​​of 20Hz and 50Hz, respectively, while in the detection spectrum, the corresponding peaks are located at 22Hz and 53Hz, the offset difference of the first peak is 22-20=2Hz, and the offset difference of the second peak is 53-50=3Hz; The peak shift difference is processed by ratio with the shift difference threshold to obtain the peak shift ratio, and the peak shift ratio is marked as Py; Obtain the number of peaks of the detection spectrum graph and the target spectrum graph in the target section, perform difference processing on the number of peaks of the detection spectrum graph and the target spectrum graph, and take the absolute value to obtain the difference in the number of peaks; The peak number difference is processed by ratio with the peak number of the target spectrum graph to obtain the peak number ratio, and the peak number ratio is marked as Bf; Based on the peak shift ratio Py and the peak number ratio Bf, the segment abnormal value Yc is calculated; By formula: , calculate the segment abnormal value Yc, where a1 and a2 are preset proportional coefficients; Compare the segment abnormal value Yc with the segment abnormal threshold value Ycz; If the segment abnormal value Yc ≥ the segment abnormal threshold Ycz, it indicates that the spectrum in the current segment is abnormally large, and an image analysis signal is generated to further analyze the device image and analyze the abnormal behavior of the device; If the segment abnormal value Yc is less than the segment abnormal threshold Ycz, it indicates that the spectrum abnormality in the current segment is within the expected range, and it is still necessary to continuously monitor the changes in the segment abnormal value Yc; The technical solution of this embodiment is: collect audio decibel values ​​emitted during the operation of the device, pre-process the audio decibel values ​​to obtain a digital audio stream, perform transformation based on the digital audio stream to obtain a detection spectrum diagram, divide the horizontal axes of the detection spectrum diagram and the target spectrum diagram into multiple segments to obtain a target segment, obtain the difference in the positions of the peaks of the detection spectrum diagram and the target spectrum diagram in the target segment, further calculate to obtain the peak shift ratio Py, obtain the difference in the number of peaks between the detection spectrum diagram and the target spectrum diagram, calculate the difference in the number of peaks and the number of peaks in the target spectrum diagram to obtain the peak number ratio Bf, calculate the segment abnormality value Yc based on the peak shift ratio Py and the peak number ratio Bf, and generate an image analysis signal if the segment abnormality value Yc ≥ the segment abnormality threshold Ycz.

[0022] Embodiment 2, step 3, based on the image analysis signal, obtain a video image of the device connection part, calculate the position offset ratio Py and the angle offset ratio Jd of the device connection part through an algorithm, calculate the component stability value Wd based on the position offset ratio Py and the angle offset ratio Jd, and the component abnormal value Bj < the component stability threshold Wdz, and generate an abnormal analysis signal; During the monitoring period, a video recording device is used to obtain video images of the device connection area; Use the YOLOv8 algorithm to obtain feature information of device connection parts in video images; Among them, the characteristic information includes the positional relationship between the connection part and other parts of the device; Specifically, the video image is obtained through the YOLOv8 algorithm, the boundary coordinates (x, y) of each connection part of the device are obtained, the specific position of the connection part in the image is obtained through the coordinates, and the center coordinates of the connection part are calculated; Get the total number of video image frames, and mark the total number of video frames as n; The positional relationship of the device connection parts of each frame of the video is used to calculate the position change difference of the center coordinates of the connection parts, and the position change difference is marked as Bh i , i=1,2,.....,n-1, n is the total number of video image frames; The YOLOv8 algorithm is used to calculate the center coordinates of the connection parts of each frame of the video (x i ,y i ), the image position coordinates of adjacent frames (x i+1 ,y i+1 ); Calculate the offset in the X-axis direction: , the offset in the Y-axis direction: , through the distance formula: , calculate the position offset Bh1; Based on the X-axis and Y-axis offsets, calculate the position offset angle θ i , i=1,2,.....,n-1, n is the total number of video image frames, through the inverse tangent function: , get the position offset angle θ i ; The position change difference Bh i Performing summation and averaging to obtain a position difference mean, calculating the difference between the position difference mean and the position difference threshold to obtain a position offset difference; The offset difference is processed by ratio with the position difference threshold to obtain the position offset ratio, which is marked as Py; The position is offset by an angle θ i Perform summation and mean processing to obtain the mean of the angle difference; The angle difference mean is compared with the angle difference threshold to obtain an angle deviation ratio, which is marked as Jd; It should be noted that the position difference threshold and the angle difference threshold are set by professionals in this field based on experience; Based on the position deviation ratio Py and the angle deviation ratio Jd, the component stability value Wd is calculated; By formula: , calculate the component stability value Wd, where ln(b1*Py+b2*Jd+1.011) is a logarithmic function with base e, and b1 and b2 are preset proportional coefficients; Compare the component stability value Wd with the component stability threshold value Wdz; If the component stability value Wd ≥ component stability threshold Wdz, it means that the component stability at the equipment connection location is good and does not require maintenance temporarily; If the component stability value Wd is less than the component stability threshold Wdz, it means that the component stability at the equipment connection part is poor, the equipment is abnormal, and an abnormal analysis signal is generated; Step 4: Based on the abnormal analysis signal, obtain the abnormal difference ratio Ycz and the stable difference ratio Wdc, perform ratio processing on the abnormal difference ratio Ycz and the stable difference ratio Wdc to obtain the equipment abnormality ratio Ycb, draw a cycle-equipment abnormality ratio Ycb change curve, connect the two ends of the curve to obtain a reference line, obtain the peak point and the trough point of the curve and connect them with the coordinate origin with a straight line, calculate the angle to be measured formed by the straight line and the reference line, and calculate with the threshold of the angle to be measured to obtain the angle offset ratio Jdb and the angle offset variance Fc, calculate the linear value Xz based on the angle offset ratio Jdb and the angle offset variance Fc, and if the linear value Xz ≥ the linear threshold Xzy, send an early warning to the system; Marking the target segment generating the image analysis signal as an abnormal segment; Obtain the difference between the segment abnormal value Yc and the segment abnormal threshold Ycz in the abnormal segment to obtain the segment abnormal difference; Obtain the section abnormal differences of all abnormal sections, sum and average the abnormal differences of all sections to obtain the mean of the abnormal differences; The mean of the abnormal difference is processed by ratio with the segment abnormal threshold value Ycz to obtain the abnormal difference ratio, and the abnormal difference ratio is marked as Ycz; The component stability value Wd is processed with the component stability threshold value Wdz to obtain the component stability difference; Calculate the ratio of the component stability difference to the component stability threshold Wdz to obtain the stability difference ratio, and mark the stability difference ratio as Wdc; The stable difference ratio Wdc is processed with the abnormal difference ratio Ycz to obtain the equipment abnormality ratio Ycb; Obtain the equipment abnormality ratio Ycb of multiple monitoring cycles, and draw the cycle-equipment abnormality ratio Ycb change curve in a two-dimensional rectangular coordinate system; Connect the two ends of the cycle-equipment abnormality ratio Ycb change curve with a straight line to obtain a reference line; Get the peak and trough points of the cycle-equipment abnormality ratio Ycb change curve; Connect the peak point, the trough point and the coordinate origin with a straight line, obtain the angle between the straight line and the reference line, and get the angle to be measured; Perform difference processing on the angle to be measured and the angle threshold to be measured to obtain the angle offset difference; Obtain all angle offset differences on the cycle-equipment abnormality ratio Ycb change curve, sum and average all angle differences to obtain the average offset difference; The mean value of the offset difference is compared with the angle threshold to be measured to obtain the angle offset ratio, which is marked as Jdb; Put all the angle offset differences on the cycle-equipment abnormality ratio Ycb change curve into the angle offset difference data group, calculate the variance of all the angle offset differences in the data group, obtain the angle offset variance, and mark the angle offset variance as Fc; Based on the angle deviation ratio Jdb and the angle deviation variance Fc, the linear value Xz is calculated; By formula: , calculate the linear value Xz, where c1 and c2 are preset proportional coefficients; Compare the linear value Xz with the linear threshold Xzy; If the linear value Xz ≥ the linear threshold value Xzy, it indicates that the cycle-equipment abnormality ratio Ycb change curve presents a linear change, that is, the abnormal behavior of the equipment continues to deteriorate, and an early warning is sent to the system to strengthen manual intervention in the equipment to prevent the abnormal behavior of the equipment from continuing to deteriorate; If the linear value Xz is less than the linear threshold value Xzy, the maximum value of the equipment abnormality ratio Ycb on the cycle-equipment abnormality ratio Ycb change curve is obtained, the maximum value of the equipment abnormality ratio Ycb is sent to the system, and a prompt message is sent to the human; Embodiment 3, as Figure 2 As shown, a device abnormal behavior recognition system based on audio and video streams includes: Audio acquisition module: collects the audio decibel value emitted by the device during operation, pre-processes the audio decibel value, and obtains a digital audio stream; Audio analysis module: based on the digital audio stream, the detection spectrum is transformed, the horizontal axes of the detection spectrum and the target spectrum are divided into multiple sections to obtain the target section, the difference in the positions of the peaks of the detection spectrum and the target spectrum in the target section is obtained, and the peak shift ratio Py is further calculated to obtain the difference in the number of peaks of the detection spectrum and the target spectrum, and the peak number difference is calculated with the number of peaks of the target spectrum to obtain the peak number ratio Bf, and the section abnormal value Yc is calculated based on the peak shift ratio Py and the peak number ratio Bf. If the section abnormal value Yc ≥ the section abnormal threshold Ycz, an image analysis signal is generated; Video analysis module: Based on the image analysis signal, the video image of the equipment connection part is obtained, and the position offset ratio Py and the angle offset ratio Jd of the equipment connection part are calculated through the algorithm. Based on the position offset ratio Py and the angle offset ratio Jd, the component stability value Wd is calculated, and the component abnormal value Bj is less than the component stability threshold Wdz, and an abnormal analysis signal is generated; Abnormal judgment module: Based on the abnormal analysis signal, the section abnormal value Yc and the section abnormal threshold Ycz are calculated to obtain the abnormal difference ratio Ycz. Similarly, the stable difference ratio Wdc is obtained. The abnormal difference ratio Ycz is compared with the stable difference ratio Wdc to obtain the equipment abnormality ratio Ycb, which is plotted as a cycle-equipment abnormality ratio Ycb change curve. The two ends of the curve are connected to obtain a reference line, and the peak point and trough point of the curve are obtained and connected with the coordinate origin. The angle to be measured enclosed by the straight line and the reference line is calculated and the angle threshold to be measured is calculated to obtain the angle offset ratio Jdb and the angle offset variance Fc. Based on Jdb and Fc, the linear value Xz is calculated by the formula, and the linear value Xz is compared with the linear threshold Xzy. If Xz≥Xzy, an early warning is sent to the system.

[0023] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the present invention.

Claims

1. A method for identifying abnormal behavior of a device based on audio and video streams, characterized in that: include: Collecting the audio decibel value emitted by the device during operation, pre-processing the audio decibel value, and obtaining a digital audio stream; Based on the digital audio stream, the detection spectrum is transformed to obtain a detection spectrum diagram, the detection spectrum diagram and the target spectrum diagram are divided into multiple target segments and compared to obtain the peak shift ratio Py and the peak number ratio Bf, based on the peak shift ratio Py and the peak number ratio Bf, the segment abnormality value Yc is calculated, if the segment abnormality value Yc ≥ the segment abnormality threshold value Ycz, if the segment abnormality value Yc ≥ the segment abnormality threshold value Ycz, an image analysis signal is generated; Based on the image analysis signal, the video image of the equipment connection part is obtained, and the position offset ratio Py and the angle offset ratio Jd of the equipment connection part are calculated by the algorithm. Based on the position offset ratio Py and the angle offset ratio Jd, the component stability value Wd is calculated, and the component abnormal value Bj is less than the component stability threshold Wdz, and the abnormal analysis signal is generated; Based on the abnormal analysis signal, the abnormal difference ratio Ycz and the stable difference ratio Wdc are obtained. The abnormal difference ratio Ycz is ratio-processed with the stable difference ratio Wdc to obtain the equipment abnormality ratio Ycb, which is plotted as a cycle-equipment abnormality ratio Ycb change curve. The two ends of the curve are connected to obtain a reference line. The peak points and trough points of the change curve are obtained and connected to the coordinate origin with a straight line. The angle to be measured enclosed by the straight line and the reference line is calculated, and the angle threshold to be measured is calculated to obtain the angle offset ratio Jdb and the angle offset variance Fc. The linear value Xz is calculated by the formula. If the linear value Xz ≥ the linear threshold Xzy, an early warning is sent to the system.

2. The method for identifying abnormal behavior of a device based on audio and video streams according to claim 1, characterized in that: The segment abnormal value Yc is obtained as follows: Based on the peak offset ratio Py and the peak number ratio Bf, the segment outlier value Yc is calculated using a weighted formula.

3. The method for identifying abnormal behavior of a device based on audio and video streams according to claim 2, characterized in that: The peak shift ratio Py is obtained as follows: Perform fast Fourier transform on the digital audio stream to convert the time domain signal of the digital audio stream into a frequency domain signal. Based on the frequency domain signal, draw a detection spectrum with frequency as the horizontal axis and amplitude as the vertical axis; Dividing the horizontal axes of the detection spectrum graph and the target spectrum graph into a plurality of segments to obtain a target segment; Obtain the peak position difference between the detection spectrum graph and the target spectrum graph in the target section to obtain the peak offset difference; The peak shift difference is ratioed to the shift difference threshold to obtain a peak shift ratio, which is marked as Py.

4. The method for identifying abnormal behavior of a device based on audio and video streams according to claim 2, characterized in that: The peak number ratio Bf is obtained as follows: Obtain the number of peaks of the detection spectrum graph and the target spectrum graph in the target section, perform difference processing on the number of peaks of the detection spectrum graph and the target spectrum graph, and take the absolute value to obtain the difference in the number of peaks; The peak number difference is processed by ratio processing with the peak number of the target spectrum graph to obtain the peak number ratio, and the peak number ratio is marked as Bf.

5. The method for identifying abnormal behavior of a device based on audio and video streams according to claim 1, characterized in that: The component stability value Wd is obtained as follows: Based on the position deviation ratio Py and the angle deviation ratio Jd, the component stability value Wd is calculated; By formula: , the component stability value Wd is calculated, where ln(b1*Py+b2*Jd+1.011) is a logarithmic function with base e, and b1 and b2 are preset proportional coefficients.

6. The method for identifying abnormal behavior of a device based on audio and video streams according to claim 5, characterized in that: The position offset ratio Py is obtained as follows: During the monitoring period, video images of the equipment connection parts are obtained; Get the total number of video image frames, and mark the total number of video frames as n; The positional relationship of the device connection parts of each frame of the video is used to calculate the position change difference of the center coordinates of the connection parts, and the position change difference is marked as Bh i , i=1,2,.....,n-1, n is the total number of video image frames; The YOLOv8 algorithm is used to calculate the coordinates of the center of the connection part of each frame of the video (x i ,y i ), the image position coordinates of adjacent frames (x i+1 ,y i+1 ); Calculate the offset in the X-axis direction: , the offset in the Y-axis direction: , through the distance formula: , calculate the position offset Bh1; The position change difference Bh i Performing summation and averaging to obtain a position difference mean, calculating the difference between the position difference mean and the position difference threshold to obtain a position offset difference; The offset difference is processed by ratio with the position difference threshold to obtain the position offset ratio, which is marked as Py; The angle deviation ratio Jd is obtained as follows: Based on the X-axis and Y-axis offsets, the position offset angle θ is calculated using the inverse tangent function. i , i=1,2,.....,n-1, n is the total number of video image frames; The position is offset by an angle θ i Perform summation and mean processing to obtain the mean of the angle difference; The angle difference mean is compared with the angle difference threshold to obtain an angle deviation ratio, and the calculated angle deviation ratio is marked as Jd.

7. The method for identifying abnormal behavior of a device based on audio and video streams according to claim 1, characterized in that: The linear value Xz is obtained as follows: Based on the angle deviation ratio Jdb and the angle deviation variance Fc, the linear value Xz is calculated; By formula: , the linear value Xz is calculated, where c1 and c2 are preset proportional coefficients.

8. The method for identifying abnormal behavior of a device based on audio and video streams according to claim 7, characterized in that: The angle deviation ratio Jdb is obtained as follows: Performing difference processing on the angle to be measured and the angle threshold to be measured to obtain the angle offset difference; Obtain all angle offset differences on the cycle-equipment abnormality ratio Ycb change curve, sum and average all angle differences to obtain the offset difference mean; The mean value of the offset difference is compared with the angle threshold to be measured to obtain the angle offset ratio, which is marked as Jdb.

9. The method for identifying abnormal behavior of a device based on audio and video streams according to claim 8, characterized in that: The angle offset variance is obtained as follows: Put all the angle offset differences on the cycle-equipment abnormality ratio Ycb change curve into the angle offset difference data group, calculate the variance of all the angle offset differences in the data group, obtain the angle offset variance, and mark the angle offset variance as Fc.

10. The method for identifying abnormal behavior of a device based on audio and video streams according to claim 9, characterized in that: The method for obtaining the angle to be measured is: Marking the target segment generating the image analysis signal as an abnormal segment; Obtain the difference between the segment abnormal value Yc and the segment abnormal threshold Ycz in the abnormal segment to obtain the segment abnormal difference; Obtain the section abnormal differences of all abnormal sections, sum and average the abnormal differences of all sections, and obtain the mean of the abnormal differences; The mean of the abnormal difference is processed by ratio with the segment abnormal threshold value Ycz to obtain the abnormal difference ratio, and the abnormal difference ratio is marked as Ycz; The component stability value Wd is processed with the component stability threshold value Wdz to obtain the component stability difference; Calculate the ratio of the component stability difference to the component stability threshold Wdz to obtain the stability difference ratio, and mark the stability difference ratio as Wdc; The stable difference ratio Wdc is processed with the abnormal difference ratio Ycz to obtain the equipment abnormality ratio Ycb; Obtain the equipment abnormality ratio Ycb of multiple monitoring cycles, and draw the cycle-equipment abnormality ratio Ycb change curve in a two-dimensional rectangular coordinate system; Connect the two ends of the cycle-equipment abnormality ratio Ycb change curve with a straight line to obtain a reference line, and obtain the peak point and trough point of the cycle-equipment abnormality ratio Ycb change curve; Connect the peak point, trough point and coordinate origin with a straight line, obtain the angle enclosed by the straight line and the reference line, and get the angle to be measured.

11. A device abnormal behavior recognition system based on audio and video streams, characterized by: include: Audio acquisition module: collects the audio decibel value emitted by the device during operation, pre-processes the audio decibel value, and obtains a digital audio stream; Audio analysis module: based on the digital audio stream, the detection spectrum is transformed to obtain the detection spectrum and the target spectrum, and the detection spectrum and the target spectrum are divided into multiple target segments and compared to obtain the peak shift ratio Py and the peak number ratio Bf. Based on the peak shift ratio Py and the peak number ratio Bf, the segment abnormal value Yc is calculated. If the segment abnormal value Yc ≥ the segment abnormal threshold Ycz, if the segment abnormal value Yc ≥ the segment abnormal threshold Ycz, an image analysis signal is generated; Video analysis module: Based on the image analysis signal, the video image of the equipment connection part is obtained, and the position offset ratio Py and the angle offset ratio Jd of the equipment connection part are calculated through the algorithm. Based on the position offset ratio Py and the angle offset ratio Jd, the component stability value Wd is calculated, and the component abnormal value Bj is less than the component stability threshold Wdz, and an abnormal analysis signal is generated; Abnormal judgment module: Based on the abnormal analysis signal, the section abnormal value Yc and the section abnormal threshold Ycz are calculated to obtain the abnormal difference ratio Ycz. Similarly, the stable difference ratio Wdc is obtained. The abnormal difference ratio Ycz and the stable difference ratio Wdc are ratio-processed to obtain the equipment abnormality ratio Ycb, which is plotted as a cycle-equipment abnormality ratio Ycb change curve. The two ends of the curve are connected to obtain a reference line, and the peak point and trough point of the change curve are obtained and connected to the coordinate origin. The angle to be measured enclosed by the straight line and the reference line and the angle threshold to be measured are calculated to obtain the angle offset ratio Jdb and the angle offset variance Fc. The linear value Xz is calculated by the formula. If the linear value Xz ≥ the linear threshold Xzy, an early warning is sent to the system.

Images