System and method for safety detection of operation and maintenance personnel

Abstract

The invention discloses a system and a method for safety detection of operation and maintenance personnel, and relates to the technical field of safety, and the system comprises a data acquisition module, an audio and video data processing unit, a single-mode detection module, a fusion module and the like. The data acquisition module is used for acquiring video data and audio data in a monitoring area; the audio and video data processing unit preprocesses the video data and the audio data and extracts audio features and video features; an action classification model and an audio model are arranged in the single-mode detection module, and the single-mode detection module is used for carrying out independent classification identification and processing on video features and audio features; the fusion module is used for analyzing the classification and recognition result of the single-mode detection module on a result level and making an optimal decision; the invention further provides a method. According to the method, the detection effect of a single model is improved through a dynamic decision fusion technology, and unsafe behaviors or states can be accurately identified.

Description

Technical Field

[0001] The present invention relates to the field of safety technology, and in particular to a system and method for safety detection of operation and maintenance personnel. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and may not constitute prior art.

[0003] With the rapid development of intelligent manufacturing and intelligent robot technology, the demand for security monitoring in various complex environments is increasing. Traditional security monitoring methods for operation and maintenance personnel usually rely on a single audio or video data source, which has problems such as incomplete information and low detection accuracy, and it is difficult to meet the needs of intelligent security monitoring.

[0004] At the operation and maintenance site, there are complex environments and a large number of people, so paying attention to the safety of operation and maintenance personnel has become a top priority. From the video modality analysis, the unsafe conditions of operation and maintenance personnel can be divided into five unsafe conditions. (1) Falling: Due to uneven, slippery or obstacles on the ground, people may fall or slip, resulting in injuries; (2) Collision: When operating machinery or moving heavy objects, people may be hit or pinched; (3) Electric shock: When operating electrical equipment, electric shock may occur due to improper operation or equipment failure; (4) Cutting or stabbing: When using sharp tools or mechanical equipment, cutting or stabbing may occur. (5) Falling: When performing maintenance or high-altitude operations, people may cause the risk of falling from a height; From the audio modality analysis of the unsafe conditions of personnel, it can be divided into two unsafe auditory states. (1) Semantic sounds: The cries for help when people encounter an emergency, such as "help" or "help", are important signals for identifying accidents; (2) Non-semantic sounds: screams, shouts, rapid breathing, etc., indicating that the person is currently in a state of fright, pain or emergency.

[0005] In the production process, mistakes made by maintenance personnel are often inevitable. Usually, cameras are used to monitor the movements of maintenance personnel to determine whether they are in a safe state in a timely manner. However, in practical applications, the safety monitoring system for maintenance personnel that relies on a single data source is easily affected by changes in the external environment, which may lead to high false alarm rates or serious underreporting problems. By introducing the audio modality, the lack of information in the video modality can be compensated. In the case of poor lighting conditions or occlusion, video surveillance may not be able to accurately capture the movements of maintenance personnel, while audio data can provide additional clues, such as cries for help or abnormal sounds of personnel, thereby improving the comprehensiveness and accuracy of detection. However, the audio modality may face challenges in noisy environments because background noise may interfere with the recognition of cries for help. Although the combination of audio and video modalities can enhance the detection capabilities of the system, the influence of factors such as environmental noise and lighting still needs to be considered to maximize the accuracy and reliability of detection.

[0006] Therefore, in order to meet the needs of the development of intelligent monitoring technology for operation and maintenance sites, the present invention proposes a system and method for safety detection of operation and maintenance personnel. Summary of the invention

[0007] The purpose of the present invention is to provide a system and method for safety detection of operation and maintenance personnel in response to the problems existing in the prior art. Through dynamic decision fusion technology, the detection effect of a single model is improved, and unsafe behaviors or states are accurately identified. In addition, the system integrates a sound source localization function. When the microphone detects an abnormal sound, the camera will automatically adjust its direction to monitor and capture images of areas where danger may exist.

[0008] The technical solution of the present invention is as follows:

[0009] A system for safety detection of operation and maintenance personnel, comprising:

[0010] A data acquisition module, the data acquisition module is used to acquire video data and audio data in the monitoring area;

[0011] An audio and video data processing unit, which pre-processes the video data and the audio data to extract audio features and video features;

[0012] A single-mode detection module, wherein the single-mode detection module has a built-in action classification model and an audio model, and is used to classify, identify and process video features and audio features separately;

[0013] The fusion module is used to analyze the classification and recognition results of the single-modality detection module at the result level to make the best decision.

[0014] Furthermore, it also includes:

[0015] Alarm and response module: When an abnormal event is detected, the alarm and response module will automatically trigger an alarm and notify relevant operation and maintenance personnel.

[0016] Furthermore, the data acquisition module includes: a microphone array and a camera; the microphone array and the camera are both deployed in the monitoring area;

[0017] The microphone array is used to collect audio data in the monitoring area, and uses a multi-channel audio interface to transmit the audio data of each microphone to the pre-processing module;

[0018] The camera is used to collect video data in the monitoring area and transmit the video data to the preprocessing module using an interface;

[0019] During audio data transmission, timestamps are used to ensure time synchronization of all microphones;

[0020] During video data transmission, timestamps are used to ensure time synchronization between video data and audio data.

[0021] Furthermore, in the audio and video data processing unit, for the audio data, the frequency spectrum is extracted using the audio signal processing technology to show the frequency distribution characteristics of the sound; at the same time, the Mel frequency cepstral coefficient characteristics are calculated;

[0022] In the audio and video data processing unit, for video data, a deep learning algorithm is used to automatically detect and extract 17 key points of the human body in the video frame. Each key point is represented by x and y coordinates and a confidence score.

[0023] Furthermore, the audio model includes: an automatic speech recognition model, a text classification model and a sound event detection model;

[0024] The audio features extracted by the audio and video data processing unit are divided into: semantic audio features and non-semantic audio features;

[0025] For audio features with semantics, the automatic speech recognition model outputs the corresponding text, and then the text classification model is used to derive the probability of the degree of the voice calling for help, that is, the possibility of the audio model judging that the sound is abnormal;

[0026] For non-semantic audio features, the probability of the non-semantic category is obtained through classification by the sound event detection model, that is, the possibility of the audio model judging that the sound is abnormal.

[0027] Furthermore, the automatic speech recognition model includes: taking a basic model trained on large-scale data, adding a fine-tuning model to the attention layer and the fully connected layer of the basic model to reduce the computing resources and time of training, and obtaining the automatic speech recognition model.

[0028] Furthermore, the action classification model includes:

[0029] A long short-term memory recurrent neural network architecture is used to predict whether the operation and maintenance personnel will fall. A classifier is used to classify and judge the data transmitted by the long short-term memory recurrent neural network, and the judgment results include two categories: fall and non-fall. The obtained key point coordinate sequence is input into the long short-term memory recurrent neural network to determine whether the current posture is in a falling state, and the posture classification result and fall confidence are output, that is, the action classification model determines the possibility of abnormal action.

[0030] Furthermore, the fusion module has a built-in audio-visual fusion network;

[0031] When the fall confidence output by the action classification model is greater than the threshold, the fusion module directly outputs the detection result of the action classification model;

[0032] When the fall confidence output by the action classification model is less than or equal to the threshold, the audio-visual fusion network is enabled and combined with the audio information to obtain the final detection result. In the audio-visual fusion network, the output probability of the single-modal detection module is integrated through the Bayesian late fusion algorithm to obtain the final probability as the output result. During the fusion process, the specific fusion weight of each sample is estimated separately to achieve a more accurate and efficient fusion result. The calculation process of the fusion score is as follows:

[0033]

[0034] in:

[0035] The fall detection result is V, and the audio classification result is A, where S indicates that the operation and maintenance personnel are in a dangerous state, and P(S|V,A) represents the final probability of judging danger.

[0036] Furthermore, it also includes: a sound source localization module, which determines the specific position of the microphone sound source through the time difference.

[0037] A method for safety detection of operation and maintenance personnel, based on the above-mentioned system for safety detection of operation and maintenance personnel, comprises:

[0038] Step S1: Acquire video data and audio data in the monitoring area;

[0039] Step S2: preprocessing the video data and the audio data to obtain audio features and video features;

[0040] Step S3: using the action classification model and the audio model, the video features and the audio features are separately classified, identified and processed;

[0041] Step S4: Analyze the classification and recognition results at the result level and make the best decision;

[0042] Step S5: Determine the specific location of the sound source through the time difference;

[0043] The step S1 comprises:

[0044] Collect audio data in the monitoring area through a microphone array;

[0045] Collect video data in the monitoring area through cameras;

[0046] The step S2 comprises:

[0047] For audio data, use audio signal processing technology to extract its spectrum to show the frequency distribution characteristics of the sound; at the same time, calculate the Mel frequency cepstral coefficient characteristics and determine whether it is a semantic audio feature;

[0048] For video data, a deep learning algorithm is used to automatically detect and extract 17 key points of the human body in the video frame. Each key point is represented by x and y coordinates and a confidence score.

[0049] The step S3 comprises:

[0050] For audio features with semantics, the automatic speech recognition model outputs the corresponding text, and then the text classification model is used to derive the probability of the degree of the voice calling for help, that is, the possibility of the audio model judging that the sound is abnormal;

[0051] For non-semantic audio features, the probability of the non-semantic category is obtained through classification by the sound event detection model, that is, the possibility of the audio model judging the sound as abnormal;

[0052] The obtained key point coordinate sequence is input into the long-term and short-term recurrent neural network to determine whether the current posture is in a falling state, and the posture classification result and fall confidence are output, that is, the action classification model determines the possibility of abnormal action;

[0053] The step S4 comprises:

[0054] When the fall confidence output by the action classification model is greater than the threshold, the fusion module directly outputs the detection result of the action classification model;

[0055] When the fall confidence output by the action classification model is less than or equal to the threshold, the audio-visual fusion network is enabled to combine the audio information to obtain the final detection result; in the audio-visual fusion network, the output probability of the single-modal detection module is integrated through the Bayesian late fusion algorithm to obtain the final probability as the output result;

[0056] The step S5 comprises:

[0057] Step S51: measuring the time delay values ​​of the same sound source reaching different microphones according to the position of the microphone array deployed in the monitoring area, thereby calculating the distance difference between the sound source and each microphone;

[0058] Step S52: using the deployment of the microphone array and the distance difference to accurately calculate the location of the sound source;

[0059] The time it takes for a sound wave to reach the i-th microphone from the sound source is:

[0060] Among them, ||sm i || represents the Euclidean distance from the sound source to the i-th microphone, and c represents the speed of sound;

[0061] For the sound source location calculation, the time difference of multiple pairs of microphones is used to establish the equation system:

[0062] Δt ij ·c=||sm i ||-||sm j ||

[0063] Among them, Δt ij c represents the difference in distance from the sound source to microphone i and microphone j.

[0064] Compared with the prior art, the present invention has the following beneficial effects:

[0065] 1. In the audio processing part, the present invention distinguishes between audio with semantics and audio without semantics, and uses different models for detection, thereby effectively improving the ability to recognize abnormal sounds (such as cries for help, alarms, etc.).

[0066] 2. The present invention adopts a dynamic decision-making mechanism to dynamically select whether to use audio information based on visual confidence, so that the system can be adaptively adjusted to improve the accuracy and robustness of detection.

[0067] 3. The present invention combines sound source localization technology to further improve the ability to locate the location of abnormal events, providing more accurate information support for subsequent emergency response. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a system block diagram for safety detection of operation and maintenance personnel;

[0069] Figure 2 It is a flow chart of a method for safety detection of operation and maintenance personnel;

[0070] Figure 3 Schematic diagram of the data collection process. DETAILED DESCRIPTION

[0071] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0072] The features and performance of the present invention are further described in detail below in conjunction with the embodiments.

[0073] Embodiment 1

[0074] First of all, it should be explained that the specific implementation platform in this embodiment is: the operating system is Ubuntu24.04, the programming language is python3, the neural network framework is Pytroch, the CUDA version is 12.1, and the computing acceleration unit is NVIDIA 3090TiGPU.

[0075] See also Figure 1 , a system for safety detection of operation and maintenance personnel, comprising:

[0076] A data acquisition module, the data acquisition module is used to acquire video data and audio data in the monitoring area;

[0077] An audio and video data processing unit, which pre-processes the video data and the audio data to extract audio features and video features;

[0078] A single-mode detection module, wherein the single-mode detection module has a built-in action classification model and an audio model, and is used to classify, identify and process video features and audio features separately;

[0079] A fusion module, which is used to analyze the classification and recognition results of the single-modality detection module at the result level to make an optimal decision;

[0080] Alarm and response module: When an abnormal event is detected, the alarm and response module will automatically trigger an alarm and notify relevant operation and maintenance personnel.

[0081] In this embodiment, specifically, the data acquisition module includes: a microphone array and a camera; the microphone array and the camera are both deployed in the monitoring area;

[0082] The microphone array is used to collect audio data in the monitoring area, and uses a multi-channel audio interface to transmit the audio data of each microphone to the pre-processing module;

[0083] The camera is used to collect video data in the monitoring area. The camera has a collection frame rate of 25 frames per second, and the video data is transmitted to the preprocessing module using an interface;

[0084] During audio data transmission, timestamps are used to ensure the time synchronization of all microphones for subsequent sound source localization and fusion processing;

[0085] During the video data transmission process, timestamps are used to ensure the time synchronization of video data and audio data, so as to facilitate the subsequent multimodal fusion processing;

[0086] That is, in order to capture the safety information of the scene in all directions, this embodiment first deploys an array of multiple microphones in the monitoring area. These microphones ensure that the sound signals of the operation and maintenance personnel from all directions can be captured. At the same time, a high-definition camera is installed synchronously to ensure that the video coverage is completely consistent with the area monitored by the microphone array, and the video images of the scene are captured in real time to provide a data source for subsequent audio-visual fusion analysis. All collected video data and audio data are transmitted to the audio and video data processing unit in real time to provide data support for subsequent processing.

[0087] In this embodiment, specifically, in the audio and video data processing unit, for audio data, the frequency spectrum is extracted using audio signal processing technology to show the frequency distribution characteristics of the sound; at the same time, the Mel Frequency Cepstral Coefficient feature (MFCC) is calculated. This feature can effectively describe the sound quality information of the sound, which is of great significance for speech recognition and sound event classification;

[0088] In the audio and video data processing unit, for video data, a deep learning algorithm is used to automatically detect and extract 17 key points of the human body in the video frame. Each key point is represented by x and y coordinates and a confidence score. These key points cover the main joints and parts of the human body, providing key information for subsequent posture analysis and behavior recognition.

[0089] Specifically,

[0090] (1) Audio feature extraction:

[0091] When the audio data passes through the MFCC process, it is first pre-emphasized to highlight the high-frequency components, and then the signal is framed and windowed to reduce spectral leakage. Secondly, the fast Fourier transform (FFT) is applied to convert the time domain signal into a frequency domain signal, and the power spectrum of each frame is calculated. Thirdly, the energy on the Mel frequency scale is extracted through the Mel filter bank. After logarithmic transformation of these energies, the cepstral coefficients are extracted through discrete cosine transform (DCT), and these coefficients are used as audio features for further analysis; in this embodiment, it is extracted every 10 milliseconds, and each input vector consists of MFCC and energy features of 10 consecutive frames;

[0092] (2) Video feature extraction:

[0093] The posture estimation model is used to detect each frame of human motion and obtain a sequence of 17 key points of the human body, including the nose, right shoulder, left shoulder, etc. These key points are connected according to the natural connection rules of the human body and connected in the shape of an inverted pendulum to construct a five-point inverted pendulum model of the unstable process of human motion. This model can effectively reflect the changes in motion posture and relatively simplify the human body structure, avoiding the interference of the heterogeneity and randomness of multiple key points of the human body on posture analysis;

[0094] Specifically, each frame of the video is detected as a separate picture, and the posture estimation model is used to obtain the posture estimation results of each frame, record the coordinates and confidence of each key point, and calculate the angle transformation and distance features between the key points to obtain the motion characteristics of the human body; it should be noted that the input of the posture estimation model includes: an image with three color channels, a neural network-based encoder to generate PIF (a head network that predicts joint confidence, precise position and size) and PAF (a head network that predicts the association between components). The decoder can convert PIF and PAF into posture estimates, and each posture contains 17 key points. Each key point is represented by x and y coordinates and a confidence score.

[0095] In this embodiment, specifically, the audio model includes: an automatic speech recognition model, a text classification model and a sound event detection model;

[0096] The audio features extracted by the audio and video data processing unit are divided into: semantic audio features and non-semantic audio features. Among them, semantic audio features, such as the words of help when people encounter an emergency, such as "help" or "help", are important signals for identifying accidents; non-semantic audio features, such as screams, shouts, rapid breathing, etc., indicate that the current person is in a state of fright, pain or emergency;

[0097] For audio features with semantics, the corresponding text is output through the automatic speech recognition model, and then the text classification model is used to determine whether the text contains dangerous content (such as distress signals, threatening language, etc.). The text classification model returns the probability that the text is "dangerous" or "non-dangerous", thereby deriving the probability of the degree of the voice call for help, that is, the possibility of the audio model judging the sound abnormality; in this embodiment, specifically, for audio features with semantics, first obtain the current audio data, obtain the audio features through the logarithmic Mel spectrogram, and transmit the features to the encoder-decoder to realize text word-by-word recognition. After that, an activation function is used to predict the output words, and finally the output text is transmitted to the text classification model to predict the text, obtain the classification results and probability distribution, and return the classification results and probabilities in JSON format;

[0098] For non-semantic audio features, the probability of non-semantic categories is obtained through classification by the sound event detection model, that is, the possibility of the audio model judging the sound abnormality; in this embodiment, the sound event detection model adopts a bidirectional long short-term memory network; that is, the current audio signal (that is, audio data) is first obtained, the audio signal passes through a preprocessing step, and then MFCC feature extraction is performed, and the obtained feature data is transmitted to the trained sound event detection model, the sound event detection model processes the input features through forward propagation, and extracts higher-level feature representations layer by layer. At the output layer, the sound event detection model classifies the input features and outputs the probability of screaming, footsteps or noise.

[0099] In this embodiment, specifically, the automatic speech recognition model includes: taking a basic model trained on large-scale data, adding a fine-tuning model to the attention layer and the fully connected layer of the basic model, that is, selecting a suitable rank to weigh the computational efficiency and model performance process, so as to reduce the computational resources and time of training, and obtain the automatic speech recognition model;

[0100] In this embodiment, the rank is dynamically adjusted to meet the needs of the training process. In the initial stage, a higher rank is used to quickly learn features, and then gradually reduced to the target rank to improve efficiency;

[0101] That is, this embodiment uses a pre-trained automatic speech recognition model and a corresponding tokenizer to implement it; wherein the output y of the basic model can be expressed as: y=Wx, wherein W represents a weight matrix, and x represents an input vector; the automatic speech recognition model with the fine-tuning model added is expressed as: y′=W′x=(W+BA)x=Wx+B(Ax), wherein A and B are two low-rank matrices introduced by the fine-tuning model, so that the change ΔW of W can be represented by A and B, and thus, in the fine-tuning process, the original weight matrix W is not updated, and only the parameters of the low-rank matrices A and B are updated;

[0102] That is, in this embodiment, the dimensions of the initial low-rank matrices A and B are (d, 12) and (12, d), respectively, and the dimensions of the final target low-rank matrices A and B during the training process are (d, 4) and (4, d), respectively.

[0103] In this embodiment, specifically, the action classification model includes:

[0104] A long short-term memory recurrent neural network architecture is used to predict whether the operation and maintenance personnel have fallen. A classifier is used to classify and judge the data transmitted by the long short-term memory recurrent neural network, and the judgment results include two categories: fall and non-fall. The obtained key point coordinate sequence is input into the long short-term memory recurrent neural network to determine whether the current posture is in a falling state, and the posture classification result and fall confidence are output, that is, the action classification model determines the possibility of abnormal action.

[0105] In this embodiment, the features are input into the action classification model, 30 consecutive frames of pictures are set to capture the complete action process, the action is judged, and the current probability value of falling is output;

[0106] In this embodiment, regarding the action classification model, a recurrent network model is defined, which includes an LSTM layer and a fully connected layer. The recurrent network model can remember and analyze long-term sequence information, thereby accurately capturing the dynamic changes of human body movements. The fully connected layer is used to output the classification results, classify different action categories according to the extracted features, and determine whether a fall has occurred.

[0107] In this embodiment, specifically, the fusion module has a built-in audio-visual fusion network;

[0108] When the fall confidence output by the action classification model is greater than the threshold, the fusion module directly outputs the detection result of the action classification model;

[0109] When the fall confidence output by the action classification model is less than or equal to the threshold, that is, when the action classification model cannot make a clear judgment alone, the audio-visual fusion network is enabled to combine the audio information to obtain the final detection result; by integrating visual and auditory information and using advanced fusion algorithms, a more accurate judgment of potential dangerous situations can be made;

[0110] In the audio-visual fusion network, the output probabilities of the single-modality detection modules are integrated through the Bayesian late fusion algorithm to obtain the final probability as the output result;

[0111] It should be noted that the general late fusion method pre-given a fixed weight to each individual classifier. This method is based on the assumption that all samples of the classifier have equal weights, thereby ignoring the differences in the predictive ability of the classifier on individual samples. However, in reality, each classifier often exhibits different predictive effectiveness for different samples. Therefore, in this embodiment, during the fusion process, the specific fusion weight of each sample is estimated separately to achieve a more accurate and efficient fusion result; the calculation process of the fusion score is as follows:

[0112]

[0113] in:

[0114] The fall detection result is V, and the audio classification result is A, where S indicates that the operation and maintenance personnel are in a dangerous state, and P(S|V,A) represents the final probability of judging danger.

[0115] In this embodiment, specifically, sound source localization technology is also added to the system, which can quickly determine the source of the cry for help or other dangerous sounds. Subsequently, the camera automatically adjusts the viewing angle, focuses on and confirms the actual location of the sound source, providing precise guidance for subsequent emergency response;

[0116] That is, it also includes: a sound source localization module, which determines the specific position of the microphone sound source through the time difference;

[0117] Specifically, firstly, according to the position of the microphone array deployed in the monitoring area, the time delay value of the same sound source reaching different microphones is measured, so as to calculate the distance difference between the sound source and each microphone;

[0118] Then, the location of the sound source is accurately calculated using the deployment and distance difference of the microphone array;

[0119] The time it takes for a sound wave to reach the i-th microphone from the sound source is:

[0120] Among them, ||sm i || represents the Euclidean distance from the sound source to the i-th microphone, and c represents the speed of sound;

[0121] For the sound source location calculation, the time difference of multiple pairs of microphones is used to establish the equation system:

[0122] Δt ij ·c=||sm i ||-||sm j ||

[0123] Among them, Δt ij c represents the difference in distance from the sound source to microphone i and microphone j.

[0124] In this embodiment, it should also be noted that the Bayesian late fusion algorithm combines the prediction results of multiple models through probability calculation without the need for additional training of the fusion part. The Bayesian formula is as follows:

[0125]

[0126] Among them, E i Represents each model M i Predict the sample and generate the predicted result. i |H) is expressed as the likelihood of generating the result under the assumption that H is true, P(H|E i ) represents the posterior probability.

[0127] Based on the above-mentioned system for safety detection of operation and maintenance personnel, this embodiment further proposes a method for safety detection of operation and maintenance personnel, which specifically includes the following steps:

[0128] Step S1: Acquire video data and audio data in the monitoring area;

[0129] Step S2: preprocessing the video data and the audio data to obtain audio features and video features;

[0130] Step S3: using the action classification model and the audio model, the video features and the audio features are separately classified, identified and processed;

[0131] Step S4: Analyze the classification and recognition results at the result level and make the best decision;

[0132] Step S5: Determine the specific location of the sound source through the time difference;

[0133] The step S1 comprises:

[0134] Collect audio data in the monitoring area through a microphone array;

[0135] Collect video data in the monitoring area through cameras;

[0136] The step S2 comprises:

[0137] For audio data, use audio signal processing technology to extract its spectrum to show the frequency distribution characteristics of the sound; at the same time, calculate the Mel frequency cepstral coefficient characteristics and determine whether it is a semantic audio feature;

[0138] For video data, a deep learning algorithm is used to automatically detect and extract 17 key points of the human body in the video frame. Each key point is represented by x and y coordinates and a confidence score.

[0139] The step S3 comprises:

[0140] For audio features with semantics, the automatic speech recognition model outputs the corresponding text, and then the text classification model is used to derive the probability of the degree of the voice calling for help, that is, the possibility of the audio model judging that the sound is abnormal;

[0141] For non-semantic audio features, the probability of the non-semantic category is obtained through classification by the sound event detection model, that is, the possibility of the audio model judging the sound as abnormal;

[0142] The obtained key point coordinate sequence is input into the long-term and short-term recurrent neural network to determine whether the current posture is in a falling state, and the posture classification result and fall confidence are output, that is, the action classification model determines the possibility of abnormal action;

[0143] The step S4 comprises:

[0144] When the fall confidence output by the action classification model is greater than the threshold, the fusion module directly outputs the detection result of the action classification model;

[0145] When the fall confidence output by the action classification model is less than or equal to the threshold, the audio-visual fusion network is enabled to combine the audio information to obtain the final detection result; in the audio-visual fusion network, the output probability of the single-modal detection module is integrated through the Bayesian late fusion algorithm to obtain the final probability as the output result;

[0146] The step S5 comprises:

[0147] Step S51: measuring the time delay values ​​of the same sound source reaching different microphones according to the position of the microphone array deployed in the monitoring area, thereby calculating the distance difference between the sound source and each microphone;

[0148] Step S52: using the deployment of the microphone array and the distance difference to accurately calculate the location of the sound source;

[0149] The time it takes for a sound wave to reach the i-th microphone from the sound source is:

[0150] Among them, ||sm i || represents the Euclidean distance from the sound source to the i-th microphone, and c represents the speed of sound;

[0151] For the sound source location calculation, the time difference of multiple pairs of microphones is used to establish the equation system:

[0152] Δt ij ·c=||sm i ||-||sm j ||

[0153] Among them, Δt ijc represents the difference in distance from the sound source to microphone i and microphone j.

[0154] The above-mentioned embodiments only express the specific implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the protection scope of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the technical solution concept of the present application, and these all belong to the protection scope of the present application.

[0155] This background section is provided to generally present the context of the invention, and the work of the presently named inventors, the work to the extent described in this background section, and aspects of this section that did not constitute prior art at the time of application are neither explicitly nor implicitly admitted to be prior art to the present invention.

Claims

1. A system for safety detection of operation and maintenance personnel, characterized in that: include: A data acquisition module, the data acquisition module is used to acquire video data and audio data in the monitoring area; An audio and video data processing unit, which pre-processes the video data and the audio data to extract audio features and video features; A single-mode detection module, wherein the single-mode detection module has a built-in action classification model and an audio model, and is used to classify, identify and process video features and audio features separately; The fusion module is used to analyze the classification and recognition results of the single-modality detection module at the result level to make the best decision.

2. A system for safety detection of operation and maintenance personnel according to claim 1, characterized in that: Also includes: Alarm and response module: When an abnormal event is detected, the alarm and response module will automatically trigger an alarm and notify relevant operation and maintenance personnel.

3. A system and method for safety detection of operation and maintenance personnel according to claim 1, characterized in that: The data acquisition module includes: a microphone array and a camera; the microphone array and the camera are both deployed in the monitoring area; The microphone array is used to collect audio data in the monitoring area, and uses a multi-channel audio interface to transmit the audio data of each microphone to the pre-processing module; The camera is used to collect video data in the monitoring area and transmit the video data to the preprocessing module using an interface; During audio data transmission, timestamps are used to ensure time synchronization of all microphones; During video data transmission, timestamps are used to ensure time synchronization between video data and audio data.

4. A system for safety detection of operation and maintenance personnel according to claim 1, characterized in that: In the audio and video data processing unit, for audio data, the frequency spectrum is extracted using audio signal processing technology to show the frequency distribution characteristics of the sound; at the same time, the Mel frequency cepstral coefficient characteristics are calculated; In the audio and video data processing unit, for video data, a deep learning algorithm is used to automatically detect and extract 17 key points of the human body in the video frame. Each key point is represented by x and y coordinates and a confidence score.

5. A system for safety detection of operation and maintenance personnel according to claim 1, characterized in that: The audio model includes: an automatic speech recognition model, a text classification model and a sound event detection model; The audio features extracted by the audio and video data processing unit are divided into: semantic audio features and non-semantic audio features; For audio features with semantics, the automatic speech recognition model outputs the corresponding text, and then the text classification model is used to derive the probability of the degree of the voice calling for help, that is, the possibility of the audio model judging that the sound is abnormal; For non-semantic audio features, the probability of the non-semantic category is obtained through classification by the sound event detection model, that is, the possibility of the audio model judging that the sound is abnormal.

6. A system for safety detection of operation and maintenance personnel according to claim 5, characterized in that: The automatic speech recognition model includes: taking a basic model trained on large-scale data, adding a fine-tuning model to the attention layer and the fully connected layer of the basic model to reduce the computing resources and time of training, and obtaining the automatic speech recognition model.

7. A system for safety detection of operation and maintenance personnel according to claim 1, characterized in that: The action classification model comprises: A long short-term memory recurrent neural network architecture is used to predict whether the operation and maintenance personnel will fall. A classifier is used to classify and judge the data transmitted by the long short-term memory recurrent neural network, and the judgment results include two categories: fall and non-fall. The obtained key point coordinate sequence is input into the long short-term memory recurrent neural network to determine whether the current posture is in a falling state, and the posture classification result and fall confidence are output, that is, the action classification model determines the possibility of abnormal action.

8. A system for safety detection of operation and maintenance personnel according to claim 7, characterized in that: The fusion module has a built-in audio-visual fusion network; When the fall confidence output by the action classification model is greater than the threshold, the fusion module directly outputs the detection result of the action classification model; When the fall confidence output by the action classification model is less than or equal to the threshold, the audio-visual fusion network is enabled to combine the audio information to obtain the final detection result; in the audio-visual fusion network, the output probability of the single-modal detection module is integrated through the Bayesian late fusion algorithm to obtain the final probability as the output result; During the fusion process, a specific fusion weight is estimated for each sample separately to achieve a more accurate and efficient fusion result. The calculation process of the fusion score is as follows: in: The fall detection result is V, and the audio classification result is A, where S indicates that the operation and maintenance personnel are in a dangerous state, and P(S|V,A) represents the final probability of judging danger.

9. A system for safety detection of operation and maintenance personnel according to claim 1, characterized in that: Also includes: A sound source localization module determines the specific position of the microphone sound source through the time difference.

10. A method for safety detection of operation and maintenance personnel, characterized in that: A system for safety detection of operation and maintenance personnel according to any one of claims 1 to 9, comprising: Step S1: Acquire video data and audio data in the monitoring area; Step S2: preprocessing the video data and the audio data to obtain audio features and video features; Step S3: using the action classification model and the audio model, the video features and the audio features are separately classified, identified and processed; Step S4: Analyze the classification and recognition results at the result level and make the best decision; Step S5: Determine the specific location of the sound source through the time difference; The step S1 comprises: Collect audio data in the monitoring area through a microphone array; Collect video data in the monitoring area through cameras; The step S2 comprises: For audio data, use audio signal processing technology to extract its spectrum to show the frequency distribution characteristics of the sound; at the same time, calculate the Mel frequency cepstral coefficient characteristics and determine whether it is a semantic audio feature; For video data, a deep learning algorithm is used to automatically detect and extract 17 key points of the human body in the video frame. Each key point is represented by x and y coordinates and a confidence score. The step S3 comprises: For audio features with semantics, the automatic speech recognition model outputs the corresponding text, and then the text classification model is used to derive the probability of the degree of the voice calling for help, that is, the possibility of the audio model judging that the sound is abnormal; For non-semantic audio features, the probability of the non-semantic category is obtained through classification by the sound event detection model, that is, the possibility of the audio model judging the sound as abnormal; The obtained key point coordinate sequence is input into the long-term and short-term recurrent neural network to determine whether the current posture is in a falling state, and the posture classification result and fall confidence are output, that is, the action classification model determines the possibility of abnormal action; The step S4 comprises: When the fall confidence output by the action classification model is greater than the threshold, the fusion module directly outputs the detection result of the action classification model; When the fall confidence output by the action classification model is less than or equal to the threshold, the audio-visual fusion network is enabled to combine the audio information to obtain the final detection result; in the audio-visual fusion network, the output probability of the single-modal detection module is integrated through the Bayesian late fusion algorithm to obtain the final probability as the output result; The step S5 comprises: Step S51: measuring the time delay values ​​of the same sound source reaching different microphones according to the position of the microphone array deployed in the monitoring area, thereby calculating the distance difference between the sound source and each microphone; Step S52: using the deployment of the microphone array and the distance difference to accurately calculate the location of the sound source; The time it takes for a sound wave to reach the i-th microphone from the sound source is: Among them, ||sm i || represents the Euclidean distance from the sound source to the i-th microphone, and c represents the speed of sound; For the sound source location calculation, the time difference of multiple pairs of microphones is used to establish the equation system: Δt ij ·c=||s-m i ||-||s-m j || Among them, Δt ij c represents the difference in distance from the sound source to microphone i and microphone j.

Images