Behavior analysis and instant intelligent voice warning method and system based on deep learning

Abstract

The invention relates to a behavior analysis and instant intelligent voice warning method and system based on deep learning, and belongs to the technical field of video monitoring, and the system comprises a data collection module, a feature extraction module, a behavior analysis module, an instruction text generation module, a voice synthesis and broadcast unit, and a multi-terminal early warning distribution network module. The data acquisition module comprises video monitoring equipment, audio capture equipment, an equipment state acquisition sensor, an equipment positioning system, an environment sensor and a personnel positioning system. According to the invention, the data acquisition module is arranged to ensure the time synchronization of video, image and voice data and provide high-quality multi-modal input data for subsequent deep learning analysis, and the feature extraction module is arranged to extract key spatial-temporal features from the video data and visual features from the image data. And acoustic features are extracted from the voice data, so that rich feature representation is provided for behavior analysis and anomaly judgment.

Description

Technical Field

[0001] The present invention relates to the field of video surveillance technology, and in particular to a method and system for behavior analysis and instant intelligent voice warning based on deep learning. Background Art

[0002] In the industrial production sector, safe production is a crucial component of enterprise management, and effective monitoring and early warning systems are key technical means to prevent accidents and ensure personnel safety. Currently, video surveillance systems in industrial production environments often face challenges such as complex environments, diverse sources of danger, and large amounts of monitored data. This makes traditional monitoring systems significantly inadequate in terms of real-time behavior analysis and emergency warning.

[0003] Similarly, in public places such as shopping malls, airports, and train stations, higher requirements are being placed on the intelligence and emergency response capabilities of video surveillance systems. These places are crowded and open, and the need for rapid identification of abnormal behavior and immediate warning is particularly urgent.

[0004] Although the application of deep learning technology in the above-mentioned fields has provided new possibilities for the intelligentization of video surveillance systems, existing technologies still have the following problems in terms of multimodal data fusion, behavior analysis accuracy, warning real-timeness, and system adaptability:

[0005] 1. Existing monitoring systems lack effective fusion strategies when processing multi-source heterogeneous data in industrial production environments, resulting in poor monitoring results;

[0006] 2. The accuracy and real-time performance of behavior recognition algorithms need to be improved when dealing with complex scenarios and dynamic changes in industrial production.

[0007] 3. The video surveillance system in public places lacks the function of instant voice warning and cannot quickly convey warning information in an emergency;

[0008] 4. The early warning system fails to make full use of multiple communication channels, resulting in low efficiency in the transmission of early warning information in industrial production and public places.

[0009] Aiming at the specific needs of industrial production and some public places, the present invention proposes a behavior analysis and instant intelligent voice warning method and system based on deep learning. Through advanced multimodal data fusion technology, accurate behavior recognition algorithm and real-time speech synthesis and broadcasting technology, the present invention provides a set of efficient and intelligent monitoring and early warning solutions for industrial production and public places, thereby significantly improving the management level of safe production and public safety. Summary of the Invention

[0010] The present invention provides a method and system for behavioral analysis and instant intelligent voice warning based on deep learning, which solves the problems raised by the above-mentioned existing background technology. It is suitable for security monitoring in industrial production and public places, and can significantly improve the management level of safe production and public safety.

[0011] The present invention solves the above technical problems with the following solution: a behavior analysis and instant intelligent voice warning system based on deep learning, comprising a data acquisition module, a feature extraction module, a behavior analysis module, an instruction text generation module, a speech synthesis and broadcasting unit, and a multi-terminal warning distribution network module. The data acquisition module includes a video monitoring device, an audio capture device, a device status acquisition sensor, a device positioning system, an environmental sensor, and a personnel positioning system.

[0012] The feature extraction module includes a deep learning model, a data center, and a central processing unit. The deep learning model algorithm is one or more of a convolutional neural network architecture, a recurrent neural network architecture, or a long short-term memory network architecture;

[0013] The behavioral analysis module includes deep learning classifiers and sequence prediction models;

[0014] The instruction text generation module uses a pre-trained generative adversarial network (GAN) or transformer model;

[0015] The speech synthesis and broadcasting unit includes a text-to-speech engine (TTS), an audio playback device, and a broadcasting system;

[0016] The multi-terminal warning distribution network module includes an in-station messaging system, a text messaging platform, and an email server;

[0017] The usage includes the following steps:

[0018] S1, data acquisition, the data acquisition module collects the target object's behavior video and pictures from the video surveillance equipment, and the audio capture device collects relevant voice data, and the device status acquisition sensor, device positioning system, environmental sensor, and personnel positioning system collect other multimodal data;

[0019] S2, feature extraction and fusion. The feature extraction module uses a deep learning algorithm to extract features from the collected multimodal data. The extracted features include the edges, textures, and shapes of the image. For the synchronously collected audio signals, a deep learning model with a convolutional neural network architecture or a recurrent neural network architecture is used to extract features. The Mel-frequency cepstral coefficients (MFCC) are used to combine the extracted features through the data center and central processing unit fusion algorithm to form a multimodal feature vector. The multimodal feature vector is input into the behavior analysis model, which uses a long short-term memory network structure to identify the behavior pattern of the target object.

[0020] S3, behavior analysis and anomaly judgment, inputs the extracted features into the deep learning classifier and sequence prediction model of the behavior analysis module to identify and classify behaviors, distinguish normal behaviors from abnormal or dangerous behaviors, and analyze the detailed type and severity of dangerous behaviors;

[0021] S4, warning instruction generation: The behavior analysis model outputs the behavior recognition results and determines whether there is abnormal behavior through a threshold judgment mechanism. When abnormal behavior is detected, the instruction text generation module triggers the warning instruction generation process. Based on the type and severity of the abnormal behavior, the instruction text generation module selects a corresponding warning template and generates the warning instruction text through natural language generation (NLG) technology;

[0022] S5, text-to-speech and timely voice broadcast: The speech synthesis and broadcast unit uses a text-to-speech engine (TTS) to convert the generated warning instruction text into a natural and fluent voice signal. The voice signal is transmitted to the broadcast system through the audio playback device in the monitoring area or through the API interface. Real-time broadcast ensures the effective transmission of warning information;

[0023] S6, Warning distribution to other terminals: The results of abnormal or dangerous behavior identification can be synchronized through the in-station message system, SMS platform and email server of the multi-terminal warning distribution network module. The multi-terminal warning distribution network can send warning information to management personnel and emergency response teams in real time through multiple channels such as in-station messages, SMS and email according to the preset warning level and distribution strategy. The multi-terminal warning distribution network supports confirmation feedback from the warning receiving end, so that the warning logic and distribution strategy can be adjusted according to the feedback information.

[0024] On the basis of the above technical solution, the present invention can also be improved as follows.

[0025] Furthermore, the video surveillance equipment and audio capture equipment collect the target object's behavioral video stream, static image data and corresponding voice signals in real time to form multimodal data. The video surveillance equipment uses a camera with no less than 400W pixels and the audio capture equipment uses a matching audio capture module to obtain the video stream and synchronous audio signal of the target area in real time. The video stream is collected at a rate of at least 30 frames per second, and the audio signal is captured at a sampling rate of 44.1kHz. The data acquisition module has a built-in time synchronization mechanism to ensure the time consistency of the video frame and the audio signal. The collected data is transmitted to the server, and the collected data is preprocessed, including denoising, contrast enhancement, and image scaling, to improve data quality and adapt to the input requirements of the deep learning model. This module ensures the time synchronization of video, image and voice data, and provides high-quality multimodal input data for subsequent deep learning analysis.

[0026] Furthermore, the deep learning model performs feature extraction on the collected multimodal data. The unit can extract key spatiotemporal features from video data, visual features from image data, and acoustic features from speech data, providing rich feature representation for behavior analysis and anomaly judgment.

[0027] Furthermore, the deep learning classifier is one or more of a support vector machine (SVM), a decision tree, a random forest or a neural network. The deep learning classifier and the sequence prediction model perform a comprehensive analysis of the extracted features to identify the behavior patterns of the target object and make abnormal behavior judgments. The behavior analysis module adopts advanced algorithms of support vector machine (SVM), random forest (RF) and recursive neural network (RNN) to ensure high-accuracy behavior recognition and anomaly detection with a low false alarm rate.

[0028] Furthermore, the instruction text generation module generates specific warning instruction text based on the behavior analysis results using natural language processing technology.

[0029] Furthermore, the text-to-speech engine (TTS) converts the warning instruction text into a natural and fluent voice signal, and realizes real-time broadcasting of voice warnings through the linkage of audio playback equipment and broadcasting system to quickly convey warning information.

[0030] Furthermore, the multi-terminal warning distribution network distributes abnormal or dangerous behavior identification results in real time to multiple warning channels such as the in-site message system, SMS platform and email server through secure communication protocols and interfaces.

[0031] Furthermore, the in-site message system, SMS platform and email server can adopt hierarchical management and personalized push to ensure the efficient transmission and reception of warning information.

[0032] The beneficial effects of the present invention are as follows: the present invention provides a method and system for behavior analysis and instant intelligent voice warning based on deep learning, which has the following advantages:

[0033] 1. The data acquisition module ensures the temporal synchronization of video, image, and voice data, providing high-quality multimodal input data for subsequent deep learning analysis. The feature extraction module extracts key spatiotemporal features from video data, visual features from image data, and acoustic features from voice data, providing rich feature representations for behavior analysis and anomaly judgment, thereby effectively improving monitoring effectiveness.

[0034] 2. The behavioral analysis module ensures highly accurate behavior recognition and low false alarm rate anomaly detection, making it applicable to complex scenarios and dynamic changes in industrial production, and improving the accuracy and real-time performance of behavior analysis;

[0035] 3. The command text generation module uses a pre-trained generative adversarial network or transformer model to generate clear, accurate, and emotionally expressive warning texts; a speech synthesis and broadcasting unit is provided to broadcast voice warnings in real time to quickly convey warning information;

[0036] 4. A multi-terminal early warning distribution network module is set up to support hierarchical management and personalized push of early warning information, ensuring the efficient transmission and reception of early warning information.

[0037] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and to implement it according to the contents of the description, the following preferred embodiments of the present invention are described in detail with reference to the accompanying drawings. The specific implementation methods of the present invention are given in detail by the following embodiments and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0039] Figure 1 A system flow chart of a method and system for behavior analysis and instant intelligent voice warning based on deep learning provided by one embodiment of the present invention;

[0040] Figure 2 This is a diagram showing the overall architecture of a method and system for behavior analysis and instant intelligent voice alerting based on deep learning, provided by one embodiment of the present invention;

[0041] Figure 3A flowchart of multi-source data collection and fusion of a method and system for deep learning-based behavior analysis and instant intelligent voice warning provided by one embodiment of the present invention;

[0042] Figure 4 A deep learning feature extraction network structure diagram of a method and system for deep learning-based behavior analysis and instant intelligent voice warning provided by one embodiment of the present invention;

[0043] Figure 5 A flowchart of a behavior analysis and abnormality judgment method and system for behavior analysis and instant intelligent voice warning based on deep learning provided by one embodiment of the present invention;

[0044] Figure 6 A multi-terminal warning flow chart of a method and system for behavior analysis and instant intelligent voice warning based on deep learning provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following is combined with Figure 1-6 The principles and features of the present invention are described, and the examples given are only for the purpose of explaining the present invention and are not intended to limit the scope of the present invention. The following paragraphs describe the present invention in more detail by way of example with reference to the accompanying drawings. The advantages and features of the present invention will become more apparent from the following description and claims. It should be noted that the drawings are in a very simplified form and are not in exact proportions, and are only used for the purpose of conveniently and clearly assisting in illustrating the embodiments of the present invention.

[0046] like Figure 1-2 As shown, the present invention provides a method and system for behavior analysis and instant intelligent voice warning based on deep learning, including a data acquisition module, a feature extraction module, a behavior analysis module, an instruction text generation module, a speech synthesis and broadcasting unit, and a multi-terminal warning distribution network. The data acquisition module includes a video monitoring device, an audio capture device, a device status acquisition sensor, a device positioning system, an environmental sensor, and a personnel positioning system. The video monitoring device and the audio capture device collect the target object's behavior video stream, static image data, and corresponding voice signals in real time to form multimodal data. This module ensures the time synchronization of video, image, and voice data, and provides high-quality multimodal input data for subsequent deep learning analysis.

[0047] The feature extraction module includes a deep learning model, a data center, and a central processing unit. The deep learning model uses a convolutional neural network architecture or a deep belief network architecture to extract features from the collected multimodal data. This unit can extract key spatiotemporal features from video data, visual features from image data, and acoustic features from speech data, providing rich feature representations for behavior analysis and anomaly judgment.

[0048] The behavior analysis module includes a deep learning classifier and sequence prediction model, which comprehensively analyzes extracted features to identify the target's behavioral patterns and determine abnormal behavior. This module uses advanced algorithms such as support vector machines (SVM), random forests (RF), and recurrent neural networks (RNN) to ensure highly accurate behavior recognition and low false positive rate anomaly detection.

[0049] The instruction text generation module uses a pre-trained generative adversarial network (GAN) or transformer model to generate clear, accurate, and emotionally expressive warning text. Based on the results of behavioral analysis, the instruction text generation module uses natural language processing technology to generate specific warning instruction text.

[0050] The speech synthesis and broadcasting unit includes a text-to-speech engine (TTS), an audio playback device, and a broadcasting system. The text-to-speech engine (TTS) can convert warning instruction text into natural and fluent voice signals. Through the linkage between the audio playback device and the broadcasting system, real-time voice warnings can be broadcast to quickly convey warning information.

[0051] Multi-terminal early warning distribution network, which includes the in-site message system, SMS platform and email server. The multi-terminal early warning distribution network distributes abnormal or dangerous behavior identification results to multiple early warning channels of the in-site message system, SMS platform and email server in real time through secure communication protocols and interfaces. The in-site message system, SMS platform and email server can adopt hierarchical management and personalized push to ensure the efficient transmission and reception of early warning information.

[0052] The specific working principle and method of use of the present invention are as follows:

[0053] S1, data acquisition, the system collects behavioral videos and pictures of the target object from the video surveillance equipment, collects relevant voice data by the audio capture device, and takes other multimodal data by the device status acquisition sensor, device positioning system, environmental sensor, and personnel positioning system. The video surveillance equipment uses a camera with no less than 400W pixels and the audio capture equipment uses a matching audio capture module to obtain the video stream and synchronous audio signal of the target area in real time. The video stream is collected at a rate of at least 30 frames per second, and the audio signal is captured at a sampling rate of 44.1kHz. The data acquisition module has a built-in time synchronization mechanism to ensure the time consistency of the video frame and the audio signal. The collected data is transmitted to the server and preprocessed on the collected data, including denoising, contrast enhancement, and image scaling, to improve data quality and adapt to the input requirements of the deep learning model;

[0054] like Figure 3-4As shown, S2, feature extraction, uses a deep learning algorithm to extract features from the collected multimodal data. The extracted features include the edge, texture and shape of the image. For the synchronously collected audio signal, a deep learning speech recognition model is used to extract features. Mel-frequency cepstral coefficients (MFCC) are used to combine the extracted features through the data center and central processing unit fusion algorithm to form a multimodal feature vector. The multimodal feature vector is input into the behavior analysis model. The model adopts a long short-term memory network structure to identify the behavior pattern of the target object. The deep learning algorithm is one or more of a convolutional neural network architecture, a recurrent neural network architecture or a long short-term memory network architecture.

[0055] like Figure 5 As shown, S3, behavior analysis and abnormality judgment, inputs the extracted features into the deep learning classifier and sequence prediction model to identify and classify the behavior, distinguish normal behavior from abnormal or dangerous behavior, and analyze the detailed type and severity of dangerous behavior; the deep learning classifier is one or more of support vector machine (SVM), decision tree, random forest or neural network;

[0056] The specific behavior analysis and abnormal classification judgment process within the factory is as follows:

[0057] 1. The multimodal data is transmitted to the behavior analysis module, which determines whether there is fire, thick smoke, or people falling down in the area. If so, it is judged as high risk. Otherwise, it proceeds to the next level of judgment;

[0058] 2. Determine whether there is anyone in the lifting restricted area. If yes, it is judged as high risk. If not, proceed to the next level of judgment;

[0059] 3. Determine whether the person is smoking, not wearing a helmet, etc. If so, it is considered medium risk; otherwise, it is considered low risk;

[0060] As shown in step 6, S4, voice warning instruction generation, the behavior analysis model outputs the behavior recognition result, and determines whether there is abnormal behavior through the threshold judgment mechanism. When abnormal behavior is detected, the instruction text generation module triggers the warning instruction generation process. According to the type and severity of the abnormal behavior, the instruction text generation module selects the corresponding warning template and generates the warning instruction text through natural language generation (NLG) technology;

[0061] S5, speech synthesis and broadcasting: Using a text-to-speech (TTS) engine, the generated warning instruction text is converted into a natural and fluent voice signal. The voice signal is transmitted to the broadcasting system through the audio playback device in the monitoring area or through the API interface. Real-time broadcasting ensures the effective transmission of warning information.

[0062] S6, other terminal warnings: Abnormal or dangerous behavior identification results can be synchronized through the station message system, SMS platform and email server. The abnormal behavior identification results are simultaneously sent to the station message system, SMS platform and email server multiple terminal warning distribution network. The multi-terminal warning distribution network can send warning information to management personnel and emergency response teams in real time through multiple channels such as station messages, SMS and email according to the preset warning level and distribution strategy. The multi-terminal warning distribution network supports confirmation feedback from the warning receiving end, so that the warning logic and distribution strategy can be adjusted according to the feedback information.

[0063] It should be noted that, in this document, relational terms such as first and second are used solely to distinguish one entity or operation from another, and do not necessarily require or imply any actual relationship or order between these entities or operations. Matters not described in detail in this specification are well known to those skilled in the art.

[0064] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any ordinary technician in this industry can smoothly implement the present invention as shown in the drawings and described above. However, any equivalent changes, modifications and evolutions made by technicians familiar with this profession without departing from the scope of the technical solution of the present invention using the technical content disclosed above are all equivalent embodiments of the present invention. At the same time, any equivalent changes, modifications and evolutions made to the above embodiments based on the essential technology of the present invention are still within the scope of protection of the technical solution of the present invention.

Claims

1. A deep learning-based behavior analysis and instant intelligent voice warning system, including a data acquisition module, a feature extraction module, a behavior analysis module, a command text generation module, a speech synthesis and broadcast unit, and a multi-terminal warning distribution network module, characterized by: The data acquisition module includes video monitoring equipment, audio capture equipment, equipment status acquisition sensors, equipment positioning system, environmental sensors, and personnel positioning system; The feature extraction module includes a deep learning model, a data center, and a central processing unit. The deep learning model algorithm is one or more of a convolutional neural network architecture, a recurrent neural network architecture, or a long short-term memory network architecture; The behavioral analysis module includes deep learning classifiers and sequence prediction models; The instruction text generation module adopts a pre-trained generative adversarial network or transformer model; The speech synthesis and broadcasting unit includes a text-to-speech engine, an audio playback device, and a broadcasting system; The multi-terminal warning distribution network module includes an in-station messaging system, a text messaging platform, and an email server; The usage includes the following steps: S1, data acquisition, the data acquisition module collects the target object's behavior video and pictures from the video surveillance equipment, and the audio capture device collects relevant voice data, and the device status acquisition sensor, device positioning system, environmental sensor, and personnel positioning system collect other multimodal data; S2, feature extraction and fusion. The feature extraction module uses a deep learning algorithm to extract features from the collected multimodal data, combines the extracted features through the data center and central processing unit fusion algorithm to form a multimodal feature vector, and inputs the multimodal feature vector into the behavior analysis model. The model uses a long short-term memory network structure to identify the behavior pattern of the target object; S3, behavior analysis and anomaly judgment, inputs the extracted features into the deep learning classifier and sequence prediction model of the behavior analysis module to identify and classify behaviors, distinguish normal behaviors from abnormal or dangerous behaviors, and analyze the detailed type and severity of dangerous behaviors; S4, warning instruction generation: The behavior analysis model outputs the behavior recognition results and determines whether there is abnormal behavior through a threshold judgment mechanism. When abnormal behavior is detected, the instruction text generation module triggers the warning instruction generation process. Based on the type and severity of the abnormal behavior, the instruction text generation module selects a corresponding warning template and generates the warning instruction text through natural language generation technology; S5, text-to-speech and timely voice broadcast: the speech synthesis and broadcast unit uses a text-to-speech engine to convert the generated warning instruction text into a natural and fluent voice signal. The voice signal is transmitted to the broadcast system through the audio playback device in the monitoring area or through the API interface. S6, Warning distribution to other terminals: The abnormal or dangerous behavior identification results can be synchronized through the multi-terminal warning distribution network module's in-station message system, SMS platform and email server. The multi-terminal warning distribution network can send warning information to managers and emergency response teams in real time through multiple channels such as in-station messages, SMS and email according to the preset warning level and distribution strategy.

2. The behavior analysis and instant intelligent voice warning system based on deep learning according to claim 1 is characterized in that: The video surveillance equipment and audio capture equipment collect the target object's behavioral video stream, static image data and corresponding voice signals in real time to form multimodal data. The video surveillance equipment uses a camera with no less than 400W pixels and the audio capture equipment uses a matching audio capture module to obtain the video stream and synchronous audio signal of the target area in real time. The video stream is collected at a rate of at least 30 frames per second, and the audio signal is captured at a sampling rate of 44.1kHz. The data acquisition module has a built-in time synchronization mechanism. The collected data is transmitted to the server and preprocessed on the collected data, including denoising, contrast enhancement, and image scaling, to improve data quality and adapt to the input requirements of the deep learning model.

3. The behavior analysis and instant intelligent voice warning system based on deep learning according to claim 1 is characterized in that: The deep learning model performs feature extraction on the collected multimodal data. The unit can extract key spatiotemporal features from video data, visual features from image data, and acoustic features from speech data. The extracted features include the edges, textures, and shapes of the images. For the synchronously collected audio signals, a deep learning model with a convolutional neural network architecture or a recurrent neural network architecture is used to extract features using Mel-frequency cepstral coefficients.

4. The behavior analysis and instant intelligent voice warning system based on deep learning according to claim 1 is characterized in that: The deep learning classifier is one or more of a support vector machine, a decision tree, a random forest, or a neural network. The deep learning classifier and the sequence prediction model perform a comprehensive analysis of the extracted features to identify the behavior patterns of the target object and make abnormal behavior judgments. The behavior analysis module uses advanced algorithms of support vector machines, random forests, and recursive neural networks.

5. The behavior analysis and instant intelligent voice warning system based on deep learning according to claim 1 is characterized in that: The instruction text generation module generates specific warning instruction text based on the behavior analysis results using natural language processing technology.

6. The behavior analysis and instant intelligent voice warning system based on deep learning according to claim 1, characterized in that: The text-to-speech engine converts the warning instruction text into a natural and smooth voice signal, and realizes real-time broadcast of voice warnings through the linkage of audio playback equipment and broadcasting system.

7. The behavior analysis and instant intelligent voice warning system based on deep learning according to claim 1, characterized in that: The multi-terminal early warning distribution network distributes abnormal or dangerous behavior identification results in real time to multiple early warning channels such as the station message system, SMS platform and email server through secure communication protocols and interfaces.

8. The behavior analysis and instant intelligent voice warning system based on deep learning according to claim 7, characterized in that: The in-site message system, SMS platform and email server can adopt hierarchical management and personalized push, and the multi-terminal warning distribution network supports confirmation feedback from the warning receiving end, so that the warning logic and distribution strategy can be adjusted according to the feedback information.

Images