Event detection method and device, storage medium and electronic equipment

Abstract

The invention discloses an event detection method and device, a storage medium and electronic equipment. The method comprises the steps that multiple sets of audio signals detected by multiple sensors are acquired, the multiple sets of audio signals correspond to the multiple sensors in a one-to-one mode, all the sensors face different directions, and all the sensors are used for collecting one set of audio signals corresponding to at least two detection areas; performing audio feature extraction operation on each group of audio signals in the plurality of groups of audio signals to obtain a plurality of audio feature values; determining a target acquisition direction according to a value relationship between the plurality of audio feature values and a feature threshold, and obtaining thermal imaging frame information corresponding to the target acquisition direction, the thermal imaging frame information being used for reflecting a temperature distribution condition in the target acquisition direction; and an event prediction result is determined based on the thermal imaging frame information, and the event prediction result is used for indicating whether the target event occurs or not. The technical problem that the event detection accuracy is low is solved.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular, to an event detection method and device, a storage medium, and an electronic device. Background Art

[0002] In related technologies, event detection methods are easily affected by environmental noise, resulting in frequent false alarms. Taking the gas leak detection event as an example, although the event detection method can provide directional information, the equipment is costly and bulky, and is not suitable for a variety of application scenarios. At the same time, due to environmental factors such as wind direction, the detection accuracy is affected. In other words, the traditional event detection method lacks a comprehensive analysis of the multi-dimensional characteristics of the event, and it is difficult to effectively distinguish between real events and interference signals in a complex environment, resulting in low detection accuracy.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide an event detection method and device, a storage medium, and an electronic device to at least solve the technical problem of low accuracy of event detection.

[0005] According to one aspect of an embodiment of the present application, an event detection method is provided, comprising: acquiring multiple groups of audio signals detected by multiple sensors, wherein the multiple groups of audio signals correspond one-to-one to the multiple sensors, each of the multiple sensors is oriented in a different direction, and each sensor is used to collect a group of audio signals corresponding to at least two detection areas; performing an audio feature extraction operation on each of the multiple groups of audio signals to obtain multiple audio feature values; determining a target collection direction according to a value relationship between the multiple audio feature values ​​and a feature threshold, and acquiring thermal imaging frame information corresponding to the target collection direction, wherein the thermal imaging frame information is used to reflect the temperature distribution in the target collection direction; determining an event prediction result based on the thermal imaging frame information, wherein the event prediction result is used to indicate whether a target event occurs.

[0006] According to another aspect of an embodiment of the present application, an event detection device is also provided, including: an acquisition module, used to acquire multiple groups of audio signals detected by multiple sensors, wherein the multiple groups of audio signals correspond one-to-one to the multiple sensors, each of the multiple sensors is oriented in a different direction, and each sensor is used to collect a group of audio signals corresponding to at least two detection areas; an extraction module, used to perform an audio feature extraction operation on each group of audio signals in the multiple groups of audio signals, respectively, to obtain multiple audio feature values; a determination module, used to determine a target collection direction according to a value relationship between the multiple audio feature values ​​and a feature threshold, and obtain thermal imaging frame information corresponding to the target collection direction, wherein the thermal imaging frame information is used to reflect the temperature distribution in the target collection direction; a detection module, used to determine an event prediction result based on the thermal imaging frame information, wherein the event prediction result is used to indicate whether a target event occurs.

[0007] Optionally, the device is used to perform audio feature extraction operations on each group of audio signals in the multiple groups of audio signals to obtain multiple audio feature values ​​in the following manner: perform short-time Fourier transform on a group of audio signals to obtain a spectrum information matrix of the group of audio signals in different time windows, wherein each row of the spectrum information matrix represents spectrum information in a time window corresponding to the group of audio signals, and each column of the spectrum information matrix represents intensity distribution at a frequency corresponding to the group of audio signals; perform local filtering on the spectrum information matrix to obtain multiple target noise arrays, wherein one target noise array among the multiple target noise arrays corresponds to a row in the spectrum information matrix; eliminate the noise component of the spectrum information matrix according to the multiple target noise arrays to obtain a target audio feature value, wherein the multiple audio feature values ​​include the target audio feature value.

[0008] Optionally, the device is used to perform short-time Fourier transform on the group of audio signals to obtain a spectrum information matrix of the group of audio signals in different time windows in the following manner: dividing the group of audio signals into multiple short-time frame signals, wherein one short-time frame signal represents an audio segment corresponding to the group of audio signals in the time dimension, and the group of audio signals corresponds to multiple audio segments; and using a Hamming window function to process the multiple short-time frame signals to obtain the spectrum information matrix.

[0009] Optionally, the device is used to perform local filtering on the spectrum information matrix to obtain multiple target noise arrays in the following manner: convert the values ​​of each element in the spectrum information matrix into absolute values ​​to obtain an intermediate spectrum information matrix; divide each row of the intermediate spectrum information matrix into N intervals, wherein each interval includes i matrix elements, N and i are both positive integers, and perform the following operations on each row to obtain N maximum local averages: calculate the average value between each matrix element and the adjacent matrix element in each interval to obtain i local averages; determine the maximum value of the i local averages as the maximum local average of the corresponding interval, and determine the value of each matrix element as the maximum local average; store the N maximum local averages in their corresponding noise arrays.

[0010] Optionally, the device is also used to: perform the following operations on each array element in the noise array to obtain the multiple target noise arrays: respectively calculate the average value of the array element and the adjacent array elements to obtain a target average value; and use the target average value to replace the value of the array element.

[0011] Optionally, the device is used to eliminate the noise component of the spectrum information matrix according to the multiple target noise arrays to obtain the target audio eigenvalue in the following manner: traverse the spectrum information matrix, determine each row as a target row in turn, wherein the target row includes M matrix elements, M is a positive integer, and perform the following operations on the target row until the traversal of the spectrum information matrix is ​​completed: determine a target noise array corresponding to the target row from the multiple target noise arrays, wherein the target noise array includes M array elements, and the matrix elements in the jth interval in the target row correspond to the jth array elements in the target noise array, and j is less than M; subtract the values ​​of the array elements at the corresponding positions in the target noise array from the values ​​of each matrix element of the target row to eliminate the noise component of the target row; determine the spectrum information matrix with the noise component eliminated as the target spectrum information matrix, and determine the target matrix elements in the target spectrum information matrix whose values ​​are greater than or equal to the frequency threshold; sum the values ​​of the target matrix elements to obtain the target audio eigenvalue.

[0012] Optionally, the device is used to determine the target acquisition direction according to the value relationship between the multiple audio feature values ​​and the feature threshold, and obtain the thermal imaging frame information corresponding to the target acquisition direction in the following manner: when at least one audio feature value among the multiple audio feature values ​​is greater than or equal to the feature threshold, determine the largest audio feature value among the multiple audio feature values ​​as the first audio feature value, and determine the largest audio feature value among the multiple audio feature values ​​except the first audio feature value as the second audio feature value; determine an offset direction value based on the first audio feature value and the second audio feature value; adjust the acquisition direction indicated by the first audio feature value according to the offset direction value to determine the target acquisition direction.

[0013] Optionally, the device is used to determine the offset direction value based on the first audio feature value and the second audio feature value in the following manner: determining the sum of the first audio feature value and the second audio feature value as a feature sum value; determining the product of a preset angle value and the second audio feature value as a feature product value; and determining a value obtained by dividing the feature product value by the feature sum value as the offset direction value.

[0014] Optionally, the device is used to obtain thermal imaging frame information corresponding to the target acquisition direction in the following manner: controlling a thermal imaging frame acquisition device to rotate to the target acquisition direction based on the target acquisition direction; and controlling the thermal imaging frame acquisition device to obtain thermal imaging frame information corresponding to the target acquisition direction.

[0015] Optionally, the device is also used to: generate a warning message when the event detection result indicates that the number of target objects in the target sub-area within the target detection area has reached a maximum number, wherein the warning message indicates that the target sub-area needs to be evacuated; and / or reacquire the multiple groups of audio signals when the event detection result indicates that the number of objects in the target sub-area has not reached the maximum number.

[0016] Optionally, the device is used to determine an event prediction result based on the thermal imaging frame information in the following manner: input the thermal imaging frame information into an object detection model, and output the event prediction result; wherein the object detection model is used to identify object features in the thermal imaging frame information, and determine whether the target event occurs based on the object features.

[0017] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above event detection method when running.

[0018] According to another aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above event detection method.

[0019] According to another aspect of the embodiments of the present application, there is further provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the event detection method through the computer program.

[0020] In the embodiment of the present application, multiple groups of audio signals detected by multiple sensors are obtained, and audio feature extraction is performed separately to obtain multiple audio feature values. Then, the target collection direction of the gas leakage is determined by analyzing the relationship between each feature value and the threshold, and the thermal imaging frame information of the direction is obtained to reflect the temperature distribution. The thermal imaging frame information is input into the object detection model to output the prediction result of whether the gas leakage occurs, thereby achieving the purpose of accurately locating and identifying the gas leakage event, thereby achieving the technical effect of improving the detection accuracy and response speed of the gas leakage early warning system. In general, the embodiment of the present application improves the comprehensiveness and accuracy of feature detection based on multi-sensor information fusion, combines the temperature sensitivity of thermal imaging technology and the intelligent analysis ability of the object detection model, and realizes the rapid positioning and accurate judgment of the target event (for example, gas leakage event, etc.), avoids the limitations of a single sensor or technology, improves the overall performance and practicality of event detection, and thus solves the technical problem of low accuracy of event detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0022] Figure 1 is a schematic diagram of an application environment of an optional event detection method according to an embodiment of the present application;

[0023] Figure 2 is a flow chart of an optional event detection method according to an embodiment of the present application;

[0024] Figure 3 is a schematic diagram of sensor deployment in an optional event detection method according to an embodiment of the present application;

[0025] Figure 4 is a flow chart of an optional event detection method according to an embodiment of the present application;

[0026] Figure 5 is a schematic diagram of sensor deployment of another optional event detection method according to an embodiment of the present application;

[0027] Figure 6 is a schematic diagram of target acquisition direction of another optional event detection method according to an embodiment of the present application;

[0028] Figure 7 is a schematic structural diagram of an optional event detection device according to an embodiment of the present application;

[0029] Figure 8 is a schematic diagram of the structure of an optional event detection product according to an embodiment of the present application;

[0030] Fig. 9 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] The present application is described below in conjunction with embodiments:

[0034] According to one aspect of an embodiment of the present application, an event detection method is provided. Optionally, in this embodiment, the event detection method can be applied to Figure 1 In the hardware environment composed of the server 101 and the terminal device 103 shown in FIG. Figure 1As shown, the server 101 is connected to the terminal device 103 via a network, and can be used to provide services for the terminal device or an application 107 installed on the terminal device. The application can be a video application, an instant messaging application, a browser application, an educational application, a game application, etc. A database 105 may be set up on the server or independently of the server to provide data storage services for the server 101, for example, a game data storage server. The above-mentioned network may include but is not limited to: a wired network, a wireless network, wherein the wired network includes: a local area network, a metropolitan area network and a wide area network, and the wireless network includes: Bluetooth, WIFI and other networks that realize wireless communication. The terminal device 103 may be a terminal configured with an application, and may include but is not limited to at least one of the following: a mobile phone (such as an Android phone, an iOS phone, etc.), a laptop computer, a tablet computer, a PDA, a MID (Mobile Internet Devices), a PAD, a desktop computer, a smart TV, an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, a virtual reality (Virtual Reality, VR for short) terminal, an augmented reality (Augmented Reality, AR for short) terminal, a mixed reality (Mixed Reality, MR for short) terminal and other computer devices. The above-mentioned server may be a single server, or a server cluster consisting of multiple servers, or a cloud server.

[0035] Combination Figure 1 As shown, the above event detection method can be executed by an electronic device, which can be a terminal device or a server. The above event detection method can be implemented by the terminal device or the server respectively, or by the terminal device and the server together.

[0036] The above is only an example and is not specifically limited in this embodiment.

[0037] Optionally, as an optional implementation, as Figure 2 As shown, the above event detection method includes:

[0038] S202, acquiring multiple groups of audio signals detected by multiple sensors, wherein the multiple groups of audio signals correspond one-to-one to the multiple sensors, each of the multiple sensors faces a different direction, and each sensor is used to collect a group of audio signals corresponding to at least two detection areas;

[0039] Optionally, in an embodiment of the present application, the above-mentioned multiple sensors refer to devices that can sense environmental changes and convert them into electrical signals, including but not limited to sound sensors, etc., which can be distributed in different positions to capture sound signals in the environment; the above-mentioned multiple groups of audio signals refer to a series of sound signal data obtained by these sensors distributed in different positions, and each group of audio signals corresponds to a specific sensor position.

[0040] It should be noted that the types, quantity and distribution locations of sensors can be adjusted according to specific application scenarios and needs. For example, when the environment to be monitored is large or complex, the number and types of sensors can be increased to capture environmental information more comprehensively. This application does not limit this.

[0041] Exemplarily, multiple sensors are deployed in the detection area, and the sounds in the detection area will be captured by the sensors and transmitted to the central processing unit. In the central processing unit, these multiple groups of audio signals will be further processed to extract features for event detection.

[0042] In an exemplary embodiment, Figure 3 is a schematic diagram of sensor deployment in an optional event detection method according to an embodiment of the present application, wherein the deployment positions of the above-mentioned multiple sensors are as follows: Figure 3 As shown, each sensor faces a different direction, and each sensor is used to collect a group of audio signals corresponding to at least two detection areas.

[0043] S204, performing an audio feature extraction operation on each of the multiple groups of audio signals to obtain multiple audio feature values;

[0044] Optionally, in an embodiment of the present application, the above-mentioned audio feature extraction operation refers to the process of preprocessing and feature analysis of the audio signal, including but not limited to applying short-time Fourier transform, Hamming window filtering, interval average calculation, peak detection, etc.

[0045] It should be noted that the extraction of audio feature values ​​can be implemented using a variety of algorithms according to different scenarios and needs, such as automatic feature extraction based on deep learning, multi-scale analysis based on wavelet transform, etc., and this application does not limit this.

[0046] Exemplarily, first, a short-time Fourier transform is performed on each group of audio signals, and a Hamming window is used for filtering to reduce noise interference; secondly, the transformed signals are averaged by interval to further extract the frequency domain characteristics of the gas leakage sound; thirdly, the peak points in each interval are detected, and the average value of the peak points is used as the characteristic value of the group of audio signals; finally, the obtained audio characteristic value is compared with a preset threshold to determine whether a gas leakage event may occur.

[0047] S206, determining a target acquisition direction according to a value relationship between a plurality of audio feature values ​​and feature thresholds, and acquiring thermal imaging frame information corresponding to the target acquisition direction, wherein the thermal imaging frame information is used to reflect temperature distribution in the target acquisition direction;

[0048] Optionally, in an embodiment of the present application, the value relationship between the above-mentioned multiple audio feature values ​​and feature thresholds refers to determining the most likely source direction of the gas leakage sound by comparing the size of each audio feature value with the preset feature threshold, including but not limited to maximum value detection, second maximum value analysis, direction weight calculation, etc. The above-mentioned thermal imaging frame information refers to image data collected by equipment such as infrared thermal imaging sensors, which can reflect the temperature distribution in the target collection direction; it not only contains temperature information at various positions in the space, but also can identify local temperature changes caused by the event to be detected (such as gas leakage) through temperature comparison analysis, including but not limited to heat source positioning, identification of temperature abnormality areas, etc.

[0049] It should be noted that the method for determining the target collection direction may vary according to the specific application scenario and the characteristics of the hardware equipment. For example, a three-coordinate positioning algorithm, angle difference positioning technology or other multi-sensor fusion positioning strategies may be used. As long as the direction of the gas leak can be effectively determined based on the audio feature values, this application does not limit this.

[0050] Exemplarily, the target acquisition direction determination and thermal imaging frame information acquisition process in the embodiment of the present application includes: first, according to the size of multiple audio feature values, determine the sensor direction with the largest feature value as the preliminary gas leakage direction; second, calculate the direction offset between the sensor with the second largest feature value and the sensor with the maximum value to refine the positioning accuracy; third, control the infrared thermal imaging sensor to face the calculated target direction and collect thermal imaging frame information in this direction; finally, analyze the thermal imaging frame information, identify the temperature anomaly in the gas leakage area, and thus verify and accurately determine the specific location of the gas leakage. In the whole process, through the fusion of multi-sensor information, gas leakage can be quickly and accurately located, improving the response speed and accuracy of the early warning system.

[0051] S208, determining an event prediction result based on the thermal imaging frame information, wherein the event prediction result is used to indicate whether a target event occurs.

[0052] Exemplarily, it may include but is not limited to inputting thermal imaging frame information into an object detection model and outputting event prediction results; a thermal imaging frame information comparison method may also be used (that is, comparing the thermal imaging frame information of the current frame with the thermal imaging frame information under historical normal conditions, and if a significant abnormal temperature change is found, it is predicted to be a target event); a threshold detection method may also be used (that is, directly analyzing the temperature distribution in the thermal imaging video frame, and if the temperature of a specific area exceeds a preset threshold, it is determined that a target event has occurred).

[0053] The object detection model is used to identify object features in thermal imaging frame information and determine whether a target event occurs based on the object features.

[0054] Optionally, in an embodiment of the present application, the above-mentioned object detection model refers to the use of deep learning technology, such as convolutional neural network, to analyze the thermal imaging frame information, identify possible gas leakage heat source characteristics, including but not limited to the heat source shape, temperature distribution pattern, etc., to determine whether there is a gas leakage event. The above-mentioned target events include but are not limited to gas leakage events, fire events, smoke events, carbon monoxide poisoning events, natural gas leakage events, etc.

[0055] It should be noted that the output of event prediction results can be implemented based on different deep learning frameworks and algorithms, such as YOLO, SSD, Faster R-CNN, etc. As long as the heat source characteristics of gas leakage can be accurately identified from the thermal imaging frame information, this application does not limit this.

[0056] Exemplarily, first, the collected thermal imaging frame information is preprocessed, such as scale normalization, image enhancement, etc., to improve the accuracy of subsequent object detection; secondly, the preprocessed thermal imaging frame information is input into an object detection model that has been trained with a large number of gas leakage heat source features; thirdly, the model analyzes the temperature distribution pattern in the thermal imaging frame information to identify possible gas leakage heat sources; finally, based on the identified heat source features, a prediction result of whether a gas leakage target event has occurred is output.

[0057] In an exemplary embodiment, taking the application scenario of gas leak detection in a home kitchen as an example, after detecting the sound characteristics of the gas leak, the approximate direction of the gas leak is determined, and the infrared thermal imaging sensor is controlled to turn to that direction to collect thermal imaging frame information. Subsequently, the thermal imaging frame information is input into the object detection model based on deep learning. The model has been trained with the characteristics of the heat source of the gas leak and can recognize specific temperature distribution patterns, such as the shape of a jet-like heat source. After analyzing the thermal imaging frame information, the model confirms the existence of the gas leak and outputs the prediction result of the gas leak event, which then triggers the corresponding early warning mechanism to promptly notify the user to take measures to avoid potential dangers.

[0058] In an exemplary embodiment, Figure 4 is a flow chart of an optional event detection method according to an embodiment of the present application, such as Figure 4 As shown, including but not limited to:

[0059] S402, start;

[0060] S404, multiple sensors collect audio signals;

[0061] S406, filtering each group of audio signals;

[0062] S408, obtaining multiple audio feature values ​​after filtering, and performing feature threshold determination;

[0063] S410, whether there is an audio feature value exceeding the feature threshold, if yes, execute S412, otherwise execute S402;

[0064] S412, calculating the target acquisition direction using multiple audio feature values;

[0065] S414, controlling the thermal imaging device to turn to the target acquisition direction to acquire video frame data (the above-mentioned thermal imaging frame information);

[0066] S416, analyzing the video frame data using an object detection model;

[0067] S418, determining whether to generate an alarm message according to the model output result, corresponding to whether the above target event occurs, if an alarm message is generated, executing S420, otherwise executing S402;

[0068] S420, controlling the broadcasting of alarm information in the target area corresponding to the target collection direction, and also notifying other personnel in the detection area where no target event occurs to pay attention to whether there is a potential risk in the area.

[0069] Through the embodiments of the present application, multiple groups of audio signals detected by multiple sensors are obtained, and audio feature extraction is performed separately to obtain multiple audio feature values. Then, the target collection direction of the gas leakage is determined by analyzing the relationship between each feature value and the threshold, and the thermal imaging frame information in this direction is obtained to reflect the temperature distribution. The thermal imaging frame information is input into the object detection model to output the prediction result of whether the gas leakage occurs, thereby achieving the purpose of accurately locating and identifying the gas leakage event, thereby achieving the technical effect of improving the detection accuracy and response speed of the gas leakage early warning system. In general, the embodiments of the present application improve the comprehensiveness and accuracy of feature detection based on multi-sensor information fusion, and combine the temperature sensitivity of thermal imaging technology and the intelligent analysis ability of the object detection model to achieve rapid positioning and accurate judgment of target events (for example, gas leakage events, etc.), avoid the limitations of a single sensor or technology, improve the overall performance and practicality of event detection, and thus solve the technical problem of low accuracy of event detection.

[0070] As an optional scheme, the above-mentioned audio feature extraction operation is performed on each group of audio signals in the above-mentioned multiple groups of audio signals respectively to obtain multiple audio feature values, including: performing short-time Fourier transform on a group of audio signals to obtain a spectrum information matrix of the above-mentioned group of audio signals in different time windows, wherein each row of the above-mentioned spectrum information matrix represents the spectrum information in a time window corresponding to the above-mentioned group of audio signals, and each column of the above-mentioned spectrum information matrix represents the intensity distribution at a frequency corresponding to the above-mentioned group of audio signals; performing local filtering on the above-mentioned spectrum information matrix to obtain multiple target noise arrays, wherein one target noise array among the above-mentioned multiple target noise arrays corresponds to a row in the above-mentioned spectrum information matrix; according to the above-mentioned multiple target noise arrays, eliminating the noise component of the above-mentioned spectrum information matrix to obtain the target audio feature value, wherein the above-mentioned multiple audio feature values ​​include the above-mentioned target audio feature value.

[0071] Optionally, in an embodiment of the present application, performing a short-time Fourier transform on a set of audio signals refers to converting the audio signals in the time domain into a spectrum representation in the frequency domain to analyze the intensity distribution of different frequency components. The spectrum information matrix refers to a two-dimensional data structure containing the spectrum information of the audio signal, each row corresponds to the spectrum information in a time window, and each column represents the intensity distribution at a frequency. Local filtering refers to filtering a local area in the spectrum information matrix to reduce the impact of noise and obtain multiple target noise arrays, each of which corresponds to a row in the spectrum information matrix.

[0072] It should be noted that there are many ways to implement local filtering, and median filtering, mean filtering, adaptive filtering and other technologies may be used. As long as the noise component in the spectrum information matrix can be effectively reduced, this application does not limit this.

[0073] Exemplarily, first, a short-time Fourier transform is performed on the audio signal to obtain a spectrum information matrix; then, each row of the matrix data is locally filtered to eliminate noise; then, the characteristic value of the gas leakage sound is extracted by comparing the filtered spectrum information with a preset threshold; finally, the characteristic values ​​obtained by all audio sensors are comprehensively analyzed to determine whether the target event has occurred.

[0074] In an exemplary embodiment, taking the application scenario of gas leak detection in a home kitchen as an example, the system first performs a short-time Fourier transform on the audio signal in the kitchen to convert the signal into a spectrum information matrix. Subsequently, each row of spectrum information in the matrix is ​​locally filtered to reduce the interference caused by environmental noise and obtain a target noise array; the filtered spectrum information is further compared with the characteristic threshold of the gas leak sound to identify the spectrum characteristics of the gas leak, and finally the audio characteristic value of the gas leak is determined to determine whether a gas leak has occurred in the kitchen.

[0075] Through the embodiments of the present application, short-time Fourier transform is combined with local filtering to achieve the technical effect of extracting specified sound features from a noisy environment, thereby achieving the purpose of improving event detection accuracy and response speed.

[0076] As an optional scheme, the above-mentioned short-time Fourier transform is performed on the above-mentioned group of audio signals to obtain the spectrum information matrix of the above-mentioned group of audio signals in different time windows, including: dividing the above-mentioned group of audio signals into multiple short-time frame signals, wherein the above-mentioned one short-time frame signal represents an audio segment corresponding to the above-mentioned group of audio signals in the time dimension, and the above-mentioned group of audio signals corresponds to multiple audio segments; using the Hamming window function to process the above-mentioned multiple short-time frame signals to obtain the above-mentioned spectrum information matrix.

[0077] Optionally, in an embodiment of the present application, the above-mentioned group of audio signals refers to an audio data sequence continuously collected from an environment, and the above-mentioned multiple short-time frame signals are several segments into which this continuous signal is divided in the time dimension, each segment represents a short time segment of the audio signal, including but not limited to frame signals of lengths such as 10ms, 20ms, and 30ms.

[0078] It should be noted that the parameter selection of the Hamming window function, such as the length of the window function and the moving step, will be adjusted according to the actual application scenario and the characteristics of the audio signal. This application does not limit this to adapt to the audio processing requirements in different environments.

[0079] Exemplarily, first, a group of audio signals is divided into multiple short-time frame signals based on the time dimension, and each frame signal contains audio data within a specific time window; then, a Hamming window function is applied to each short-time frame signal for windowing processing to reduce spectral leakage; then, the windowed short-time frame signal is subjected to Fourier transform to convert the time domain signal into a frequency domain signal to generate a spectral information matrix.

[0080] In an exemplary embodiment, taking gas leakage detection as an example, first, audio signals at different positions in the area to be detected are continuously collected and divided into multiple short-time frame signals of 10ms in length; then, a Hamming window function is applied to each short-time frame signal to reduce the error during spectrum analysis; then, the windowed frame signal is Fourier transformed and converted into a spectrum information matrix so that the characteristics of the gas leakage can be extracted in subsequent processing.

[0081] Through the embodiments of the present application, a method of dividing an audio signal into short-time frame signals and processing them using a Hamming window function is adopted to achieve the technical effect of reducing the spectral leakage of the audio signal and improving the spectral resolution, thereby achieving the purpose of optimizing audio feature extraction and enhancing the overall performance of event detection.

[0082] As an optional scheme, the above-mentioned spectrum information matrix is ​​locally filtered to obtain multiple target noise arrays, including: converting the values ​​of each element in the above-mentioned spectrum information matrix into absolute values ​​to obtain an intermediate spectrum information matrix; dividing each row of the above-mentioned intermediate spectrum information matrix into N intervals, wherein each interval includes i matrix elements, N, i are both positive integers, and performing the following operations on each row to obtain N maximum local averages: calculating the average value between each matrix element and the adjacent matrix element in each interval to obtain i local averages; determining the maximum value of the above i local averages as the above-mentioned maximum local average of the corresponding interval, and determining the value of each matrix element as the above-mentioned maximum local average; and storing the above N maximum local averages in their corresponding noise arrays.

[0083] Optionally, in an embodiment of the present application, the above-mentioned spectrum information matrix refers to a two-dimensional data matrix obtained by short-time Fourier transform, representing the spectrum information of the audio signal; the above-mentioned intermediate spectrum information matrix is ​​a new matrix obtained by taking the absolute values ​​of the values ​​of all matrix elements on the basis of the spectrum information matrix. The N maximum local average values ​​refer to the maximum values ​​selected in each row of the spectrum information matrix after dividing the row into N intervals and calculating the local average value within each interval for further noise processing.

[0084] It should be noted that the values ​​of N and i can be flexibly set according to the length and frequency resolution requirements of the audio signal, and this application does not limit this, so as to meet the noise processing accuracy and efficiency in different application scenarios.

[0085] Exemplarily, first, the absolute values ​​of the elements in the spectrum information matrix are taken to form an intermediate spectrum information matrix; then, each row is divided into N intervals, each interval containing i matrix elements; then, the average values ​​of the matrix elements and adjacent elements in each interval are calculated to obtain the local average values; then, the maximum value is selected from the i local average values ​​of each interval as the maximum local average value of the interval, and the values ​​of all matrix elements in the interval are updated to the maximum local average value; finally, the N maximum local average values ​​of each row are respectively stored in the corresponding noise array for subsequent processing.

[0086] In an exemplary embodiment, taking a household gas leak warning system as an example, when processing audio signals, the spectrum information matrix obtained by short-time Fourier transform is first converted to an absolute value to form an intermediate spectrum information matrix. Subsequently, each row of the intermediate spectrum information matrix is ​​divided into 100 intervals, each interval containing 10 matrix elements; the above-mentioned maximum local average calculation and update operation is performed on each interval to reduce spectrum noise and enhance the characteristics of gas leak sound. Finally, the 100 maximum local averages obtained are stored in their respective corresponding noise arrays as the basis for subsequent positioning analysis.

[0087] Through the embodiments of the present application, a processing flow of performing absolute value conversion, interval division, maximum local average value calculation and storage in a noise array on the spectrum information matrix is ​​adopted to achieve the technical effects of denoising and feature enhancement of the spectrum information, thereby achieving the purpose of improving the accuracy of locating target events and reducing the false alarm rate.

[0088] As an optional scheme, the above method also includes: performing the following operations on each array element in the above noise array to obtain the above multiple target noise arrays: respectively calculating the average value of the above array element and the adjacent array element to obtain the target average value; using the above target average value to replace the value of the above array element.

[0089] Optionally, in the embodiment of the present application, the noise array is an array used to store the maximum local average value in each row of spectrum information during the spectrum information matrix processing. The target noise array is an array set that has been further smoothed to reduce noise and improve signal feature stability.

[0090] It should be noted that the range for calculating the average value of array elements and adjacent array elements can be any adjacent elements, such as one before and after, two before and after, etc. This depends on the specific denoising requirements and signal characteristics, and this application does not limit this.

[0091] Exemplarily, first, each array element in the noise array is selected; then, the average value of the element and its adjacent elements before and after is calculated to obtain a target average value; then, the calculated target average value is used to replace the value of the original array element, thereby smoothing the noise array and reducing the impact of noise; the final smoothed array set is a plurality of target noise arrays.

[0092] As an optional scheme, the above-mentioned elimination of the noise component of the above-mentioned spectrum information matrix according to the above-mentioned multiple target noise arrays to obtain the target audio eigenvalue includes: traversing the above-mentioned spectrum information matrix, determining each row as a target row in turn, wherein the above-mentioned target row includes M matrix elements, M is a positive integer, and performing the following operations on the above-mentioned target row until the traversal of the above-mentioned spectrum information matrix is ​​completed: determining a target noise array corresponding to the above-mentioned target row in the above-mentioned multiple target noise arrays, wherein the above-mentioned target noise array includes M array elements, and the matrix elements in the j-th interval in the above-mentioned target row correspond to the j-th array elements in the above-mentioned target noise array, and the above-mentioned j is less than the above-mentioned M; subtracting the values ​​of the array elements at the corresponding positions in the above-mentioned target noise array from the values ​​of each matrix element of the above-mentioned target row to eliminate the noise component of the above-mentioned target row; determining the above-mentioned spectrum information matrix with the above-mentioned noise component eliminated as the target spectrum information matrix, and determining the target matrix elements in the above-mentioned target spectrum information matrix whose values ​​are greater than or equal to the frequency threshold; summing the values ​​of the above-mentioned target matrix elements to obtain the above-mentioned target audio eigenvalue.

[0093] Optionally, in an embodiment of the present application, the target row refers to a row of matrix elements currently being processed in the spectrum information matrix, which is used to calculate the spectrum information after noise smoothing; the target spectrum information matrix is ​​a spectrum matrix after noise component elimination processing, and its eigenvalue more accurately reflects the audio characteristics of the gas leakage event; the target matrix element refers to a matrix element in the target spectrum information matrix whose value is greater than or equal to a preset frequency threshold.

[0094] It should be noted that the selection of the above frequency threshold depends on the spectral characteristics of the audio signal, as well as the specific application environment and noise level, and this application does not limit this.

[0095] Exemplarily, each row of the spectrum information matrix is ​​traversed, and each row is taken as the target row in turn; the target noise array corresponding to the target row is determined, and the value of each matrix element in the target row is subtracted from the value of the noise array element at the corresponding position to eliminate the noise component; the processed matrix is ​​traversed, and the matrix elements with values ​​greater than or equal to the frequency threshold are screened out as the target matrix elements; finally, the values ​​of all the target matrix elements are summed to obtain the target audio eigenvalue.

[0096] In an exemplary embodiment, taking the gas leak warning system as an example, after the noise smoothing of the spectrum information matrix is ​​processed, the processed spectrum information matrix is ​​traversed; the first row of the matrix is ​​determined as the target row, and the corresponding target noise array is found; then, the system calculates the difference between each element of the target row and the corresponding noise array element to eliminate the influence of noise. After that, the system checks the values ​​of these elements, marks the elements greater than or equal to the preset frequency threshold as target matrix elements, sums the values ​​of all target matrix elements, and obtains the first target audio eigenvalue. The system continues to traverse other rows of the spectrum information matrix, repeating the above process until the matrix traversal is completed and all target audio eigenvalues ​​are obtained.

[0097] For example, the spectrum information matrix has a total of 100 rows, one of which is used as the target row. The target row is divided into 5 intervals, each interval has 10 matrix elements, and the corresponding target noise array is expressed as [1, 2, 3, 4, 5]. Then the 10 matrix elements in the first interval correspond to the first array element "1" of the target noise array, and these 10 matrix elements will be subtracted from this array element respectively; similarly, the 10 matrix elements in the second interval correspond to the first array element "2" of the target noise array, and these 10 matrix elements will be subtracted from this array element respectively.

[0098] Through the embodiments of the present application, noise is eliminated by subtracting matrix elements from corresponding noise array element values, and target matrix elements are screened and summed to obtain target audio eigenvalues, thereby achieving the technical effect of accurately extracting characteristic audio signals corresponding to target events from spectral information, thereby achieving the purpose of improving the accuracy and response speed of event detection.

[0099] As an optional scheme, the above-mentioned determining the target acquisition direction according to the value relationship between the above-mentioned multiple audio feature values ​​and the feature threshold, and obtaining the thermal imaging frame information corresponding to the above-mentioned target acquisition direction, includes: when at least one audio feature value among the above-mentioned multiple audio feature values ​​is greater than or equal to the above-mentioned feature threshold, determining the largest audio feature value among the above-mentioned multiple audio feature values ​​as the first audio feature value, and determining the largest audio feature value among the above-mentioned multiple audio feature values ​​except the above-mentioned first audio feature value as the second audio feature value; determining an offset direction value based on the above-mentioned first audio feature value and the above-mentioned second audio feature value; adjusting the acquisition direction indicated by the above-mentioned first audio feature value according to the above-mentioned offset direction value, and determining the above-mentioned target acquisition direction.

[0100] Optionally, in an embodiment of the present application, the above-mentioned multiple audio feature values ​​refer to a set of numerical values ​​reflecting the sound characteristics of gas leakage obtained after extraction and processing by multiple audio sensors; the above-mentioned first audio feature value and second audio feature value are respectively the largest and second largest feature values ​​among all audio feature values, and are used for subsequent direction positioning calculations.

[0101] It should be noted that the comparison between the above-mentioned audio feature values ​​and feature thresholds can be applied to different scenarios, such as gas leak warning, abnormal sound monitoring, etc. The specific value of the feature threshold can be flexibly set according to the application scenario and environmental conditions, and this application does not limit this.

[0102] Exemplarily, taking the gas leak warning system as an example, all audio feature values ​​are checked to determine whether there is at least one feature value greater than or equal to a preset feature threshold; if the condition is met, the largest audio feature value is found and determined as the first audio feature value; then, the largest one is found from the remaining feature values ​​and determined as the second audio feature value; then, based on the first audio feature value and the second audio feature value, the offset direction value is calculated to determine the relative position of the gas leak; finally, the direction of the audio sensor is adjusted according to the offset direction value to determine the target collection direction to more accurately locate the source of the leak.

[0103] In an exemplary embodiment, also taking the gas leak warning system as an example, the system first performs threshold detection on all audio feature values. Assuming that the feature values ​​of the s1, s2, s3, and s4 sensors are V1, V2, V3, and V4, respectively, and V1 and V3 are respectively greater than the preset feature threshold T, the system determines V1 as the first audio feature value and V3 as the second audio feature value (assuming V1 > V3 > V2 > V4). Next, the system calculates the offset direction value based on the first audio feature value and the second audio feature value to determine the target acquisition direction of the gas leak. Finally, the system adjusts the direction of the thermal imaging detector according to the target acquisition direction to accurately scan the suspected area of ​​the gas leak.

[0104] In an exemplary embodiment, Figure 5 FIG. 1 is a schematic diagram of sensor deployment of another optional event detection method according to an embodiment of the present application. There are 4 sensors, and their deployment positions are as follows: Figure 5 As shown, each sensor is placed between two adjacent detection areas to obtain the audio signals of the two adjacent detection areas. Then, Figure 6 is a schematic diagram of a target collection direction of another optional event detection method according to an embodiment of the present application, wherein the collection direction indicated by the first audio feature value is as follows: Figure 6 As shown in s1 in , the collection direction indicated by the second audio feature value is as follows Figure 6 As shown in s3 in the figure, after the above offset direction value is adjusted, the target acquisition direction is as follows Figure 6 As shown in D1 in FIG. 1 , at this time, the target acquisition direction points to the detection area 2.

[0105] Through the embodiments of the present application, a technical solution is adopted for determining a first audio feature value and a second audio feature value from a plurality of audio feature values, and calculating an offset direction value based on the first and second audio feature values. This achieves the technical effect of quickly and accurately locating the leakage source in a gas leak warning system, thereby achieving the purpose of improving the positioning accuracy and response speed of the warning system.

[0106] As an optional scheme, the above-mentioned determination of the offset direction value based on the above-mentioned first audio feature value and the above-mentioned second audio feature value includes: determining the sum of the above-mentioned first audio feature value and the above-mentioned second audio feature value as the feature sum value; determining the product of the preset angle value and the above-mentioned second audio feature value as the feature product value; and determining the value obtained by dividing the above-mentioned feature product value by the above-mentioned feature sum value as the above-mentioned offset direction value.

[0107] For example, it is assumed that the calculation formula for determining the target acquisition direction is as follows:

[0108] ;

[0109] Among them, A is the target acquisition direction;

[0110] The direction of the sound sensor with the maximum value among V1 to V4 (the target acquisition direction mentioned above);

[0111] Ω is the direction of the sensor offset with the second largest eigenvalue (corresponding to the offset direction), and the calculation formula of Ω is:

[0112] ;

[0113] Among them, Vmax refers to the above-mentioned first audio feature value, and Vsecond refers to the above-mentioned second audio feature value.

[0114] As an optional solution, the above-mentioned obtaining of thermal imaging frame information corresponding to the above-mentioned target acquisition direction includes: controlling the thermal imaging frame acquisition device to rotate to the above-mentioned target acquisition direction based on the above-mentioned target acquisition direction; controlling the above-mentioned thermal imaging frame acquisition device to obtain the thermal imaging frame information corresponding to the above-mentioned target acquisition direction.

[0115] Optionally, in an embodiment of the present application, the thermal imaging frame acquisition device mentioned above refers to a device that can obtain thermal imaging information of objects in the environment, including but not limited to infrared thermal imagers, thermal imaging cameras, etc., which can convert infrared radiation emitted by objects into visible images.

[0116] It should be noted that the rotation mechanism and speed of the above-mentioned thermal imaging frame acquisition device can be designed according to the specific application scenario, including but not limited to mechanical rotation, electronic zoom, etc., to ensure that the device can quickly and accurately align with the target acquisition direction, and this application does not limit this.

[0117] In an exemplary embodiment, taking a gas leak warning system as an example, after the system calculates the target acquisition direction, it will control the thermal imaging frame acquisition device (such as a steerable infrared thermal imager) to rotate to that direction. Specifically, the system converts the direction information into a device control signal, and adjusts the direction of the thermal imaging device through a motor or other drive device to align it with the calculated target direction. After the device is adjusted in place, the system will start thermal imaging frame acquisition to obtain thermal imaging frame information of the suspected leakage area for subsequent image analysis and leak confirmation.

[0118] Through the embodiments of the present application, the rotation of the thermal imaging frame acquisition device is automatically controlled based on the target acquisition direction, and the thermal imaging frame information of the area corresponding to the target acquisition direction is acquired, thereby achieving the technical effect of accurately and quickly locating the target event, and achieving the purpose of improving the response speed and positioning accuracy of event detection.

[0119] As an optional scheme, the above method also includes: when the above event detection result indicates that the number of target objects in the target sub-area within the target detection area has reached a maximum number, generating a warning message, wherein the above warning message indicates that the above target sub-area needs to be evacuated; and / or when the above event detection result indicates that the number of objects in the above target sub-area has not reached the above maximum number, re-acquiring the above multiple groups of audio signals.

[0120] Optionally, in an embodiment of the present application, the above-mentioned target sub-area refers to a local area within the target detection area where a target event is determined to occur, and the number of target objects refers to the number of target objects detected in the target sub-area; the maximum number is a preset safety threshold used to determine whether the target sub-area is in an emergency evacuation state.

[0121] For example, by analyzing the thermal imaging frame information and audio feature values, the gas leakage situation and the number of target objects in the target sub-area are determined according to the event detection algorithm. If the number of target objects reaches or exceeds the preset maximum number, the system will generate an early warning message indicating that the target sub-area needs to be evacuated. On the contrary, if the number of target objects does not reach the maximum number, the system will re-acquire multiple groups of audio signals for a new round of detection and analysis.

[0122] In an exemplary embodiment, taking the gas leak warning system as an example, assuming that the target object is a person, the system determines the suspected area of ​​gas leakage through audio signal processing and thermal imaging analysis, and detects the gathering of staff in the area. If the system determines that the number of people in the area exceeds the preset maximum safe number, it will immediately generate a warning message to instruct the people in the kitchen to evacuate quickly. If the number of people in the area does not reach the evacuation threshold, the system will re-acquire the audio signal and conduct continuous monitoring to ensure safety.

[0123] In an exemplary embodiment, taking a gas leak warning system as an example, assuming that the target object is natural gas, liquefied petroleum gas or city gas (corresponding to coal gas), the system determines through audio signal processing and thermal imaging analysis that the gas concentration exceeds the safety value, and will immediately generate a warning message to instruct personnel in the detection area to evacuate quickly.

[0124] Through the embodiments of the present application, a control process of warning message generation and re-detection based on event detection results is adopted to achieve the technical effect of dynamically adjusting response measures in the gas leak warning system and performing continuous monitoring under non-emergency conditions, thereby achieving the purpose of improving the intelligence level and safety management level of the warning system.

[0125] As an optional solution, the method further includes: each of the sensors is oriented in different directions; and each of the sensors is used to collect a group of audio signals corresponding to at least two detection areas.

[0126] As an optional scheme, the above-mentioned determination of event prediction results based on the above-mentioned thermal imaging frame information includes: inputting the above-mentioned thermal imaging frame information into an object detection model, and outputting the above-mentioned event prediction results; wherein the above-mentioned object detection model is used to identify object features in the above-mentioned thermal imaging frame information, and determine whether the above-mentioned target event occurs based on the above-mentioned object features.

[0127] Exemplarily, sensor A is deployed between detection area A and detection area B, then the audio signal that sensor A can obtain comes from detection area A and detection area B; sensor B is deployed between detection area C and detection area B, then the audio signal that sensor A can obtain comes from detection area C and detection area B. That is, the audio signal generated by one detection area can be obtained by different sensors, and different sensors have different orientations. This deployment method ensures that the audio signals of each detection area can be fully and multi-angle captured even in a complex environment, effectively improving the coverage and accuracy of event detection.

[0128] Through the embodiments of the present application, all-round monitoring of audio signals in the detection area is achieved, the problem of missing important audio information due to the limited viewing angle of a single sensor is avoided, and the robustness and reliability of obtaining audio signals are enhanced. For example, even if a sensor fails to capture key information, the data of other sensors can provide supplementary information to ensure the comprehensiveness and accuracy of event detection.

[0129] In an exemplary embodiment, the event detection method proposed in this application can realize a gas leak warning system based on audio extraction filtering and multi-sensor positioning and infrared imaging gas identification, including but not limited to:

[0130] S1, four sound sensors s1, s2, s3, s4 and an infrared thermal imaging probe D1 are placed in a certain area where multiple gas stoves are placed. Each sensor is connected to the central control and analysis terminal through wireless communication. The relative position diagram of the multi-sensor equipment and the area to be detected is shown in the figure below. Figure 5 As shown;

[0131] S2, perform specific audio filtering on the sound read by each sensor and extract the filtered audio feature value V. The filtering extraction algorithm is as follows:

[0132] S2-1, input signal s and frequency threshold t;

[0133] S2-2, apply short-time Fourier transform to the input signal, use Hamming window with size of 512 and jump of 256, the transformed signal is Sa (the length of each short-time frame is 512 data points, and the obtained spectrum signal Sa is in matrix form, where each row represents the spectrum information of a short-time frame, and each column represents the strength of the signal at a specific frequency);

[0134] S2-3, take the absolute value of signal Sa to obtain signal S;

[0135] S2-4, split the column data of each row of S into 100 intervals (since the spectrum information of each short-time frame has 512 data points, and it is divided into 100 intervals, each interval will contain about 5.12 data points on average (actually 512 / 100=5.12). This means that on the spectrum, each interval will cover a part of the continuous frequency range, and 5.12 data points represent the strength of the signal within this frequency range);

[0136] S2-5, in the processing of an interval, the column value in each interval is averaged by the current value and the next five columns, and the highest value of h in the interval is taken as the value Mj of all columns in the interval;

[0137] S2-6, insert all the highest average values ​​M (1-100) of the 100 intervals calculated in S2-5 into the array Profile;

[0138] S2-7, in each row, calculate the average value within the range of 5 for each element Mij in the array Profile, and replace the original position value M ij , i indicates the number of rows, j indicates the number of columns, and finally the array Profile after local filtering is obtained;

[0139] S2-8, traverse all points in the signal Sa, and subtract Profilej from each point (i, j) in the signal Sa (this is a characteristic value or background noise level related to the i-th row (i.e., the i-th short-time frame), each row corresponds to an array Profile, and the data point of each row is subtracted from the element at the corresponding position of the corresponding array Profile), and the value less than 0 after subtraction is directly assigned to 0;

[0140] S2-9, S2-8 obtain the frequency domain signal of the filtered audio, and then traverse each data point of the signal S obtained by S2-8, and if it is greater than the threshold t, the value of the data point is accumulated into the variable V;

[0141] S2-10, the audio signals of the four sound sensors are respectively subjected to the above algorithm to obtain audio feature values ​​V1, V2, V3, and V4;

[0142] S3, when one of the four audio feature values ​​in S2 exceeds the feature threshold T, the thermal imaging sensor is triggered to turn according to the value of V1-4 and start thermal imaging video acquisition. Assume that the target acquisition direction is determined by the following calculation formula:

[0143] ;

[0144] Where A is the target direction, is the direction of the sound sensor with the maximum value among V1 to V4, Ω is the direction of the sensor with the second largest eigenvalue, and the calculation formula of Ω is:

[0145] ;

[0146] In the above formula, Vmax refers to the maximum eigenvalue of the four sensors, and Vsecond refers to the second largest eigenvalue of the four sensors; assuming that the eigenvalues ​​detected by s1 and s3 are the first and second largest, respectively, the schematic diagram of calculating the warning target direction is as follows: Figure 6 As shown;

[0147] S4, after the target acquisition direction is calculated by the method in S3, the D1 infrared thermal imaging probe will turn to the direction, and then use the D1 infrared thermal imaging probe to collect thermal imaging frame information in the target direction and transmit it to the analysis end;

[0148] S5, the infrared radiation shape of the gas leak will appear in the image generated by the infrared thermal imaging detection collected in S4. The gas leak often appears in a jet shape. The analysis end uses the artificial intelligence recognition algorithm CNN (object detection model) that has been pre-trained with a large number of infrared thermal imaging images of gas leaks to determine whether an alarm has been generated. After inputting the image, a feature value is obtained, and an alarm threshold is preset to output an early warning result:

[0149] The eigenvalue corresponding to the input image (the above thermal imaging frame information) is greater than the alarm threshold, and there is an alarm; or, the eigenvalue corresponding to the input image is greater than the alarm threshold, and there is no alarm; if the object detection model determines that a gas leak has occurred, the regional alarm at the corresponding location (one will be deployed in each detection area) will sound an alarm. Figure 3 For example, the area alarm deployed in detection zone 2 will sound an alarm.

[0150] Through the embodiment of the present application, after the audio is collected, the audio signal will be fully filtered, various natural noises in the environment will be eliminated, and the sound of gas leakage will be filtered according to the characteristics of the high-frequency domain, effectively removing the interference of environmental noise and improving the accuracy of recognition; at the same time, with the positioning algorithm of multiple sensors, the position of gas leakage can be intelligently and quickly located; through the linkage recognition of the steerable video thermal imaging detector, thanks to the sensitive recognition of gas by infrared thermal imaging technology. The gas invisible to the human eye is converted into a visible gas thermal imaging image, and the framework trained by the thermal imaging image of gas leakage is used as an early warning algorithm for identifying gas leakage by using the deep learning method. The thermal imaging video frame of the target area is analyzed by the artificial intelligence recognition method, and the corresponding alarm generated further improves the accuracy of the alarm of abnormal gas leakage events; the technical effects of accurate alarm location, low false alarm rate, fast response, and high gas leakage recognition rate are achieved.

[0151] It is understandable that in the specific implementation of this application, related data such as user information is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0152] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0153] According to another aspect of the embodiments of the present application, an event detection device for implementing the above event detection method is also provided. Figure 7 As shown, the device comprises:

[0154] An acquisition module 702 is used to acquire multiple groups of audio signals detected by multiple sensors, wherein the multiple groups of audio signals correspond one-to-one to the multiple sensors;

[0155] An extraction module 704 is used to perform an audio feature extraction operation on each of the multiple groups of audio signals to obtain multiple audio feature values, each of the multiple sensors is oriented in a different direction, and each sensor is used to collect a group of audio signals corresponding to at least two detection areas;

[0156] A determination module 706 is used to determine a target acquisition direction according to a value relationship between a plurality of audio feature values ​​and feature thresholds, and obtain thermal imaging frame information corresponding to the target acquisition direction, wherein the thermal imaging frame information is used to reflect the temperature distribution in the target acquisition direction;

[0157] The detection module 708 is used to determine an event prediction result based on the thermal imaging frame information, wherein the event prediction result is used to indicate whether a target event occurs.

[0158] As an optional solution, the device is used to perform audio feature extraction operations on each group of audio signals in a plurality of groups of audio signals in the following manner to obtain multiple audio feature values: perform short-time Fourier transform on a group of audio signals to obtain a spectrum information matrix of a group of audio signals in different time windows, wherein each row of the spectrum information matrix represents spectrum information in a time window corresponding to a group of audio signals, and each column of the spectrum information matrix represents intensity distribution at a frequency corresponding to a group of audio signals; perform local filtering on the spectrum information matrix to obtain multiple target noise arrays, wherein one target noise array among the multiple target noise arrays corresponds to a row in the spectrum information matrix; and eliminate the noise component of the spectrum information matrix according to the multiple target noise arrays to obtain target audio feature values, wherein the multiple audio feature values ​​include the target audio feature values.

[0159] As an optional solution, the device is used to perform short-time Fourier transform on a group of audio signals in the following manner to obtain a spectrum information matrix of a group of audio signals in different time windows: divide a group of audio signals into multiple short-time frame signals, wherein a short-time frame signal represents an audio segment corresponding to a group of audio signals in the time dimension, and a group of audio signals corresponds to multiple audio segments; use a Hamming window function to process multiple short-time frame signals to obtain a spectrum information matrix.

[0160] As an optional solution, the device is used to perform local filtering on the spectrum information matrix in the following manner to obtain multiple target noise arrays: the values ​​of each element in the spectrum information matrix are converted into absolute values ​​to obtain an intermediate spectrum information matrix; each row of the intermediate spectrum information matrix is ​​divided into N intervals, wherein each interval includes i matrix elements, N, i are both positive integers, and the following operations are performed on each row to obtain N maximum local averages: in each interval, the average value between each matrix element and the adjacent matrix element is calculated to obtain i local averages; the maximum value of the i local averages is determined as the maximum local average of the corresponding interval, and the value of each matrix element is determined as the maximum local average; the N maximum local averages are respectively stored in their corresponding noise arrays.

[0161] As an optional solution, the device is also used to: perform the following operations on each array element in the noise array to obtain multiple target noise arrays: respectively calculate the average value of the array element and the adjacent array elements to obtain the target average value; use the target average value to replace the value of the array element.

[0162] As an optional solution, the device is used to eliminate the noise component of the spectrum information matrix according to multiple target noise arrays in the following manner to obtain the target audio eigenvalue: traverse the spectrum information matrix, determine each row as the target row in turn, wherein the target row includes M matrix elements, M is a positive integer, and perform the following operations on the target row until the traversal of the spectrum information matrix is ​​completed: determine a target noise array corresponding to the target row in multiple target noise arrays, wherein a target noise array includes M array elements, and the matrix elements in the jth interval in the target row correspond to the jth array elements in a target noise array, and j is less than M; subtract the values ​​of the array elements at the corresponding positions in a target noise array from the values ​​of each matrix element of the target row to eliminate the noise component of the target row; determine the spectrum information matrix with the noise component eliminated as the target spectrum information matrix, and determine the target matrix elements in the target spectrum information matrix whose values ​​are greater than or equal to the frequency threshold; sum the values ​​of the target matrix elements to obtain the target audio eigenvalue.

[0163] As an optional solution, the device is used to determine the target acquisition direction according to the value relationship between multiple audio feature values ​​and feature thresholds, and obtain thermal imaging frame information corresponding to the target acquisition direction in the following manner: when at least one audio feature value among the multiple audio feature values ​​is greater than or equal to the feature threshold, the largest audio feature value among the multiple audio feature values ​​is determined as the first audio feature value, and the largest audio feature value among the multiple audio feature values ​​except the first audio feature value is determined as the second audio feature value; determine an offset direction value based on the first audio feature value and the second audio feature value; adjust the acquisition direction indicated by the first audio feature value according to the offset direction value to determine the target acquisition direction.

[0164] As an optional scheme, the device is used to determine the offset direction value based on the first audio feature value and the second audio feature value in the following manner: the sum of the first audio feature value and the second audio feature value is determined as the feature sum value; the product of the preset angle value and the second audio feature value is determined as the feature product value; and the value obtained by dividing the feature product value by the feature sum value is determined as the offset direction value.

[0165] As an optional solution, the device is used to obtain thermal imaging frame information corresponding to the target acquisition direction in the following manner: based on the target acquisition direction, the thermal imaging frame acquisition device is controlled to rotate to the target acquisition direction; and the thermal imaging frame acquisition device is controlled to obtain thermal imaging frame information corresponding to the target acquisition direction.

[0166] As an optional solution, the device is also used to: generate a warning message when the event detection result indicates that the number of target objects in the target sub-area within the target detection area has reached a maximum number, wherein the warning message indicates that the target sub-area needs to be evacuated; and / or re-acquire multiple groups of audio signals when the event detection result indicates that the number of objects in the target sub-area has not reached the maximum number.

[0167] As an optional solution, the device is also used for: each sensor is oriented to a different direction; each sensor is used for collecting a group of audio signals corresponding to at least two detection areas.

[0168] As an optional solution, the device is used to determine an event prediction result based on the thermal imaging frame information in the following manner: input the thermal imaging frame information into an object detection model and output the event prediction result; wherein the object detection model is used to identify object features in the thermal imaging frame information and determine whether the target event occurs based on the object features.

[0169] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0170] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0171] According to one aspect of the present application, a computer program product is provided. The computer program product includes a computer program.

[0172] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0173] Figure 8 The structure block diagram of a computer system for implementing an electronic device according to an embodiment of the present application is schematically shown.

[0174] It should be noted that Figure 8 The computer system 800 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0175] like Figure 8As shown, the computer system 800 includes a central processing unit 801 (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory 802 (ROM) or the program loaded from the storage part 808 to the random access memory 803 (RAM). Various programs and data required for system operation are also stored in the random access memory 803. The central processing unit 801, the read-only memory 802 and the random access memory 803 are connected to each other through a bus 804. The input / output interface 805 (Input / Output interface, i.e., I / O interface) is also connected to the bus 804.

[0176] The following components are connected to the input / output interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed so that a computer program read therefrom is installed into the storage section 808 as needed.

[0177] In particular, according to an embodiment of the present application, the process described in each method flow chart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program contains a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processor 801, various functions defined in the system of the present application are executed.

[0178] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processor 801, various functions provided by the embodiment of the present application are performed.

[0179] According to another aspect of the embodiment of the present application, an electronic device for implementing the above event detection method is also provided. The electronic device may be Figure 1 The terminal device or server shown in the figure. This embodiment is described by taking the electronic device as a terminal device as an example. Fig. 9 As shown, the electronic device includes a memory 902 and a processor 904. The memory 902 stores a computer program, and the processor 904 is configured to execute the steps in any of the above method embodiments through the computer program.

[0180] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0181] Optionally, in this embodiment, the above-mentioned processor can be configured to execute the methods in each embodiment of the present application through a computer program.

[0182] Alternatively, a person skilled in the art may understand that: Fig. 9 The structure shown is for illustration only. Fig. 9 The structure of the electronic device is not limited. Fig. 9 More or fewer components (such as network interfaces, etc.) as shown in, or with Fig. 9 Different configurations are shown.

[0183] Among them, the memory 902 can be used to store software programs and modules, such as the program instructions / modules corresponding to the event detection method and device in the embodiments of the present application. The processor 904 executes various functional applications and data processing by running the software programs and modules stored in the memory 902, that is, realizing the above-mentioned event detection method. The memory 902 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 902 may further include a memory remotely located relative to the processor 904, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 902 can be specifically, but not limited to, used to store information such as audio signals. As an example, such as Fig. 9 As shown, the memory 902 may include but is not limited to the acquisition module 702, extraction module 704, determination module 706, and detection module 708 in the event detection device. In addition, it may also include but is not limited to other module units in the event detection device, which will not be repeated in this example.

[0184] Optionally, the transmission device 906 is used to receive or send data via a network. Specific examples of the network may include a wired network and a wireless network. In one example, the transmission device 906 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 906 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0185] In addition, the electronic device further includes: a display 908 for displaying the processing process of the audio signal; and a connection bus 910 for connecting various module components in the electronic device.

[0186] In other embodiments, the terminal device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the multiple nodes through network communication. The nodes may form a peer-to-peer network, and any form of computing device, such as a server, terminal or other electronic device, may become a node in the blockchain system by joining the peer-to-peer network.

[0187] According to one aspect of the present application, a computer-readable storage medium is provided, and a processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the event detection method provided in various optional implementations of the above-mentioned event detection aspects.

[0188] Optionally, in this embodiment, the above-mentioned computer-readable storage medium can be configured to store data for executing the methods in various embodiments of the present application.

[0189] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0190] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0191] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for one or more electronic devices to execute all or part of the steps of the methods described in each embodiment of the present application.

[0192] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0193] In the several embodiments provided in the present application, it should be understood that the disclosed application can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0194] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0195] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0196] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An event detection method, characterized in that: include: Acquire multiple groups of audio signals detected by multiple sensors, wherein the multiple groups of audio signals correspond to the multiple sensors one by one, each of the multiple sensors faces a different direction, and each sensor is used to collect a group of audio signals corresponding to at least two detection areas; Performing an audio feature extraction operation on each group of audio signals in the multiple groups of audio signals to obtain multiple audio feature values; Determine a target acquisition direction according to a value relationship between the multiple audio feature values ​​and the feature thresholds, and obtain thermal imaging frame information corresponding to the target acquisition direction, wherein the thermal imaging frame information is used to reflect the temperature distribution in the target acquisition direction; An event prediction result is determined based on the thermal imaging frame information, wherein the event prediction result is used to indicate whether a target event occurs.

2. The method according to claim 1, characterized in that: The performing audio feature extraction operations on each group of audio signals in the plurality of groups of audio signals to obtain a plurality of audio feature values ​​comprises: Performing short-time Fourier transform on a group of audio signals to obtain spectrum information matrices of the group of audio signals in different time windows, wherein each row of the spectrum information matrix represents spectrum information in a time window corresponding to the group of audio signals, and each column of the spectrum information matrix represents intensity distribution at a frequency corresponding to the group of audio signals; Performing local filtering on the spectrum information matrix to obtain a plurality of target noise arrays, wherein one target noise array among the plurality of target noise arrays corresponds to a row in the spectrum information matrix; According to the multiple target noise arrays, the noise components of the frequency spectrum information matrix are eliminated to obtain target audio eigenvalues, wherein the multiple audio eigenvalues ​​include the target audio eigenvalues.

3. The method according to claim 2, characterized in that The performing short-time Fourier transform on the group of audio signals to obtain a frequency spectrum information matrix of the group of audio signals in different time windows includes: Dividing the group of audio signals into a plurality of short-time frame signals, wherein one short-time frame signal represents an audio segment corresponding to the group of audio signals in a time dimension, and the group of audio signals corresponds to a plurality of audio segments; The multiple short-time frame signals are processed using a Hamming window function to obtain the spectrum information matrix.

4. The method according to claim 2, characterized in that: The locally filtering the spectrum information matrix to obtain multiple target noise arrays includes: Convert the values ​​of each element in the spectrum information matrix into absolute values ​​to obtain an intermediate spectrum information matrix; Each row of the intermediate spectrum information matrix is ​​divided into N intervals, where each interval includes i matrix elements, and N and i are both positive integers. The following operation is performed on each row to obtain N maximum local average values: In each interval, the average value between each matrix element and the adjacent matrix elements is calculated to obtain i local average values; Determine the maximum value among the i local average values ​​as the maximum local average value of the corresponding interval, and determine the value of each matrix element as the maximum local average value; The N maximum local averages are stored in their corresponding noise arrays respectively.

5. The method according to claim 4, characterized in that The method further comprises: Perform the following operation on each array element in the noise array to obtain the multiple target noise arrays: Calculating the average value of the array element and adjacent array elements respectively to obtain a target average value; The target average value is used to replace the value of the array element.

6. The method according to claim 4, characterized in that Eliminating the noise components of the spectrum information matrix according to the multiple target noise arrays to obtain the target audio feature value includes: The spectrum information matrix is ​​traversed, and each row is sequentially determined as a target row, wherein the target row includes M matrix elements, where M is a positive integer, and the following operations are performed on the target row until the spectrum information matrix traversal is completed: Determine a target noise array corresponding to the target row from among the multiple target noise arrays, wherein the target noise array includes M array elements, a matrix element in a j-th interval in the target row corresponds to a j-th array element in the target noise array, and j is less than M; Subtracting the value of the array element at the corresponding position in the one target noise array from the value of each matrix element of the target row to eliminate the noise component of the target row; Determine the frequency spectrum information matrix from which the noise component is eliminated as a target frequency spectrum information matrix, and determine a target matrix element in the target frequency spectrum information matrix whose value is greater than or equal to a frequency threshold; The values ​​of the target matrix elements are summed to obtain the target audio feature value.

7. The method according to claim 1, characterized in that The step of determining a target acquisition direction according to a value relationship between the plurality of audio feature values ​​and feature thresholds, and acquiring thermal imaging frame information corresponding to the target acquisition direction, includes: When at least one audio feature value among the multiple audio feature values ​​is greater than or equal to the feature threshold, determining the largest audio feature value among the multiple audio feature values ​​as a first audio feature value, and determining the largest audio feature value among the multiple audio feature values ​​except the first audio feature value as a second audio feature value; Determine a shift direction value based on the first audio feature value and the second audio feature value; The collection direction indicated by the first audio feature value is adjusted according to the offset direction value to determine the target collection direction.

8. The method according to claim 7, characterized in that The determining the offset direction value based on the first audio feature value and the second audio feature value comprises: Determine a sum of the first audio feature value and the second audio feature value as a feature sum value; Determine the product of the preset angle value and the second audio feature value as a feature product value; The value obtained by dividing the feature product value by the feature sum value is determined as the offset direction value.

9. The method according to claim 1, characterized in that: The obtaining of thermal imaging frame information corresponding to the target acquisition direction includes: Based on the target acquisition direction, control the thermal imaging frame acquisition device to rotate to the target acquisition direction; The thermal imaging frame acquisition device is controlled to acquire thermal imaging frame information corresponding to the target acquisition direction.

10. The method according to claim 1, characterized in that The method further comprises: generating a warning message when the event detection result indicates that the number of target objects in the target sub-area of ​​the target detection area has reached a maximum number, wherein the warning message indicates that the target sub-area needs to be evacuated; And / or when the event detection result indicates that the number of objects in the target sub-area does not reach the maximum number, reacquiring the multiple groups of audio signals.

11. The method according to claim 1, characterized in that: The determining of the event prediction result based on the thermal imaging frame information comprises: Inputting the thermal imaging frame information into an object detection model and outputting the event prediction result; The object detection model is used to identify object features in the thermal imaging frame information, and determine whether the target event occurs based on the object features.

12. An event detection device, characterized in that: include: An acquisition module, used to acquire multiple groups of audio signals detected by multiple sensors, wherein the multiple groups of audio signals correspond to the multiple sensors one by one, each of the multiple sensors faces a different direction, and each sensor is used to collect a group of audio signals corresponding to at least two detection areas; An extraction module, configured to perform an audio feature extraction operation on each of the plurality of audio signal groups to obtain a plurality of audio feature values; A determination module, used to determine a target acquisition direction according to a value relationship between the multiple audio feature values ​​and the feature threshold, and obtain thermal imaging frame information corresponding to the target acquisition direction, wherein the thermal imaging frame information is used to reflect the temperature distribution in the target acquisition direction; A detection module is used to determine an event prediction result based on the thermal imaging frame information, wherein the event prediction result is used to indicate whether a target event occurs.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein the computer program can be executed by an electronic device to perform the method described in any one of claims 1 to 11.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 11 are implemented.

15. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 11 through the computer program.

Images