Environment monitoring method and device, electronic equipment and storage medium
By using the causal forest model to causally infer the image and audio features in smart home systems, the false alarms and missed reports of existing systems when dealing with complex events are solved, and the monitoring accuracy and reliability of the system are improved.
Patent Information
- Application Number
- CN202411014434.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-12-06
AI Technical Summary
When handling multiple sensor inputs, existing smart home systems rely on simple trigger conditions and preset rules, lack an in-depth understanding of the causal relationship of complex events, resulting in abnormal false alarms and missed reports in user homes.
By obtaining the current image and audio in the environment, feature extraction is performed and image features and audio features are input into the causal forest model, causal inference analysis is performed to predict abnormal events in the environment and monitor them based on the predicted events.
It improves the in-depth understanding and handling of causal relationships in complex events, reduces false alarms and missed reports of environmental abnormalities, and enhances the accuracy and reliability of smart home systems.
Smart Images

Figure CN119091369A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of smart home appliances, and in particular to an environment monitoring method, device, electronic device and storage medium. Background Art
[0002] At present, the rapid development of smart homes has brought about highly automated and personalized home plumbing solutions. Among them, smart homes integrate a variety of sensors and can process a variety of sensor inputs (such as audio and video).
[0003] However, although existing smart homes can handle a variety of sensor inputs, they mainly rely on simple trigger conditions and preset rules, lacking in-depth understanding and processing capabilities of causal relationships in complex events, resulting in abnormal false alarms and missed alarms in users' homes. Summary of the invention
[0004] In order to solve the technical problem that the existing smart homes can process multiple sensor inputs, but they mainly rely on simple trigger conditions and preset rules, lack the in-depth understanding and processing capabilities of causal relationships in complex events, and lead to abnormal false alarms and missed alarms in the user's home, this application provides an environmental monitoring method, device, electronic device and storage medium. The specific technical solution is as follows:
[0005] In a first aspect, the present application provides an environmental monitoring method, the method comprising:
[0006] Acquire a current image in the environment, perform feature extraction on the current image, and obtain image features;
[0007] Acquire current audio in the environment, perform feature extraction on the current audio, and obtain audio features;
[0008] Inputting the image features and the audio features into a causal forest model to obtain a predicted event;
[0009] The environment is monitored according to the predicted event.
[0010] In an optional implementation, extracting features from the current image to obtain image features includes:
[0011] Using a filter to remove random noise in the current image, to obtain the current image with the random noise removed;
[0012] Normalizing the current image from which random noise is removed to obtain an image normalization result;
[0013] Perform feature extraction on the image normalization result to obtain image features.
[0014] In an optional implementation, extracting features from the current audio to obtain audio features includes:
[0015] Using a filter to remove background noise in the current audio, to obtain the current audio with the background noise removed;
[0016] Normalizing the current audio with background noise removed to obtain an audio normalization result;
[0017] Feature extraction is performed on the audio normalization result to obtain audio features.
[0018] In an optional implementation, the step of extracting features from the image normalization result to obtain image features includes:
[0019] Using a Canny edge detector or a Sobel edge detector to extract edge features in the image normalization result;
[0020] Extracting texture features from the image normalization result using a gray level co-occurrence matrix or a local binary pattern;
[0021] Using a color histogram or a color moment, extracting color distribution features in the image normalization result;
[0022] The edge features, the texture features, and the color distribution features are determined as image features.
[0023] In an optional implementation, extracting features from the audio normalization result to obtain audio features includes:
[0024] Using Fourier transform, feature extraction is performed on the audio normalization result to obtain frequency features;
[0025] Determine a square sum or a root mean square value of each audio frame in the audio normalization result, and extract an amplitude feature according to the square sum or the root mean square value;
[0026] Screening the activity segments whose energy exceeds a preset threshold from the audio normalization result, and extracting duration features according to the activity segments;
[0027] The frequency feature, the amplitude feature, and the duration feature are determined as audio features.
[0028] In an optional embodiment, the causal forest model includes a plurality of decision trees;
[0029] The step of inputting the image features and the audio features into a causal forest model to obtain a predicted event includes:
[0030] For any of the decision trees, input the image features and the audio features into the decision tree to obtain candidate events;
[0031] The candidate events of each decision tree are aggregated to obtain a predicted event, where the predicted event is any one of the candidate events of each decision tree.
[0032] In an optional implementation, monitoring the environment according to the predicted event includes:
[0033] Finding the event score corresponding to the predicted event, and determining whether the event score is greater than a preset score threshold;
[0034] When the event score is greater than the preset score threshold, it is determined that an abnormality corresponding to the predicted event occurs in the environment, and an alarm is issued.
[0035] In a second aspect, the present application provides an environment monitoring device, the device comprising:
[0036] An image feature extraction module is used to obtain a current image in the environment, perform feature extraction on the current image, and obtain image features;
[0037] An audio feature extraction module is used to obtain current audio in the environment, perform feature extraction on the current audio, and obtain audio features;
[0038] An event prediction module, used for inputting the image features and the audio features into a causal forest model to obtain a predicted event;
[0039] A monitoring module is used to monitor the environment according to the predicted event.
[0040] In a third aspect, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;
[0041] Memory, used to store computer programs;
[0042] The processor is used to implement any environmental monitoring method described in the first aspect when executing the program stored in the memory.
[0043] In a fourth aspect, a storage medium is also provided, wherein instructions are stored in the storage medium, and when the storage medium is run on a computer, the computer executes any of the environmental monitoring methods described in the first aspect.
[0044] In a fifth aspect, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned environmental monitoring methods.
[0045] The above-mentioned technical scheme provided by the embodiment of the present application has the following advantages compared with the prior art: the method provided by the embodiment of the present application obtains the current image in the environment, performs feature extraction on the current image to obtain image features, obtains the current audio in the environment, performs feature extraction on the current audio to obtain audio features, inputs the image features and the audio features into the causal forest model to obtain predicted events, and monitors the environment according to the predicted events.
[0046] By extracting image features and audio features and using the causal forest model to perform causal inference analysis on the image features and audio features, we can obtain predicted events. Based on the predicted events, we can monitor the environment. This can improve our in-depth understanding and processing capabilities of causal relationships in complex events and avoid false alarms and omissions of environmental anomalies. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0049] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.
[0050] Figure 1 A schematic diagram of an implementation flow of an environmental monitoring method provided in an embodiment of the present application;
[0051] Figure 2 A schematic diagram of an implementation flow of an audio feature extraction method provided in an embodiment of the present application;
[0052] Figure 3 A schematic diagram of an implementation flow of an image feature extraction method provided in an embodiment of the present application;
[0053] Figure 4 A schematic diagram of the structure of an environmental monitoring device provided in an embodiment of the present application;
[0054] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0056] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.
[0057] In an embodiment of the present application, for the environmental monitoring system, a central processing unit (CPU) is used to coordinate and analyze various sensor data from various places in the home (taking the home scene as an example, it can be extended to other scenes). Among them, multiple sound and image sensors, such as microphones and cameras, are deployed in the home, distributed in key locations in the home, such as entrances, windows, main passages, and specific room lights. All sensor data are transmitted to the central processing unit in real time, and the causal forest model running therein is responsible for processing and analyzing these sensor data.
[0058] Based on this, Figure 1 FIG. 1 is a schematic diagram of an implementation flow of an environmental monitoring method provided in an embodiment of the present application. The method may specifically include the following steps:
[0059] S101, obtaining a current image in an environment, performing feature extraction on the current image, and obtaining image features.
[0060] In the embodiment of the present application, taking the home scene as an example, the current images of various places in the environment can be obtained through cameras distributed throughout the home. Among them, the number of current images can be multiple, and each camera can collect a current image of its own monitoring area.
[0061] For the acquired current image, feature extraction can be performed on the current image to obtain image features. Wherein, there can be multiple current images, and feature extraction needs to be performed on any current image, so that multiple image features can be obtained.
[0062] It should be noted that the current image may be a static image or a dynamic image. The current image may include user activities, pet movement or other important events (such as broken windows), etc., which is not limited in the present embodiment of the application.
[0063] S102, obtaining current audio in the environment, performing feature extraction on the current audio, and obtaining audio features.
[0064] In the embodiment of the present application, taking the home scene as an example, the current audio in various places in the environment can be obtained through microphones distributed throughout the home. Among them, the number of current audios can be multiple, and each microphone can collect a current audio for the area it monitors.
[0065] For the acquired current audio, feature extraction can be performed on the current audio to obtain audio features. Wherein, the number of current audios can be multiple, and feature extraction is performed on any current audio, so multiple audio features can be obtained.
[0066] It should be noted that the current audio generally includes sounds generated by various household activities, such as conversations, TV sounds, and potential abnormal sounds (such as breaking sounds), etc., which are not limited in the embodiments of the present application.
[0067] S103, inputting the image features and the audio features into the causal forest model to obtain predicted events.
[0068] In an embodiment of the present application, the extracted image features and audio features may be input into a causal forest model to obtain a predicted event, which represents a certain event occurring in a home, such as broken glass.
[0069] S104, monitoring the environment according to the predicted event.
[0070] In an embodiment of the present application, by performing causal inference analysis on image features and audio features through a causal forest model, predicted events can be obtained, and the environment is monitored based on the predicted events. This can improve the in-depth understanding and processing capabilities of causal relationships in complex events and avoid false alarms and omissions of environmental anomalies.
[0071] Through the above description of the technical solution provided in the embodiment of the present application, the current image in the environment is obtained, and features are extracted from the current image to obtain image features. The current audio in the environment is obtained, and features are extracted from the current audio to obtain audio features. The image features and audio features are input into a causal forest model to obtain predicted events, and the environment is monitored based on the predicted events.
[0072] By extracting image features and audio features and using the causal forest model to perform causal inference analysis on the image features and audio features, we can obtain predicted events. Based on the predicted events, we can monitor the environment. This can improve our in-depth understanding and processing capabilities of causal relationships in complex events and avoid false alarms and omissions of environmental anomalies.
[0073] In addition, in the embodiment of the present application, denoising and normalization can be performed on the current image and the current audio. Denoising can clearly capture the key information of the event, while normalization can eliminate data deviations under different devices or different conditions.
[0074] To this end, for the current image, a filter can be used to remove random noise in the current image to obtain the current image with random noise removed, so as to clearly capture the key information of the event; the current image with random noise removed is normalized, that is, the current image with random noise removed is converted to a uniform scale, such as 0 to 1, or -1 to 1, to obtain an image normalization result, so as to eliminate data deviations under different devices or different conditions; the image normalization result is subjected to feature extraction to obtain image features. In this way, for the current image, after denoising and normalization, features are extracted to obtain image features.
[0075] Similarly, for the current audio, a filter can be used to remove the background noise in the current audio to obtain the current audio without background noise, so as to clearly capture the key information of the event; the current audio without background noise is normalized, that is, the current audio without background noise is converted to a uniform scale, such as 0 to 1, or -1 to 1, to obtain an audio normalization result, so as to eliminate data deviations under different devices or different conditions; the audio normalization result is feature extracted to obtain audio features. In this way, for the current audio, after denoising and normalization, features are extracted to obtain audio features.
[0076] In the embodiments of the present application, the audio features generally refer to features such as frequency, amplitude, duration, etc. These features can help identify the source and type of the audio. Figure 2 FIG. 1 is a schematic diagram of an implementation flow of an audio feature extraction method provided in an embodiment of the present application. The method may specifically include the following steps:
[0077] S201, using Fourier transform, extracting features from the audio normalization result to obtain frequency features.
[0078] In the embodiment of the present application, the current audio is subjected to denoising and normalization processing, and then features are extracted to obtain audio features. Among them, Fourier transform can be used to extract features from the audio normalization result after denoising and normalization processing to obtain frequency features.
[0079] It should be noted that, for Fourier Transform, such as Fast Fourier Transform (FFT), the audio signal in the time domain is converted to the frequency domain, which is not limited in the embodiments of the present application.
[0080] Specifically, the above audio normalization result may be divided into a plurality of small time periods (referred to as frames), each frame being usually 20 to 40 milliseconds, and a Hamming window function is applied to each frame to reduce spectrum leakage.
[0081] For each frame to which the Hamming window function is applied, for example, a fast Fourier transform is performed to obtain a frequency domain representation, at which time frequency features such as spectral centroid, spectral roll-off point, spectral bandwidth, etc. can be extracted.
[0082] S202, determining the sum of squares or the root mean square value of each audio frame in the audio normalization result, and extracting amplitude features according to the sum of squares or the root mean square value.
[0083] In the embodiment of the present application, the current audio is subjected to denoising and normalization processing, and then features are extracted to obtain audio features. Among them, the amplitude envelope of the audio normalization result after denoising and normalization processing can be directly analyzed.
[0084] Specifically, the sum of squares or the root mean square value of each audio frame in the audio normalization result is determined, and the amplitude feature is extracted according to the sum of squares or the root mean square value. The amplitude feature here can be, for example, the maximum value, minimum value, mean value, variance, etc. of the sum of squares or the root mean square value.
[0085] In addition, the above-mentioned amplitude features may also include features of changes between adjacent audio frames in the audio normalization result, such as the amplitude envelope change rate, etc., which is not limited in this embodiment of the present application.
[0086] S203, screening out activity segments whose energy exceeds a preset threshold from the audio normalization result, and extracting duration features according to the activity segments.
[0087] In the embodiment of the present application, the current audio is subjected to denoising and normalization processing, and then features are extracted to obtain audio features. The start and end times of an event or activity in the audio normalization result after denoising and normalization processing can be detected, and its duration can be calculated.
[0088] Based on this, active segments (segments exceeding a certain energy level) in the audio normalization results are detected by setting a threshold, that is, active segments whose energy exceeds the preset threshold are screened from the audio normalization results, and duration features are extracted based on the active segments.
[0089] There may be multiple activity segments. For each activity segment, its start time and end time are recorded, thereby calculating the duration of each activity segment and statistically analyzing the overall duration characteristics, such as average duration, maximum duration, activity frequency, etc.
[0090] S204: Determine the frequency feature, the amplitude feature, and the duration feature as audio features.
[0091] In an embodiment of the present application, after the above steps, frequency characteristics, amplitude characteristics, and duration characteristics can be obtained, thereby determining the frequency characteristics, amplitude characteristics, and duration characteristics as audio features, that is, the audio features to be extracted can be frequency characteristics, amplitude characteristics, duration characteristics, etc.
[0092] In addition, in an embodiment of the present application, other features may also be included for audio features, such as Mel-Frequency Cepstral Coefficients (MFCCs). For the audio normalization result, it is converted to the Mel frequency scale and the Mel frequency cepstral coefficients are calculated.
[0093] Specifically, for each audio frame in the audio normalization result, use FFT to convert it to the frequency domain, use the Mel filter bank to weight the spectrum, and obtain the Mel frequency energy spectrum. Take the logarithm of the Mel frequency energy spectrum, and then apply discrete cosine transform (DCT) to obtain MFCCs.
[0094] In addition, for audio features, other features may also be included, such as the zero crossing rate (ZCR), which calculates the number of times the waveform in the audio signal crosses the zero point. Specifically, for each audio frame in the audio normalization result, the number of zero crossing points is calculated, and then the overall zero crossing rate feature is counted.
[0095] Through the above steps, rich audio features can be extracted from the current audio, which can be used for the user's subsequent causal inference analysis and event detection.
[0096] In the embodiments of the present application, image features generally refer to features such as edges, textures, and color distribution, which can help identify objects and events in the current image. Figure 3 FIG. 1 is a schematic diagram of an implementation flow of an image feature extraction method provided in an embodiment of the present application. The method may specifically include the following steps:
[0097] S301, using a Canny edge detector or a Sobel edge detector to extract edge features in the image normalization result.
[0098] In the embodiment of the present application, for the current image, after denoising and normalization, features are extracted to obtain image features. Among them, the edge features in the image normalization result can be extracted using a Canny edge detector or a Sobel edge detector. Edge feature extraction is used to detect the contour and shape of objects in an image.
[0099] For the Canny edge detector, Gaussian filtering is performed on the normalized image result to smooth the image and reduce noise; then the gradient amplitude and direction of the image (i.e. the result after Gaussian filtering) are calculated, usually using the Sobel operator; then non-edge points are suppressed, that is, points with non-maximum gradient amplitude are removed; high and low dual threshold detection is applied to identify strong edges and potential edges; strong edges and potential edges are connected through edge tracking to form the final edge map.
[0100] For the Sobel edge detector, the normalized result of the image is converted into a grayscale image; the Sobel operator is used for convolution operation to calculate the gradients in the horizontal and vertical directions; the gradients in the horizontal and vertical directions are synthesized to obtain a gradient amplitude map.
[0101] S302, extracting texture features from the image normalization result using a gray level co-occurrence matrix or a local binary pattern.
[0102] In the embodiment of the present application, for the current image, after denoising and normalization, features are extracted to obtain image features. Among them, the texture features in the image normalization result can be extracted using the gray level co-occurrence matrix or the local binary pattern. The texture features are used to describe the repetitive patterns and structures in the image.
[0103] For the gray-level co-occurrence matrix (GLCM), the normalized image result is transformed, that is, converted into a grayscale image; the grayscale value combinations of pixel pairs (such as adjacent pixels) in the grayscale image are statistically combined to form a co-occurrence matrix; statistical features such as contrast, homogeneity, energy, entropy, etc. are extracted from the co-occurrence matrix as texture features.
[0104] For the local binary pattern (LBP), the image normalization result is converted into a grayscale image; for each pixel in the grayscale image, its grayscale value is compared with that of the surrounding pixels to generate a binary pattern (LBP code); the LBP codes in the entire image or image area are counted to form a histogram feature as a texture feature.
[0105] S303: Extract color distribution features from the image normalization result using a color histogram or a color moment.
[0106] In the embodiment of the present application, after the current image is subjected to denoising and normalization processing, features are extracted to obtain image features. Among them, the color distribution features in the image normalization result can be extracted using a color histogram or a color moment. The color distribution features are used to describe the color information in the image.
[0107] For the color histogram, for the above image normalization result, it is converted from the RGB color space to other color spaces, such as HSV or Lab, to reduce the impact of lighting; the histogram of each channel in the color space is calculated, and the frequency of each color value is counted as the color distribution feature.
[0108] For color moments, for the above image normalization results, they are converted from RGB color space to other color spaces, such as HSV or Lab, to reduce the impact of lighting; the statistical characteristics of the color channel, including the first-order moment (mean), second-order moment (variance), and third-order moment (skewness), are calculated as color distribution features.
[0109] S304: determining edge features, texture features, and color distribution features as image features.
[0110] In the embodiment of the present application, after the above steps, edge features, texture features, and color distribution features can be extracted, so that the edge features, texture features, and color distribution features can be determined as image features.
[0111] In addition, in an embodiment of the present application, the causal forest model includes multiple decision trees. For any of the decision trees, the image features and the audio features are input into the decision tree to obtain candidate events; the candidate events of each of the decision trees are aggregated to obtain a predicted event, and the predicted event is any one of the candidate events of each of the decision trees.
[0112] For a predicted event, find the event score corresponding to the predicted event, and determine whether the event score is greater than a preset score threshold; if the event score is greater than the preset score threshold, determine that an abnormality corresponding to the predicted event occurs in the environment, and issue an alarm.
[0113] In the embodiment of the present application, the above-mentioned image features and audio features can be used to construct a causal forest, which can be specifically divided into the following steps:
[0114] Sampling: Randomly extract samples from the above image features and audio features to build each decision tree. This process is called bootstrap sampling.
[0115] Feature selection and split point finding: At each decision node, a set of features is randomly selected, and statistical methods (such as calculating the maximum information gain and minimizing the mean square error) are used to determine the best split point.
[0116] Node splitting: Split the data into two subsets based on the selected split point. This process is repeated at each node of the tree until a preset stopping condition is met, such as reaching a minimum node size or a maximum tree depth.
[0117] Results aggregation: Each decision tree gives an estimate of the causal effect at its leaf node. After all decision trees are completed, the results of all relevant trees are averaged or aggregated through other statistical methods; the aggregated results form a comprehensive estimate of the causal effect of a specific data point, which reflects the expected output result given the input features.
[0118] Through this series of steps, the causal forest model can effectively learn and infer causal relationships from multimodal data, providing accurate event detection and response decisions for smart home systems. This approach leverages the power of machine learning to improve the intelligence of the system, enabling it to better serve the safety and convenience of users.
[0119] The following describes the environmental monitoring method provided by the embodiment of the present application in conjunction with the scenario of broken glass:
[0120] Data collection: A brief but intense sound fluctuation is captured by the microphone, while the camera captures the sudden change in the window area.
[0121] Pre-processing: Sound and image data are immediately converted into digital format, sound data is processed through denoising, and image data undergoes contrast and brightness adjustment to highlight abnormal areas.
[0122] Feature extraction: extract the frequency characteristics and amplitude from the sound, and extract the edges and textures of the moving area from the image.
[0123] Causal Forest Analysis: The causal forest model evaluates the temporal and spatial consistency of the shattering sound and the shattered window in the image based on the extracted features.
[0124] Decision trees independently evaluate the contribution of each feature in a causal forest and provide a composite causality score by aggregating the outputs of all trees.
[0125] Response decision: If the causal score is higher than the preset threshold, the system determines it as an intrusion event, automatically activates the alarm system, and notifies family members and security services.
[0126] Audio Processing:
[0127] First, the analog sound signal captured by the microphone is converted into a digital signal. This involves an analog-to-digital converter (ADC), which samples and quantizes the sound signal at a certain sampling rate.
[0128] The digital signal is then pre-processed using digital signal processing techniques, including denoising, such as using a bandpass filter to filter out noise at non-target frequencies, and dynamic range compression to enhance key features in the sound signal.
[0129] The intensity of the sound is evaluated by calculating the short-time energy of the sound signal. Short-time energy refers to the sum of the squares of the amplitude of the sound signal within a short time window, which can effectively indicate the intensity level of the sound.
[0130] The duration of a sound is determined by analyzing the start and end points of the sound signal. Generally, a sound is short in duration and has abrupt start and end characteristics indicating that it is a transient event.
[0131] Perform a Fast Fourier Transform (FFT) on the sound signal to analyze its spectral characteristics. Sudden high-decibel sounds such as breaking glass usually show a sudden increase in frequency within a certain range.
[0132] Combining the above characteristics, such as high short-term energy, short duration and specific frequency distribution, the system's preset threshold judgment is used to identify whether this type of sound meets the characteristics of "short and intense sound fluctuations".
[0133] Image Processing:
[0134] The camera captures images of the home interior in real time. All images are first digitally processed, including adjusting brightness, contrast, and applying image denoising algorithms to improve image quality.
[0135] Using the inter-frame difference method, the system compares the differences between two or more consecutive frames. By calculating the changes in pixels, moving objects can be detected, such as window glass fragments that move or fall due to external force.
[0136] Apply edge detection algorithms (such as the Canny algorithm) to identify significant changes in edges in the image. This is particularly effective in sudden events such as glass breaking, because fragments and cracks will create new edges in the image.
[0137] Evaluation process:
[0138] (1) Sound and image features have been extracted from a potential event (e.g., window glass breaking). These features may include:
[0139] Sound characteristics:
[0140] Decibel level: 90dB.
[0141] Duration: 0.5 seconds.
[0142] Main frequency: high frequency area.
[0143] Image features:
[0144] Obvious signs of motion detection: rapid pixel changes in the window area.
[0145] Number of new edges appearing: High.
[0146] Light change: sudden brightening.
[0147] These features are integrated into feature vectors and input into each decision tree in the pre-trained causal forest.
[0148] (2) Independent evaluation process of decision trees
[0149] Each decision tree independently evaluates the input feature vector according to the decision logic learned during its training. For example:
[0150] Tree A might focus on sound features, specifically decibel level and duration, and classify events based on those features.
[0151] Tree B may emphasize motion detection and edge changes in image features.
[0152] The tree C can combine all the features of the sound and the image and evaluate the impact of these combinations of features.
[0153] Each decision tree ultimately produces a prediction at its leaf nodes, which could be “normal,” “glass broken,” or other events.
[0154] (3) Aggregation
[0155] The aggregation process aims to integrate the outputs of all decision trees to form a comprehensive judgment of the event.
[0156] The specific method is as follows:
[0157] Majority voting: The most commonly used method is majority voting, which selects the event that appears most frequently in the tree's predictions as the final predicted event. For example, if most trees predict that the event is "glass breaking", the overall predicted event is "glass breaking".
[0158] Average Probability Method: Another approach is to calculate the average probability of each event, especially when each tree outputs probabilities instead of events. This means that each tree provides a probability distribution, the probability distributions of all trees are averaged, and the final predicted event is the event with the highest probability.
[0159] (4) Causality score
[0160] The final aggregation result can be further transformed into a causality score, which reflects the likelihood and severity of a specific event. For example, the aggregated predicted event "glass breaking" may correspond to a high causality score, which reflects a high confidence in the occurrence of the event.
[0161] Corresponding to the above method embodiment, the present application embodiment also provides an environment monitoring device, such as Figure 4 As shown, the device may include: an image feature extraction module 410 , an audio feature extraction module 420 , an event prediction module 430 , and a monitoring module 440 .
[0162] The image feature extraction module 410 is used to obtain a current image in the environment, perform feature extraction on the current image, and obtain image features;
[0163] The audio feature extraction module 420 is used to obtain the current audio in the environment, perform feature extraction on the current audio, and obtain audio features;
[0164] An event prediction module 430, configured to input the image features and the audio features into a causal forest model to obtain a predicted event;
[0165] The monitoring module 440 is used to monitor the environment according to the predicted event.
[0166] The present application also provides an electronic device, such as Figure 5 As shown, it includes a processor 51, a communication interface 52, a memory 53 and a communication bus 54, wherein the processor 51, the communication interface 52, and the memory 53 communicate with each other through the communication bus 54.
[0167] A memory 53, for storing computer programs;
[0168] The processor 51 is used to execute the program stored in the memory 53 to implement the following steps:
[0169] Acquire a current image in an environment, perform feature extraction on the current image, and obtain image features; acquire current audio in the environment, perform feature extraction on the current audio, and obtain audio features; input the image features and the audio features into a causal forest model to obtain predicted events; and monitor the environment according to the predicted events.
[0170] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0171] The communication interface is used for communication between the above air conditioner and other devices.
[0172] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0173] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0174] In another embodiment provided in the present application, a storage medium is also provided, in which instructions are stored. When the storage medium is executed on a computer, the computer executes the environment monitoring method described in any one of the above embodiments.
[0175] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the environment monitoring method described in any one of the above embodiments.
[0176] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a storage medium, or transmitted from one storage medium to another storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.
[0177] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0178] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0179] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.
Claims
1. An environmental monitoring method, characterized in that: The method comprises: Acquire a current image in the environment, perform feature extraction on the current image, and obtain image features; Acquire current audio in the environment, perform feature extraction on the current audio, and obtain audio features; Inputting the image features and the audio features into a causal forest model to obtain a predicted event; The environment is monitored according to the predicted event.
2. The method according to claim 1, characterized in that The step of extracting features from the current image to obtain image features includes: Using a filter to remove random noise in the current image, to obtain the current image with the random noise removed; Performing normalization processing on the current image from which random noise is removed to obtain an image normalization result; Perform feature extraction on the image normalization result to obtain image features.
3. The method according to claim 1, characterized in that The extracting features of the current audio to obtain audio features includes: Using a filter to remove background noise in the current audio, to obtain the current audio with the background noise removed; Normalizing the current audio with background noise removed to obtain an audio normalization result; Feature extraction is performed on the audio normalization result to obtain audio features.
4. The method according to claim 2, characterized in that: The step of extracting features from the image normalization result to obtain image features includes: Using a Canny edge detector or a Sobel edge detector to extract edge features in the image normalization result; Extracting texture features from the image normalization result using a gray level co-occurrence matrix or a local binary pattern; Using a color histogram or a color moment, extracting color distribution features in the image normalization result; The edge features, the texture features, and the color distribution features are determined as image features.
5. The method according to claim 3, characterized in that: The step of extracting features from the audio normalization result to obtain audio features includes: Using Fourier transform, feature extraction is performed on the audio normalization result to obtain frequency features; Determine a square sum or a root mean square value of each audio frame in the audio normalization result, and extract an amplitude feature according to the square sum or the root mean square value; Screening the activity segments whose energy exceeds a preset threshold from the audio normalization result, and extracting duration features according to the activity segments; The frequency feature, the amplitude feature, and the duration feature are determined as audio features.
6. The method according to claim 1, characterized in that The causal forest model includes a plurality of decision trees; The step of inputting the image features and the audio features into a causal forest model to obtain a predicted event includes: For any of the decision trees, input the image features and the audio features into the decision tree to obtain candidate events; The candidate events of each decision tree are aggregated to obtain a predicted event, where the predicted event is any one of the candidate events of each decision tree.
7. The method according to claim 1, characterized in that The monitoring of the environment according to the predicted event includes: Finding the event score corresponding to the predicted event, and determining whether the event score is greater than a preset score threshold; When the event score is greater than the preset score threshold, it is determined that an abnormality corresponding to the predicted event occurs in the environment, and an alarm is issued.
8. An environmental monitoring device, characterized in that: The device comprises: An image feature extraction module is used to obtain a current image in the environment, perform feature extraction on the current image, and obtain image features; An audio feature extraction module is used to obtain current audio in the environment, perform feature extraction on the current audio, and obtain audio features; An event prediction module, used for inputting the image features and the audio features into a causal forest model to obtain a predicted event; A monitoring module is used to monitor the environment according to the predicted event.
9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-7 when executing a program stored in a memory.
10. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Monitoring method and device for abnormal event , electronic equipment and storage medium
CN110991289A
Moving object identification method, device and system based on random forest algorithm
CN114398958A
Analysis method for classifying non-human primate neurons by adopting multi-modal information
CN116628601A
