Intelligent monitoring method and monitoring system of artificial intelligence camera

Abstract

The invention provides an intelligent monitoring method and monitoring system of an artificial intelligence camera. The method comprises the following steps: acquiring a monitoring video stream; performing color adjustment on the monitoring video stream to obtain an adjusted monitoring video stream, selecting a monitoring video key frame from the adjusted monitoring video stream, and determining the similarity between adjacent frames corresponding to the monitoring video key frame; performing decision difference on the monitoring video key frame based on the similarity between adjacent frames to obtain a decision difference video frame of the monitoring area, and determining a monitoring decision value of the artificial intelligence camera in the monitoring area through the decision difference video frame; the method comprises the following steps: acquiring an audio data stream, performing threshold segmentation on the audio data stream to obtain a monitoring audio segment, and performing feature construction on the monitoring audio segment to obtain a fundamental tone feature vector of a monitoring area; and according to the monitoring decision value and the fundamental tone feature vector, carrying out early warning on an abnormal condition of the monitoring area. Based on the above scheme, the environmental interference of the monitoring area can be suppressed during anomaly detection, so that intelligent early warning of the artificial intelligence camera is realized.

Description

Technical Field

[0001] The present application relates to the field of intelligent monitoring technology, and more specifically, to an intelligent monitoring method and monitoring system for an artificial intelligence camera. Background Art

[0002] An artificial intelligence (AI) camera is a camera that uses artificial intelligence technology to enhance its functionality. AI cameras are widely used in security surveillance, smart homes, personal electronic devices, traffic monitoring, industrial automation, retail and service industries, etc. Compared with traditional cameras, AI cameras can perform complex tasks such as image recognition, object detection, motion recognition, facial recognition, etc., providing smarter and more automated services.

[0003] Artificial intelligence cameras require a lot of computing resources for real-time video processing and analysis, especially for complex deep learning models, which require high-performance hardware support. The intelligent monitoring of artificial intelligence cameras uses deep learning and computer vision technology to analyze monitoring video streams in real time, realize target detection, abnormal behavior recognition and other functions, and improve the intelligence and response speed of intelligent monitoring systems. However, in the existing technology, the intelligent monitoring system will generate false alarms due to interference factors, that is, it will mistakenly identify normal behavior as abnormal behavior, which will reduce the reliability and availability of the intelligent monitoring system. Therefore, how to suppress environmental interference in the monitoring area when performing anomaly detection to achieve intelligent early warning of artificial intelligence cameras is a difficult problem that the industry needs to solve urgently. Summary of the invention

[0004] The present application provides an intelligent monitoring method and monitoring system for an artificial intelligence camera, which can suppress environmental interference in the monitoring area when performing abnormality detection to achieve intelligent early warning of the artificial intelligence camera.

[0005] In a first aspect, the present application provides an intelligent monitoring method of an artificial intelligence camera, comprising the following steps: Obtain the surveillance video stream shot by the AI ​​camera in the surveillance area; Performing color calibration on the surveillance video stream to obtain a calibrated surveillance video stream, selecting surveillance video key frames from the calibrated surveillance video stream, and then determining the similarity between adjacent frames corresponding to the surveillance video key frames; Perform decision differentiation on the key frames of the monitoring video based on the similarity between adjacent frames to obtain a decision differential video frame of the monitoring area, and determine the monitoring decision value of the artificial intelligence camera in the monitoring area through the decision differential video frame; Acquire the audio data stream recorded by the artificial intelligence camera in the monitoring area, perform threshold segmentation on the audio data stream based on the short-time energy and zero-crossing rate of the audio data stream to obtain a monitoring audio segment, and then perform feature construction on the monitoring audio segment to obtain a fundamental pitch feature vector of the monitoring area; An early warning is issued for abnormal conditions in the monitoring area based on the monitoring decision value and the fundamental pitch feature vector.

[0006] In some embodiments, color calibration is performed on the surveillance video stream to obtain the calibrated surveillance video stream, specifically including: Performing color balancing on the surveillance video stream to obtain a balanced surveillance video stream; Performing color correction on the balanced surveillance video stream to obtain a calibrated surveillance video stream.

[0007] In some embodiments, determining the similarity between adjacent frames corresponding to the key frame of the surveillance video specifically includes: Obtaining adjacent surveillance video frames corresponding to the surveillance video key frames in the adjusted surveillance video stream; Determine the color distribution feature vectors of the surveillance video key frame and the adjacent surveillance video frame respectively; The similarity between adjacent frames corresponding to the key frame of the surveillance video is determined based on the color distribution feature vector of the key frame of the surveillance video and the color distribution feature vector of the adjacent surveillance video frame.

[0008] In some embodiments, performing decision differentiation on the key frames of the surveillance video based on the similarity between adjacent frames to obtain the decision differential video frames of the surveillance area specifically includes: When the similarity between the adjacent frames is greater than a preset decision value, the surveillance video key frame is differentiated from the first adjacent surveillance video frame and the second adjacent surveillance video frame in the corresponding adjacent surveillance video frames to obtain a first differential video frame and a second differential video frame; Binarizing the first differential video frame and the second differential video frame respectively to obtain a binarized first differential video frame and a binarized second differential video frame; A decision differential video frame of the monitoring area is determined according to the binarized first differential video frame and the binarized second differential video frame.

[0009] In some embodiments, performing threshold segmentation on the audio data stream based on the short-time energy and zero-crossing rate of the audio data stream is to perform threshold segmentation on the audio data stream by using the short-time energy and the zero-crossing rate as the short-time energy threshold and the zero-crossing rate threshold respectively.

[0010] In some embodiments, constructing features for the monitored audio segment to obtain a pitch feature vector of the monitored area specifically includes: Determining the scale characteristics of the monitoring audio segment; Extracting the fundamental frequency value of each frame of audio in the monitoring audio segment, thereby obtaining all fundamental frequency values, and determining the fundamental frequency mean of the monitoring audio segment according to all fundamental frequency values; Determining the unvoiced sound fluctuation characteristics of the monitoring audio segment; A pitch feature vector of the monitoring area is constructed by using the scale feature, the pitch mean and the unvoiced sound fluctuation feature.

[0011] In some embodiments, issuing an early warning for abnormal conditions in a monitoring area based on the monitoring decision value and the pitch feature vector specifically includes: When the monitoring decision value does not exceed the set threshold, it is determined that there is no abnormality in the monitoring area of ​​the artificial intelligence camera; When the monitoring decision value exceeds a set threshold, the monitoring video stream and the pitch feature vector are input into an anomaly detection model to obtain an anomaly detection result of the monitoring area; An early warning is issued for abnormal conditions in the monitored area based on the abnormal detection results.

[0012] In a second aspect, the present application provides an intelligent monitoring system of an artificial intelligence camera, including an early warning unit, wherein the early warning unit includes: An acquisition module is used to acquire the surveillance video stream shot by the artificial intelligence camera in the surveillance area; A processing module, used for color calibration of the monitoring video stream to obtain a calibrated monitoring video stream, selecting a monitoring video key frame from the calibrated monitoring video stream, and then determining the similarity between adjacent frames corresponding to the monitoring video key frame; The processing module is further used to perform decision differentiation on the key frames of the monitoring video based on the similarity between adjacent frames to obtain a decision differential video frame of the monitoring area, and determine the monitoring decision value of the artificial intelligence camera in the monitoring area through the decision differential video frame; The processing module is further used to obtain the audio data stream recorded by the artificial intelligence camera in the monitoring area, perform threshold segmentation on the audio data stream based on the short-time energy and zero-crossing rate of the audio data stream to obtain a monitoring audio segment, and then perform feature construction on the monitoring audio segment to obtain a fundamental tone feature vector of the monitoring area; An execution module is used to issue an early warning of abnormal conditions in a monitoring area according to the monitoring decision value and the fundamental pitch feature vector.

[0013] In a third aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the computer device executes the above-mentioned intelligent monitoring method of the artificial intelligence camera.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium, in which instructions or codes are stored. When the instructions or codes are run on a computer, the computer implements the above-mentioned intelligent monitoring method of the artificial intelligence camera when executed.

[0015] The technical solution provided by the embodiments disclosed in this application has the following beneficial effects: The present application provides an intelligent monitoring method and monitoring system for an artificial intelligence camera, which comprises the following steps: obtaining a monitoring video stream shot by the artificial intelligence camera in a monitoring area; performing color correction on the monitoring video stream to obtain a corrected monitoring video stream, selecting monitoring video key frames from the corrected monitoring video stream, and then determining the similarity between adjacent frames corresponding to the monitoring video key frames; performing decision differential on the monitoring video key frames based on the similarity between adjacent frames to obtain a decision differential video frame of the monitoring area, and determining a monitoring decision value of the artificial intelligence camera in the monitoring area through the decision differential video frame; obtaining an audio data stream recorded by the artificial intelligence camera in the monitoring area, performing threshold segmentation on the audio data stream based on the short-time energy and zero-crossing rate of the audio data stream to obtain a monitoring audio segment, and then performing feature construction on the monitoring audio segment to obtain a fundamental tone feature vector of the monitoring area; and issuing an early warning of abnormal conditions in the monitoring area based on the monitoring decision value and the fundamental tone feature vector.

[0016] It can be seen that in this application, by combining visual information (monitoring decision value) and auditory information (pitch feature vector) to warn of abnormal situations in the monitoring area, the detection accuracy of abnormal behavior by the intelligent monitoring system can be improved to realize the intelligent warning of the artificial intelligence camera; wherein, firstly, the color-adjusted monitoring video stream is more consistent in color, so that a more accurate similarity between adjacent frames can be obtained, so that the moving target object in the monitoring area can be better identified based on the similarity between adjacent frames; secondly, the monitoring decision value determined based on the decision difference video frame is used to identify abnormal behavior, thereby realizing real-time monitoring and warning of abnormal behavior in the monitoring area; then, the pitch feature vector extracted based on the audio data stream can provide an additional feature dimension to help distinguish different sound patterns and voice features; finally, after judging based on the monitoring decision value and the pitch feature vector, the abnormal situation in the monitoring area is warned according to the judgment result, which can make up for the limitations and misjudgments that may exist in a single information source, thereby enhancing the accuracy and reliability of abnormal detection, and when an abnormal situation is detected, the warning mechanism can be triggered immediately; in summary, the technical solution provided by the present application suppresses environmental interference in the monitoring area when performing abnormal detection, so as to realize the intelligent warning of the artificial intelligence camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0018] Figure 1 is an exemplary flow chart of an intelligent monitoring method of an artificial intelligence camera according to some embodiments of the present application; Figure 2 is an exemplary flow chart of determining the similarity between adjacent frames according to some embodiments of the present application; Figure 3 is an exemplary flow chart of determining a pitch feature vector according to some embodiments of the present application; Figure 4 is a schematic diagram of exemplary hardware and / or software of an early warning unit according to some embodiments of the present application; Figure 5 It is a structural diagram of a computer device for implementing an intelligent monitoring method of an artificial intelligence camera according to some embodiments of the present application. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] The embodiment of the present application provides an intelligent monitoring method and monitoring system for an artificial intelligence camera, the core of which is to obtain a monitoring video stream shot by an artificial intelligence camera in a monitoring area; perform color calibration on the monitoring video stream to obtain a calibrated monitoring video stream, select a monitoring video key frame from the calibrated monitoring video stream, and then determine the similarity between adjacent frames corresponding to the monitoring video key frame; perform decision differential on the monitoring video key frame based on the similarity between adjacent frames to obtain a decision differential video frame of the monitoring area, and determine the monitoring decision value of the artificial intelligence camera in the monitoring area through the decision differential video frame; obtain an audio data stream recorded by the artificial intelligence camera in the monitoring area, perform threshold segmentation on the audio data stream based on the short-time energy and zero-crossing rate of the audio data stream to obtain a monitoring audio segment, and then perform feature construction on the monitoring audio segment to obtain a fundamental tone feature vector of the monitoring area; and warn of abnormal conditions in the monitoring area based on the monitoring decision value and the fundamental tone feature vector. Based on the above scheme, environmental interference in the monitoring area can be suppressed when performing abnormality detection to achieve intelligent warning of the artificial intelligence camera.

[0021] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods. Figure 1 , which is an exemplary flow chart of an intelligent monitoring method of an artificial intelligence camera according to some embodiments of the present application. The intelligent monitoring method 100 of the artificial intelligence camera mainly includes the following steps: In step 101, a surveillance video stream captured by an artificial intelligence camera in a surveillance area is obtained.

[0022] It should be noted that, in this application, the monitoring video stream refers to the video of the monitoring area taken by the artificial intelligence camera within a specified time period, and the monitoring video stream is composed of multiple monitoring video frames in chronological order; in the specific implementation, first, you can log in and access the management interface of the intelligent monitoring system or the related application; then, select the artificial intelligence camera device for which to obtain the monitoring video stream in the intelligent monitoring system, and confirm its location and monitoring area; finally, obtain the monitoring video stream in the interface of the intelligent monitoring system.

[0023] In step 102, color calibration is performed on the surveillance video stream to obtain a calibrated surveillance video stream, and surveillance video key frames are selected from the calibrated surveillance video stream to determine the similarity between adjacent frames corresponding to the surveillance video key frames.

[0024] In some embodiments, color calibration is performed on the surveillance video stream to obtain the calibrated surveillance video stream in the following manner, namely: Performing color balancing on the surveillance video stream to obtain a balanced surveillance video stream; Performing color correction on the balanced surveillance video stream to obtain a calibrated surveillance video stream.

[0025] In specific implementation, first, color balancing of the surveillance video stream can be achieved by adjusting the gain and offset of different color channels of the surveillance video stream. For example, the color temperature deviation in the image can be eliminated by adjusting the gain of the red, green and blue channels to achieve the balance of different color channels, thereby obtaining a balanced surveillance video stream; then, color correction is performed on the balanced surveillance video stream to correct the color deviation or bad color in the image. The color deviation in the surveillance video stream can be corrected by adjusting the correction curves of different color channels, thereby obtaining a calibrated surveillance video stream.

[0026] In some embodiments, a surveillance video key frame is selected from the adjusted surveillance video stream; in specific implementation, a second surveillance video frame can be selected from the adjusted surveillance video stream as an initial surveillance video key frame, and the surveillance video key frame is used to identify moving targets in the monitoring area.

[0027] Preferably, in some embodiments, reference Figure 2 As shown in FIG. 1 , this figure is an exemplary flow chart of determining the similarity between adjacent frames in some embodiments of the present application. In this embodiment, determining the similarity between adjacent frames corresponding to the key frames of the surveillance video can be implemented by the following steps: First, in step 1021, adjacent surveillance video frames corresponding to the surveillance video key frames in the adjusted surveillance video stream are obtained; Then, in step 1022, the color distribution feature vectors of the surveillance video key frame and the adjacent surveillance video frame are determined respectively; Finally, in step 1023, the similarity between adjacent frames corresponding to the key frame of the surveillance video is determined based on the color distribution feature vector of the key frame of the surveillance video and the color distribution feature vector of the adjacent surveillance video frame.

[0028] It should be noted that, in the present application, the color distribution feature vector is a feature vector used to represent the color distribution information of the video frame, the similarity between adjacent frames is an indicator used to represent the similarity between the surveillance video key frame and the adjacent surveillance video frame, and the similarity between adjacent frames includes the similarity between the first adjacent frames and the similarity between the second adjacent frames; in specific implementation, first, the adjacent surveillance video frames of the surveillance video key frame in the adjusted surveillance video stream are obtained, and the adjacent surveillance video frames are composed of the first adjacent surveillance video frame and the second adjacent surveillance video frame. In the present application, when the surveillance video key frame is the second surveillance video frame in the surveillance video stream, the first adjacent surveillance video frame and the second adjacent surveillance video frame are respectively the first surveillance video frame and the third surveillance video frame in the adjusted surveillance video stream; then, for the surveillance video key frame, the surveillance video key frame can be calculated. The first-order moment, second-order moment and third-order moment of the color of the monitoring video key frame are obtained, and the feature vector composed of the first-order moment, second-order moment and third-order moment of the color of the monitoring video key frame is used as the color distribution feature vector of the monitoring video key frame. The color distribution feature vector of the adjacent monitoring video frames of the monitoring video key frame can be obtained in the above manner; finally, the Bhattacharyya distance between the color distribution feature vector of the monitoring video key frame and the color distribution feature vector of the first adjacent monitoring video frame in the adjacent monitoring video frames can be used as the first adjacent frame similarity, and the Bhattacharyya distance between the color distribution feature vector of the monitoring video key frame and the color distribution feature vector of the second adjacent monitoring video frame in the adjacent monitoring video frames can be used as the second adjacent frame similarity, so that the adjacent frame similarity corresponding to the monitoring video key frame is obtained through the first adjacent frame similarity and the second adjacent frame similarity.

[0029] It should be noted that color calibration can improve the visual effect of the surveillance video stream and make the surveillance video stream clearer. The surveillance video stream that has been color-calibrated is more consistent in color and visual effect. Therefore, when performing similarity analysis between adjacent frames, more accurate results can be obtained, thereby facilitating the identification of moving target objects in the surveillance area based on the similarity between adjacent frames.

[0030] In step 103, decision differentiation is performed on the key frames of the monitoring video based on the similarity between adjacent frames to obtain decision differential video frames of the monitoring area, and the monitoring decision value of the artificial intelligence camera in the monitoring area is determined by the decision differential video frames.

[0031] In some embodiments, the decision difference of the key frames of the surveillance video is performed based on the similarity between the adjacent frames to obtain the decision difference video frame of the surveillance area, which can be specifically obtained in the following manner, namely: When the similarity between the adjacent frames is greater than a preset decision value, the surveillance video key frame is differentiated from the first adjacent surveillance video frame and the second adjacent surveillance video frame in the corresponding adjacent surveillance video frames to obtain a first differential video frame and a second differential video frame; Binarizing the first differential video frame and the second differential video frame respectively to obtain a binarized first differential video frame and a binarized second differential video frame; A decision differential video frame of the monitoring area is determined according to the binarized first differential video frame and the binarized second differential video frame.

[0032] In specific implementation, first, the preset decision value can be obtained by means of the maximum inter-class variance; then, when the similarity between adjacent frames is greater than the preset decision value, that is, the similarity between the first adjacent frames and the similarity between the second adjacent frames in the similarity between adjacent frames are both greater than the preset decision value, the monitoring video key frame and the first adjacent monitoring video frame can be differentiated, that is, the difference between each pixel in the monitoring video key frame and the first adjacent monitoring video frame is calculated respectively, and the result is used as the first differential video frame, and the second differential video frame can be obtained by the above method; secondly, a threshold is preset, and the first differential video frame and the second differential video frame can be binarized respectively by the threshold to obtain the binarized first differential video frame and the binarized second differential video frame; finally, the binarized first differential video frame and the binarized second differential video frame can be ANDed, and the result obtained can be used as the decision differential video frame of the monitoring area.

[0033] It should be noted that when the similarity between adjacent frames is not greater than the preset decision value, it is necessary to reselect the surveillance video key frame in the surveillance video stream and repeat the steps after selecting the surveillance video key frame to obtain the similarity between adjacent frames of the reselected surveillance video key frame, and compare the similarity between adjacent frames with the preset decision value until the similarity between adjacent frames is greater than the preset decision value.

[0034] Preferably, in some embodiments, the monitoring decision value of the artificial intelligence camera in the monitoring area is determined by the decision differential video frame; it should be noted that in the present application, the monitoring decision value is an indicator for judging whether there is a moving target in the monitoring area of ​​the artificial intelligence camera; in specific implementation, the total number of pixels with a pixel value of 1 in the decision differential video frame can be counted, and the ratio of the total number of pixels to the total number of pixels in the decision differential video frame can be used as the monitoring decision value of the artificial intelligence camera in the monitoring area. Through the monitoring decision value, the situation of whether there is a moving target in the monitoring area monitored by the artificial intelligence camera can be quantified, thereby realizing intelligent monitoring of the monitoring area.

[0035] It should be noted that the decision differential video frame can help identify moving targets in the monitoring area, and the monitoring decision value determined based on the decision differential video frame is used to identify abnormal behavior, thereby realizing real-time monitoring and early warning of abnormal behavior in the monitoring area.

[0036] In step 104, the audio data stream recorded by the artificial intelligence camera in the monitoring area is obtained, and the audio data stream is threshold-segmented based on the short-time energy and zero-crossing rate of the audio data stream to obtain a monitoring audio segment, and then the monitoring audio segment is feature constructed to obtain the fundamental frequency feature vector of the monitoring area.

[0037] In some embodiments, an audio data stream recorded by an artificial intelligence camera in a monitoring area is obtained; it should be noted that, in the present application, the audio data stream refers to the audio of the monitoring area recorded by the artificial intelligence camera within a specified time period, and the audio data stream is composed of multiple audio frames in chronological order; in specific implementation, first, you can log in and access the management interface of the intelligent monitoring system or related applications; then, select the artificial intelligence camera device for which to obtain the audio data stream in the intelligent monitoring system, and confirm its location and monitoring area; finally, obtain the audio data stream in the interface of the intelligent monitoring system.

[0038] In some embodiments, the audio data stream is threshold segmented based on the short-time energy and zero-crossing rate of the audio data stream to obtain a monitoring audio segment. In a specific implementation, first, the short-time energy of each frame of audio in the audio data stream can be determined, and the average of the short-time energies of all audio frames is used as the short-time energy of the audio data stream, and the short-time energy is used to reflect the characteristics of the voiced sound part in the audio data stream; then, the zero-crossing rate of each frame of audio in the audio data stream can be determined, and the average of the zero-crossing rates of all audio frames is used as the zero-crossing rate of the audio data stream, and the zero-crossing rate is used to reflect the characteristics of the voiced sound part in the audio data stream. characteristics of the unvoiced part in the audio data stream; finally, threshold segmentation is performed on the audio data stream based on the short-time energy and zero-crossing rate of the audio data stream, that is, the short-time energy and the zero-crossing rate are used as the short-time energy threshold and the zero-crossing rate threshold to perform threshold segmentation on the audio data stream respectively, and the short-time energy threshold and the zero-crossing rate threshold can be input into a double threshold endpoint detection algorithm to perform threshold segmentation on the audio data stream to obtain a monitoring audio segment. Through threshold segmentation, background noise and interference content in the audio data stream can be removed, so that the monitoring audio segment contains the audio characteristics of the monitoring area required for intelligent monitoring.

[0039] Preferably, in some embodiments, reference Figure 3 As shown in FIG. 1 , this figure is an exemplary flow chart of determining a fundamental pitch feature vector in some embodiments of the present application. In this embodiment, the monitoring audio segment is feature constructed to obtain the fundamental pitch feature vector of the monitoring area, which can be implemented by the following steps: First, in step 1041, the scale feature of the monitoring audio segment is determined; Next, in step 1042, the fundamental frequency value of each frame of the monitoring audio segment is extracted, and then all the fundamental frequency values ​​are obtained, and the fundamental frequency mean of the monitoring audio segment is determined according to all the fundamental frequency values; Then, in step 1043, the unvoiced sound fluctuation characteristics of the monitoring audio segment are determined; Finally, in step 1044, a pitch feature vector of the monitoring area is constructed by using the scale feature, the pitch mean, and the unvoiced sound fluctuation feature.

[0040] It should be noted that in the present application, the fundamental frequency feature vector is a feature vector used to describe the audio feature information of the area monitored by the artificial intelligence camera; in the specific implementation, first, the scale feature of the monitoring audio segment can be calculated by the Mel filter group, and the scale feature is used to represent the overall scale characteristics of the monitoring audio segment; secondly, the fundamental frequency value of each frame of audio in the monitoring audio segment can be extracted by the cepstrum method, so as to obtain all the fundamental frequency values, and the fundamental frequency value represents the basic frequency value of the audio frame, and the mean of all the fundamental frequency values ​​can be used as the fundamental frequency mean of the monitoring audio segment, and the fundamental frequency mean can be used to distinguish the monitoring audio The high-pitched part of the frequency band; then, the zero-crossing rate variance of each frame of audio in the monitoring audio segment can be calculated, so that the mean of all zero-crossing rate variances can be used as the unvoiced sound fluctuation feature of the monitoring audio segment, and the unvoiced sound fluctuation feature is used to represent the fluctuation degree of the unvoiced sound part in the monitoring audio segment; finally, initialize a 14-dimensional empty vector, and fill the scale feature, the fundamental pitch mean and the unvoiced sound fluctuation feature into the empty vector respectively, that is, construct, so as to obtain the fundamental pitch feature vector of the monitoring area. In this application, the dimension of the scale feature is 12 dimensions, the dimension of the fundamental pitch mean is 1 dimension, and the dimension of the unvoiced sound fluctuation feature is 1 dimension.

[0041] In addition, it should be noted that the monitoring audio segments obtained through segmentation provide sound information of the monitoring area, which is convenient for comprehensive analysis of the monitoring scene. Some abnormal behaviors may produce signs through sound, such as sudden screams, breaking sounds, etc. These abnormal situations can be discovered more accurately by analyzing the audio data. The fundamental frequency feature vector extracted based on the audio data stream can provide additional feature dimensions to help distinguish different sound patterns and speech features.

[0042] In step 105, an early warning is issued for abnormal conditions in the monitoring area based on the monitoring decision value and the pitch feature vector.

[0043] In some embodiments, the following method can be used to warn of abnormal conditions in the monitoring area based on the monitoring decision value and the pitch feature vector, namely: When the monitoring decision value does not exceed the set threshold, it is determined that there is no abnormality in the monitoring area of ​​the artificial intelligence camera; When the monitoring decision value exceeds a set threshold, the monitoring video stream and the pitch feature vector are input into an anomaly detection model to obtain an anomaly detection result of the monitoring area; An early warning is issued for abnormal conditions in the monitored area based on the abnormal detection results.

[0044] In specific implementation, first, a threshold can be set based on expert knowledge and historical data; then, the monitoring decision value is compared with the set threshold. When the monitoring decision value does not exceed the set threshold, it indicates that there is no moving target in the area monitored by the artificial intelligence camera, and it is determined that there is no abnormality in the monitoring area and no warning is triggered; when the monitoring decision value exceeds the set threshold, it indicates that there is a moving target in the area monitored by the artificial intelligence camera, and there may be abnormalities in the monitoring area. At this time, the monitoring video stream and the fundamental frequency feature vector are input into the anomaly detection model for analysis to obtain the abnormal detection result of the monitoring area. The anomaly detection model in this application is a model built based on machine learning, which is used to identify abnormal situations in the monitoring area. In actual implementation, the anomaly detection model can also be constructed through deep learning and other technologies. If the anomaly detection result is that there is an abnormal situation, the warning mechanism is triggered, an alarm notification is sent to relevant personnel, the abnormal situation is recorded, and corresponding measures are taken to deal with the abnormal situation.

[0045] It can be seen that in this application, by combining visual information (monitoring decision value) and auditory information (pitch feature vector) to warn of abnormal situations in the monitoring area, the detection accuracy of abnormal behavior by the intelligent monitoring system can be improved to realize the intelligent warning of the artificial intelligence camera; wherein, firstly, the color-calibrated monitoring video stream is more consistent in color, so that a more accurate similarity between adjacent frames can be obtained, so that the moving target object in the monitoring area can be better identified based on the similarity between adjacent frames; secondly, the monitoring decision value determined based on the decision differential video frame is used to identify abnormal behavior, thereby realizing real-time monitoring and warning of abnormal behavior in the monitoring area; then, the pitch feature vector extracted based on the audio data stream can provide an additional feature dimension to help distinguish different sound patterns and voice features; finally, warning of abnormal situations in the monitoring area based on the monitoring decision value and the pitch feature vector can make up for the limitations and misjudgments that may exist in a single information source, thereby enhancing the accuracy and reliability of abnormal detection, and when an abnormal situation is detected, the warning mechanism can be triggered immediately; in summary, the technical solution provided by the present application can suppress environmental interference in the monitoring area when performing abnormal detection, so as to realize the intelligent warning of the artificial intelligence camera.

[0046] In addition, in another aspect of the present application, in some embodiments, the present application provides an intelligent monitoring system of an artificial intelligence camera, the system including an early warning unit, referring to Figure 4 , which is a schematic diagram of exemplary hardware and / or software of an early warning unit according to some embodiments of the present application, the early warning unit 400 includes: an acquisition module 401, a processing module 402 and an execution module 403, which are described as follows: Acquisition module 401, in this application, acquisition module 401 is mainly used to acquire the monitoring video stream shot by the artificial intelligence camera in the monitoring area; Processing module 402, in the present application, the processing module 402 is used to perform color calibration on the surveillance video stream to obtain a calibrated surveillance video stream, select surveillance video key frames from the calibrated surveillance video stream, and then determine the similarity between adjacent frames corresponding to the surveillance video key frames; It should be noted that the processing module 402 in the present application is also used to perform decision differentiation on the key frames of the monitoring video based on the similarity between the adjacent frames to obtain a decision differential video frame of the monitoring area, and determine the monitoring decision value of the artificial intelligence camera in the monitoring area through the decision differential video frame; In addition, the processing module 402 in the present application is also used to obtain the audio data stream recorded by the artificial intelligence camera in the monitoring area, perform threshold segmentation on the audio data stream based on the short-time energy and zero-crossing rate of the audio data stream to obtain a monitoring audio segment, and then perform feature construction on the monitoring audio segment to obtain a fundamental tone feature vector of the monitoring area; Execution module 403, in the present application, the execution module 403 mainly warns of abnormal conditions in the monitoring area based on the monitoring decision value and the fundamental pitch feature vector.

[0047] The above describes in detail the examples of the intelligent monitoring method and monitoring system of the artificial intelligence camera provided in the embodiments of the present application. It can be understood that in order to realize the above functions, the corresponding device includes hardware structures and / or software modules corresponding to the execution of each function. It should be easily appreciated by those skilled in the art that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0048] In some embodiments, the present application also provides a computer device, comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the computer device executes the above-mentioned intelligent monitoring method of the artificial intelligence camera.

[0049] In some embodiments, reference Figure 5, the dotted line in the figure indicates that the unit or the module is optional, and the figure is a structural schematic diagram of a computer device according to an intelligent monitoring method of an artificial intelligence camera provided in an embodiment of the present application. The intelligent monitoring method of the artificial intelligence camera in the above embodiment can be Figure 5 The computer device 500 shown in the figure is implemented, and the computer device 500 includes at least one processor 501, a memory 502 and at least one communication unit 505. The computer device 500 can be a terminal device, a server or a chip.

[0050] The processor 501 may be a general-purpose processor or a special-purpose processor. For example, the processor 501 may be a central processing unit (CPU), which may be used to control the computer device 500, execute software programs, and process data of the software programs. The computer device 500 may also include a communication unit 505 to implement signal input (reception) and output (transmission).

[0051] For example, the computer device 500 may be a chip, the communication unit 505 may be an input and / or output circuit of the chip, or the communication unit 505 may be a communication interface of the chip, and the chip may be a component of a terminal device, a network device, or other devices.

[0052] For another example, the computer device 500 may be a terminal device or a server, and the communication unit 505 may be a transceiver of the terminal device or the server, or the communication unit 505 may be a transceiver circuit of the terminal device or the server.

[0053] The computer device 500 may include one or more memories 502, on which a program 504 is stored. The program 504 can be executed by the processor 501 to generate instructions 503, so that the processor 501 performs the method described in the above method embodiment according to the instructions 503. Optionally, data (such as a target audit model) can also be stored in the memory 502. Optionally, the processor 501 can also read the data stored in the memory 502, and the data can be stored at the same storage address as the program 504, or the data can be stored at a different storage address from the program 504.

[0054] The processor 501 and the memory 502 may be provided separately or integrated together, for example, integrated on a system on chip (SOC) of the terminal device.

[0055] It should be understood that each step of the above method embodiment can be completed by a hardware-based logic circuit or software-based instructions in the processor 501. The processor 501 can be a central processing unit, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, such as discrete gates, transistor logic devices or discrete hardware components.

[0056] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0057] For example, in some embodiments, the present application also provides a computer-readable storage medium, in which instructions or codes are stored. When the instructions or codes are run on a computer, the computer implements the above-mentioned intelligent monitoring method of the artificial intelligence camera when executing.

[0058] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0059] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An intelligent monitoring method of an artificial intelligence camera, characterized in that: The steps include: Obtain the surveillance video stream shot by the AI ​​camera in the surveillance area; Performing color calibration on the surveillance video stream to obtain a calibrated surveillance video stream, selecting surveillance video key frames from the calibrated surveillance video stream, and then determining the similarity between adjacent frames corresponding to the surveillance video key frames; Perform decision differentiation on the key frames of the monitoring video based on the similarity between adjacent frames to obtain a decision differential video frame of the monitoring area, and determine the monitoring decision value of the artificial intelligence camera in the monitoring area through the decision differential video frame; Acquire the audio data stream recorded by the artificial intelligence camera in the monitoring area, perform threshold segmentation on the audio data stream based on the short-time energy and zero-crossing rate of the audio data stream to obtain a monitoring audio segment, and then perform feature construction on the monitoring audio segment to obtain a fundamental pitch feature vector of the monitoring area; An early warning is issued for abnormal conditions in the monitoring area based on the monitoring decision value and the fundamental pitch feature vector.

2. The method according to claim 1, characterized in that Performing color calibration on the surveillance video stream to obtain the calibrated surveillance video stream specifically includes: Performing color balancing on the surveillance video stream to obtain a balanced surveillance video stream; Performing color correction on the balanced surveillance video stream to obtain a calibrated surveillance video stream.

3. The method according to claim 1, characterized in that Determining the similarity between adjacent frames corresponding to the key frames of the surveillance video specifically includes: Obtaining adjacent surveillance video frames corresponding to the surveillance video key frames in the adjusted surveillance video stream; Determine the color distribution feature vectors of the surveillance video key frame and the adjacent surveillance video frame respectively; The similarity between adjacent frames corresponding to the key frame of the surveillance video is determined based on the color distribution feature vector of the key frame of the surveillance video and the color distribution feature vector of the adjacent surveillance video frame.

4. The method according to claim 1, characterized in that The decision difference of the key frame of the monitoring video is performed based on the similarity between the adjacent frames to obtain the decision difference video frame of the monitoring area, which specifically includes: When the similarity between the adjacent frames is greater than a preset decision value, the surveillance video key frame is differentiated from the first adjacent surveillance video frame and the second adjacent surveillance video frame in the corresponding adjacent surveillance video frames to obtain a first differential video frame and a second differential video frame; Binarizing the first differential video frame and the second differential video frame respectively to obtain a binarized first differential video frame and a binarized second differential video frame; A decision differential video frame of the monitoring area is determined according to the binarized first differential video frame and the binarized second differential video frame.

5. The method according to claim 1, characterized in that Performing threshold segmentation on the audio data stream based on the short-time energy and zero-crossing rate of the audio data stream is to perform threshold segmentation on the audio data stream by taking the short-time energy and the zero-crossing rate as the short-time energy threshold and the zero-crossing rate threshold respectively.

6. The method according to claim 1, characterized in that Constructing features for the monitored audio segment to obtain a pitch feature vector of the monitored area specifically includes: Determining the scale characteristics of the monitoring audio segment; Extracting the fundamental frequency value of each frame of audio in the monitoring audio segment, thereby obtaining all fundamental frequency values, and determining the fundamental frequency mean of the monitoring audio segment according to all fundamental frequency values; Determining the unvoiced sound fluctuation characteristics of the monitoring audio segment; A pitch feature vector of the monitoring area is constructed by using the scale feature, the pitch mean and the unvoiced sound fluctuation feature.

7. The method according to claim 1, characterized in that Providing an early warning of an abnormal situation in a monitoring area according to the monitoring decision value and the pitch feature vector specifically includes: When the monitoring decision value does not exceed the set threshold, it is determined that there is no abnormality in the monitoring area of ​​the artificial intelligence camera; When the monitoring decision value exceeds a set threshold, the monitoring video stream and the pitch feature vector are input into an anomaly detection model to obtain an anomaly detection result of the monitoring area; An early warning is issued for abnormal conditions in the monitored area based on the abnormal detection results.

8. An intelligent monitoring system of an artificial intelligence camera, the intelligent monitoring system of the artificial intelligence camera includes an early warning unit, characterized in that: The early warning unit comprises: An acquisition module is used to acquire the surveillance video stream shot by the artificial intelligence camera in the surveillance area; A processing module, used for color calibration of the monitoring video stream to obtain a calibrated monitoring video stream, selecting a monitoring video key frame from the calibrated monitoring video stream, and then determining the similarity between adjacent frames corresponding to the monitoring video key frame; The processing module is further used to perform decision differentiation on the key frames of the monitoring video based on the similarity between adjacent frames to obtain a decision differential video frame of the monitoring area, and determine the monitoring decision value of the artificial intelligence camera in the monitoring area through the decision differential video frame; The processing module is further used to obtain the audio data stream recorded by the artificial intelligence camera in the monitoring area, perform threshold segmentation on the audio data stream based on the short-time energy and zero-crossing rate of the audio data stream to obtain a monitoring audio segment, and then perform feature construction on the monitoring audio segment to obtain a fundamental tone feature vector of the monitoring area; An execution module is used to issue an early warning of abnormal conditions in a monitoring area according to the monitoring decision value and the fundamental pitch feature vector.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the computer device executes the intelligent monitoring method of the artificial intelligence camera described in any one of claims 1 to 7.

10. A computer-readable storage medium, wherein instructions or codes are stored in the computer-readable storage medium. When the instructions or codes are executed on a computer, the computer implements the intelligent monitoring method of the artificial intelligence camera as described in any one of claims 1 to 7.

Images