Abnormal target detection method, device, electronic device, storage medium and system

Abstract

The present invention provides an abnormal target detection method, device, electronic device, storage medium and system, wherein the method comprises: performing abnormal identification based on audio data and / or thermal imaging data of a scene to be detected to determine the position of a suspicious target; determining an abnormal target detection result based on the position of the object to be detected in the image data of the scene to be detected, and the position of the suspicious target. The abnormal target detection method, device, electronic device, storage medium and system provided by the present invention perform abnormal identification based on audio data and / or thermal imaging data of the scene to be detected to determine the position of the suspicious target, and can preliminarily screen out the position of suspicious targets with abnormalities, and then combine the position of the object to be detected in the image data of the scene to be detected, and the position of the suspicious target, to distinguish whether the suspicious target is the object to be detected or an environmental interferer, thereby weakening the interference of environmental interferers on abnormal target detection and improving the accuracy of detection.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an abnormal target detection method, device, electronic device, storage medium and system. Background Art

[0002] With the development of smart devices, the technology of using microphone arrays, infrared detectors and other devices to monitor whether there are abnormal targets in a scene has become popular. Among them, the use of microphone array technology can determine whether there is an abnormal sound and locate the sound source, and the use of infrared detectors can detect whether there are targets with abnormal temperatures in the scene.

[0003] However, using a single sensor data to detect abnormal targets in a scene is easily disturbed by the environment, resulting in poor detection accuracy of abnormal targets. For example, when there are high-temperature interference objects or noise sources in the environment, the interference objects or noise sources may be mistaken for abnormal targets. Summary of the invention

[0004] The present invention provides an abnormal target detection method, device, electronic device, storage medium and system, which are used to solve the defect of poor abnormal target detection accuracy in the prior art.

[0005] The present invention provides an abnormal target detection method, comprising:

[0006] Perform anomaly recognition based on audio data and / or thermal imaging data of the scene to be detected to determine the location of the suspicious target;

[0007] Based on the position of the object to be detected in the image data of the scene to be detected and the position of the suspicious target, an abnormal target detection result is determined.

[0008] According to an abnormal target detection method provided by the present invention, the abnormal target detection result is determined based on the position of the object to be detected in the image data of the scene to be detected and the position of the suspicious target, including:

[0009] Performing target detection on the image data to obtain the position of the object to be detected;

[0010] Converting the position of the suspicious object into the image coordinate system of the image data;

[0011] If it is determined that the converted position of the suspicious target coincides with the position of any object to be detected, the suspicious target is determined to be an abnormal target.

[0012] According to an abnormal target detection method provided by the present invention, the step of determining whether the converted position of the suspicious target coincides with the position of any object to be detected comprises:

[0013] Based on the position of the object to be detected, determining an enclosing area of ​​the object to be detected;

[0014] It is determined that the converted position of the suspicious target is within the enclosed area of ​​any object to be detected, and then it is determined that the converted position of the suspicious target coincides with the position of any object to be detected.

[0015] According to an abnormal target detection method provided by the present invention, the abnormal target detection result is determined based on the position of the object to be detected in the image data of the scene to be detected and the position of the suspicious target, including:

[0016] If it is determined that the position of the suspicious target coincides with the position of any object to be detected, multiple abnormality identifications are performed on the suspicious target to obtain the abnormal target detection result.

[0017] According to an abnormal target detection method provided by the present invention, the abnormal identification of the suspicious target is performed multiple times to obtain the abnormal target detection result, including:

[0018] Performing image tracking on the suspicious target to obtain a tracking position of the suspicious target; wherein the suspicious target is in a moving state;

[0019] Based on the audio data and / or thermal imaging data at the tracking position of the suspicious target, multiple abnormality identifications are performed on the suspicious target to obtain the abnormal target detection result.

[0020] According to an abnormal target detection method provided by the present invention, an abnormality recognition is performed based on audio data of a scene to be detected, comprising:

[0021] Based on a preset sound discrimination rule or an abnormal sound recognition model, performing abnormal sound recognition on the audio data;

[0022] The abnormal sound recognition model is trained based on sample audio data and sample abnormal sound recognition results of the sample audio data.

[0023] According to an abnormal target detection method provided by the present invention, an abnormality recognition is performed based on audio data of a scene to be detected, comprising:

[0024] Based on the sound frequency range of the object to be detected, filtering the audio data to obtain target audio data;

[0025] Or, based on the sound extraction model, extract the target sound belonging to the object to be detected from the audio data to obtain the target audio data;

[0026] Performing anomaly identification based on the target audio data;

[0027] Among them, the sound extraction model is trained based on sample mixed audio data and sample mixing ratio data of the sample mixed audio data; the sample mixing ratio data includes the proportion of the target sound frequency corresponding to each frequency point in the sample mixed audio data.

[0028] The present invention also provides an abnormal target detection device, comprising:

[0029] An anomaly identification unit, used to identify anomalies based on audio data and / or thermal imaging data of the scene to be detected and determine the location of the suspicious target;

[0030] The detection result determination unit is used to determine the abnormal target detection result based on the position of the object to be detected in the image data of the scene to be detected and the position of the suspicious target.

[0031] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above-mentioned abnormal target detection methods when executing the computer program.

[0032] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above-mentioned abnormal target detection methods are implemented.

[0033] The present invention also provides an abnormal target detection system, characterized in that it includes a processor, an image acquisition device, and an audio acquisition device and / or a thermal image acquisition device;

[0034] The image acquisition device is used to acquire image data of the scene to be detected and transmit the image data to the processor;

[0035] The audio acquisition device is used to collect audio data of the scene to be detected and transmit the audio data to the processor;

[0036] The thermal image acquisition device is used to acquire thermal image data of the scene to be detected and transmit the thermal image data to the processor;

[0037] The processor is used to execute the steps of any of the above-mentioned abnormal target detection methods.

[0038] The abnormal target detection method, device, electronic device, storage medium and system provided by the present invention perform abnormality recognition based on the audio data and / or thermal imaging data of the scene to be detected, determine the position of the suspicious target, and can preliminarily screen out the position of the suspicious target with abnormalities. Combined with the position of the object to be detected in the image data of the scene to be detected and the position of the suspicious target, it can be distinguished whether the suspicious target is the object to be detected or an environmental interferer, thereby weakening the interference of environmental interferers on abnormal target detection and improving the accuracy of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 One of the flow charts of the abnormal target detection method provided by the present invention;

[0041] Figure 2 A schematic diagram of a flow chart of a method for confirming a test result provided by the present invention;

[0042] Figure 3 The second flowchart of the abnormal target detection method provided by the present invention;

[0043] Figure 4 A schematic diagram of the structure of the abnormal target detection device provided by the present invention;

[0044] Figure 5 A schematic diagram of the structure of the abnormal target detection system provided by the present invention;

[0045] Figure 6 This is a schematic structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] With the development of smart devices, the technology of using microphone arrays, infrared detectors and other devices to monitor whether there are abnormal targets in a scene has become popular. Among them, the use of microphone array technology can determine whether there is an abnormal sound and locate the sound source, and the use of infrared detectors can receive the infrared radiation energy of the target to be measured and reflect it on the photosensitive element of the infrared detector, thereby obtaining an infrared thermal image, and detecting whether there is a target with abnormal temperature in the scene.

[0048] However, using a single sensor data to detect abnormal targets in a scene is easily disturbed by the environment, resulting in poor detection accuracy of abnormal targets. For example, when there is an abnormal temperature interference object in the environment, the interference object may be mistaken for an abnormal target; when there is a noise source in the surrounding environment, the noise source may also be mistaken for an abnormal target in the current scene.

[0049] In this regard, an embodiment of the present invention provides a method for detecting abnormal targets. Figure 1 One of the flow charts of the abnormal target detection method provided by the embodiment of the present invention is as follows: Figure 1 As shown, the method includes:

[0050] Step 110: perform anomaly recognition based on the audio data and / or thermal imaging data of the scene to be detected to determine the location of the suspicious target.

[0051] Here, an audio acquisition device can be used to collect audio data in the scene to be detected, and the audio data can be used to identify anomalies and detect whether there are abnormal sounds in the scene to be detected. Among them, the audio acquisition device may include a microphone or a microphone array, and the scene to be detected is a space where abnormal targets need to be detected. For example, when performing industrial gas leakage detection, the presence of gas leakage sound can be identified based on the audio data in the factory; or when detecting people with respiratory tract infections, the presence of coughing sound can be identified based on the audio data in scenes such as closed spaces or entrances and exits. If so, the sound source location of the abnormal sound can be located using sound source localization technology to obtain the position of the suspicious target. Among them, the position of the suspicious target can be a coordinate in the world coordinate system of the three-dimensional space, that is, an absolute coordinate, or a coordinate in a coordinate system with the position of the audio acquisition device as the origin, that is, a relative coordinate with the audio acquisition device as a reference object, which is not specifically limited in the embodiment of the present invention. In this case, the suspicious target is a target with abnormal sound.

[0052] In addition, thermal imaging data in the scene to be detected can also be collected by using thermal imaging sensors such as infrared detectors, and the thermal imaging data can be used for anomaly identification to detect whether there is a temperature anomaly in the scene to be detected. For example, when detecting industrial gas leaks, gas leaks are usually accompanied by changes in pipeline temperature, and when detecting people with respiratory infections, respiratory infections may be accompanied by an increase in human body temperature. Therefore, it is possible to identify whether there is a temperature anomaly based on the thermal imaging data in the scene to be detected. If so, the position of the temperature anomaly point can be located to obtain the position of the suspicious target. Among them, the position of the suspicious target can be a coordinate in the world coordinate system of three-dimensional space, that is, an absolute coordinate, or a coordinate in a coordinate system with the location of the thermal imaging sensor device as the origin, that is, a relative coordinate with the thermal imaging sensor device as a reference. In this case, the suspicious target is a target with a temperature anomaly.

[0053] In actual application scenarios, abnormal targets may have both abnormal sound and abnormal temperature. For example, when a gas leaks, it not only makes a sound but also causes the temperature of the pipe to rise. When a person has an upper respiratory tract infection, he may have a fever and cough at the same time. Therefore, it is also possible to combine the audio data and thermal imaging data in the scene to be detected to simultaneously detect whether there are abnormal sounds and abnormal temperatures in the scene, and locate the source of the abnormal sound and the location of the abnormal temperature point. Audio data and thermal imaging data are used to complement each other to improve the accuracy of abnormality identification. At this time, the suspicious target can be a target with abnormal sound or a target with abnormal temperature.

[0054] Step 120, based on the position of the object to be detected in the image data of the scene to be detected and the position of the suspicious target, determine the abnormal target detection result.

[0055] Specifically, there are a lot of environmental interferences in the scene to be detected. For example, when performing industrial gas leakage detection, the noise generated by the operation of other equipment in the factory and the environmental noise outside the factory will be mixed into the audio data of the scene to be detected. When detecting people with respiratory tract infections, the scene is also full of various environmental noises. Therefore, the abnormal sound obtained by abnormal identification of audio data may be caused by environmental interference. In order to avoid misdetecting environmental interference as an abnormal target, the suspicious target can be reconfirmed based on the position of the object to be detected in the image data of the scene to be detected, and the position of the suspicious target, so as to obtain the abnormal target detection result. Among them, the object to be detected is a target in the scene to be detected that needs to be judged whether it is abnormal, which can be determined according to the actual application scenario. For example, in the scene of industrial gas leakage detection, the object to be detected can be a gas pipeline; in the scene of respiratory tract infection detection, the object to be detected can be a person. The abnormal target detection result may include abnormal targets and their locations in the scene to be detected.

[0056] In order to locate the object to be detected in the scene to be detected, the scene to be detected can be photographed, and the object to be detected in the scene to be detected and its location can be identified using image processing technology based on the image data obtained by the shooting. The image data here can be a photographed image, or it can be an image frame extracted from a photographed video, and the embodiment of the present invention does not specifically limit this. Based on the position of each object to be detected in the scene to be detected, and the position of the suspicious target identified in the previous step, it can be confirmed whether the suspicious target is the object to be detected in the scene to be detected, or an environmental disturbance, and the abnormal target detection result is determined accordingly. Here, if the suspicious target is not the object to be detected, it can be considered that there is no abnormal target here, and the suspicious target is an environmental disturbance. By distinguishing whether the suspicious target is an object to be detected or an environmental disturbance through the position of the object to be detected in the image data, the interference of environmental disturbances on abnormal target detection can be weakened, and the accuracy of the detection method can be improved.

[0057] The method provided by the embodiment of the present invention performs anomaly recognition based on the audio data and / or thermal imaging data of the scene to be detected, determines the position of the suspicious target, and can preliminarily screen out the position of the suspicious target with abnormalities. Combined with the position of the object to be detected in the image data of the scene to be detected and the position of the suspicious target, it can be distinguished whether the suspicious target is the object to be detected or an environmental interferer, thereby weakening the interference of environmental interferers on the detection of abnormal targets and improving the accuracy of the detection method.

[0058] Based on the above embodiments, Figure 2 A flow chart of a method for confirming a test result provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, step 120 includes:

[0059] Step 121, performing target detection on the image data to obtain the position of the object to be detected;

[0060] Step 122, converting the position of the suspicious target into the image coordinate system of the image data;

[0061] Step 123: If it is determined that the converted position of the suspicious target coincides with the position of any object to be detected, the suspicious target is determined to be an abnormal target.

[0062] Here, target detection technology can be used to perform target detection on the collected image data to obtain the position of the object to be detected in the image data. Among them, a trained target detection model can be used to perform target detection on the image data, and the target detection model can be trained based on sample image data and sample target detection results of the sample image data. Here, the sample target detection result is the position of each marked object to be detected in the sample image data. The position of the object to be detected can be the position of the bounding box of the object to be detected in the corresponding image data, for example, it can include the coordinates of the center point of the bounding box and the length and width of the bounding box, and can also include the coordinates of two points on the diagonal of the bounding box, and the embodiment of the present invention does not specifically limit this. Among them, the bounding box of the object to be detected can surround the key area of ​​the object to be detected, such as a face, a torso, or a pipe.

[0063] Since the position of the suspicious target obtained by sound source positioning or temperature positioning is the position information in three-dimensional space, it is not in the same coordinate system as the position of the object to be detected, and it is difficult to directly compare. Therefore, the position of the suspicious target can be converted to the image coordinate system of the image data. Among them, the image coordinate system is a coordinate system established based on the image taken by the image acquisition device, which can indicate the position of the object in the real world in the image, and the coordinate system is related to the position of the image acquisition device itself. Here, if the position of the suspicious target is the coordinate in the world coordinate system, the camera can be calibrated in advance to obtain the conversion relationship between the camera coordinate system and the three-dimensional coordinate system, so as to use the conversion relationship to convert the position of the suspicious target to the camera coordinate system, and then according to the conversion relationship between the camera coordinate system and the image coordinate system, the position of the suspicious target is converted to the image coordinate system. If the position of the suspicious target is a relative coordinate with the audio acquisition device or thermal image acquisition device as a reference, the relative position relationship between the audio acquisition device or thermal image acquisition device and the image acquisition device can also be determined in advance, and then the position of the suspicious target is converted to the image coordinate system using the relative position relationship.

[0064] After converting the position of the suspicious target and the position of the object to be detected to the same coordinate system, it can be determined whether the position of the suspicious target after conversion coincides with the position of a certain object to be detected. If the position of the suspicious target after conversion coincides with the position of any object to be detected, it indicates that the suspicious position is the object to be detected rather than an environmental interference, so the suspicious target can be determined to be an abnormal target. If the position of the suspicious target after conversion does not coincide with the positions of all objects to be detected, it indicates that the suspicious position is an environmental interference, so the suspicious target can be excluded.

[0065] Based on any of the above embodiments, determining in step 123 that the converted position of the suspicious target coincides with the position of any object to be detected includes:

[0066] Based on the position of the object to be detected, determining an enclosing area of ​​the object to be detected;

[0067] It is determined that the converted position of the suspicious target is within the enclosing area of ​​any object to be detected, and it is determined that the converted position of the suspicious target coincides with the position of the object to be detected.

[0068] Here, the position of the enclosing frame of each object to be detected in the image data, that is, the position of the object to be detected, can be detected by using target detection technology. According to the position of the object to be detected, the enclosing area of ​​each object to be detected can be determined. Among them, the enclosing area of ​​the object to be detected includes the enclosing frame of the object to be detected, and the area range can be larger than the enclosing frame. The size of the enclosing area can be specifically set according to the size of the enclosing frame of the object to be detected. For example, if the enclosing frame of the object to be detected is small, considering the position error that may occur during sound source positioning and temperature positioning, it can be expanded outward by a certain proportion on the basis of the enclosing frame of the object to be detected as the enclosing area of ​​the object to be detected; if the enclosing frame of the object to be detected is large, the enclosing frame can also be directly used as the enclosing area of ​​the object to be detected.

[0069] Then, it is determined whether the converted position of the suspicious target is within the encircling area of ​​a certain object to be detected. If the converted position of the suspicious target is within the encircling area of ​​any object to be detected, it is determined that the converted position of the suspicious target coincides with the position of the object to be detected.

[0070] Based on any of the above embodiments, step 120 includes:

[0071] If it is determined that the position of a suspicious target coincides with the position of any object to be detected, multiple abnormality recognitions are performed on the suspicious target to obtain abnormal target detection results.

[0072] Here, since the objects to be detected are often densely packed, there may be interference between the objects to be detected. For example, the temperature of a gas leaking pipe increases, which may briefly affect the temperature of adjacent pipes, or when multiple people gather, the sound of one person coughing may cause multiple people to be considered abnormal targets. In addition, occasional abnormal conditions may also occur during the audio data acquisition and thermal imaging data acquisition process, resulting in errors in the abnormal identification process. Therefore, in order to further improve the accuracy of abnormal target detection, suspicious targets can be more fully detected to confirm whether the suspicious targets are abnormal targets.

[0073] Specifically, after determining that the position of the suspicious target coincides with the position of any object to be detected, the suspicious target can be subjected to multiple abnormal identifications, and the abnormal target detection result can be determined based on the results of the multiple abnormal identifications. Here, when multiple abnormal identifications are performed, the audio data and / or thermal imaging data of the suspicious object can be collected multiple times in a directional manner, and abnormal identification can be performed based on the audio data and / or thermal imaging data. Among them, if the results of multiple abnormal identifications all indicate that the suspicious target is abnormal, the suspicious target can be confirmed as an abnormal target and an alarm can be issued. There can be many forms of alarms, including issuing an alarm sound, vibrating reminders, or sending alarm information to a control terminal, etc. The specific alarm form can be selected according to the needs of the actual application scenario. Here, an alarm threshold can be set, for example, an alarm will be issued only when the results of more than n abnormal identifications indicate that the suspicious target is abnormal, and the alarm threshold can be set according to the actual application scenario, for example, the alarm threshold can be set lower in a high-sensitivity scenario. Here, n is the alarm threshold, which is a positive integer.

[0074] The method provided by the embodiment of the present invention, when it is determined that the position of a suspicious target coincides with the position of any object to be detected, performs multiple abnormal identifications on the suspicious target to obtain abnormal target detection results, performs more comprehensive detection on the suspicious target, avoids false detections, and further improves the accuracy of abnormal target identification.

[0075] Based on any of the above embodiments, multiple abnormal identifications are performed on suspicious targets to obtain abnormal target detection results, including:

[0076] Performing image tracking on a suspicious target to obtain a tracking position of the suspicious target; wherein the suspicious target is in a moving state;

[0077] Based on the audio data and / or thermal image data at the tracking position of the suspicious target, multiple abnormality identifications are performed on the suspicious target to obtain abnormal target detection results.

[0078] Here, when the suspicious target is in motion, such as a person in motion, in order to detect the suspicious target in a directional manner, image tracking can be performed on it. Specifically, the video data of the scene to be detected can be obtained, and after converting it into an image sequence, each image in the image sequence is subjected to target recognition to obtain the tracking position of the suspicious target. Whenever the suspicious target is tracked and its tracking position is obtained, the audio data and / or thermal image data at the tracking position can be collected in a directional manner, so as to perform abnormal recognition based on the audio data and / or thermal image data to obtain the abnormal target detection result. Here, after obtaining the tracking position of the suspicious target, the audio acquisition device can be controlled to turn to the suspicious target and perform directional sound pickup on it, thereby improving the sound pickup effect. The thermal image data acquisition device can also be controlled to turn to the suspicious target and perform directional temperature measurement on it, thereby improving the temperature measurement effect. It should be noted that if the suspicious target is a target with abnormal sound, when performing multiple abnormal recognitions, only directional sound pickup can be performed on it to confirm whether it really has abnormal sound, or both directional sound pickup and directional temperature measurement can be performed on it. By combining sound abnormality detection and temperature abnormality detection, it can be confirmed whether it is abnormal, and the accuracy of abnormal target detection can be further improved. Similarly, if the suspicious target is a target with abnormal temperature, when performing multiple abnormality identifications, it can be only subjected to directional temperature measurement to confirm whether it really has an abnormal temperature. It can also be subjected to directional temperature measurement and directional sound pickup, and sound anomaly detection and temperature anomaly detection can be performed simultaneously to confirm whether there is an abnormality.

[0079] When a suspicious target is in a stationary state, such as a stationary gas pipeline, its position will not change significantly. Therefore, the audio data and / or thermal imaging data at the position can be collected multiple times in a directional manner according to the position of the suspicious target previously located, so as to perform anomaly identification based on the audio data and / or thermal imaging data and obtain abnormal target detection results.

[0080] Based on any of the above embodiments, performing abnormality identification based on the audio data of the scene to be detected in step 110 includes:

[0081] Based on preset sound discrimination rules or abnormal sound recognition models, abnormal sound recognition is performed on audio data;

[0082] The abnormal sound recognition model is trained based on the sample audio data and the sample abnormal sound recognition results of the sample audio data.

[0083] Here, a preset sound discrimination rule can be set based on the sound generation mechanism of abnormal sound, and whether the audio data is abnormal can be determined by judging whether it meets the preset sound discrimination rule. For example, a sound amplitude threshold can be set, and if the sound amplitude of the audio data is greater than the sound amplitude threshold, the audio data is judged to be abnormal.

[0084] In addition, an abnormal sound recognition model can be trained based on machine learning technology, and the model can be used to extract audio features of audio data, and based on the audio features of the audio data, it can be judged whether it is abnormal to achieve abnormal sound recognition. Among them, a large amount of sample audio data can be collected, and the sample abnormal sound recognition results of the sample audio data can be annotated, and then the initial model can be trained based on the sample audio data and the sample abnormal sound recognition results to obtain the abnormal sound recognition model.

[0085] When performing sound anomaly recognition, it is possible to determine what kind of sound is an abnormal sound based on the actual application scenario. For example, in the scenario of detecting people with respiratory infections, the abnormal sound may be a cough; in the scenario of detecting gas leaks, the abnormal sound may be the sound of a gas leak; in the field of monitoring, the abnormal sound may be a conversation with unusual content, such as conversations that expose privacy, business secrets, etc., uncivilized language, etc. Therefore, when performing sound anomaly recognition, the features of the abnormal sound may be extracted through the definition of the abnormal sound in the current application scenario, and it is usually determined whether the audio data contains the features of the abnormal sound to achieve the detection of the abnormal sound. For example, the audio features of the audio data may be extracted through machine learning technology to determine whether the audio features are consistent with the audio features of the abnormal sound; the speech content in the audio data may also be identified through speech recognition technology to determine whether the speech content contains special words corresponding to the abnormal sound to achieve the detection of the abnormal sound.

[0086] Based on any of the above embodiments, performing abnormality identification based on the audio data of the scene to be detected in step 110 includes:

[0087] Based on the sound frequency range of the object to be detected, the audio data is filtered to obtain the target audio data;

[0088] Or, based on the sound extraction model, extract the target sound belonging to the object to be detected from the audio data to obtain the target audio data;

[0089] Perform anomaly recognition based on target audio data;

[0090] Among them, the sound extraction model is trained based on the sample mixed audio data and the sample mixed ratio data of the sample mixed audio data; the sample mixed ratio data includes the proportion of the target sound frequency corresponding to each frequency point in the sample mixed audio data.

[0091] Here, after obtaining the audio data, the sound of the object to be detected can be extracted according to its acoustic characteristics, and the environmental noise in the audio data can be removed, so as to separate the relatively pure sound of the object to be detected, that is, the target audio data. For example, in the scenario of detecting people with respiratory tract infections, the sound of people can be separated, and in the scenario of detecting gas leaks, the sound of pipes can be separated.

[0092] If the sound of the object to be detected can be distinguished from other sounds according to the frequency range, the audio data can be filtered based on the sound frequency range of the object to be detected to filter out sounds outside the sound frequency range of the object to be detected and obtain target audio data.

[0093] In addition, if the sound of the object to be detected is mixed with other audio bands, a sound extraction model can be trained based on machine learning technology, and the model can be used to extract the target sound belonging to the object to be detected in the audio data to obtain the target audio data. Among them, pure sample target audio data and background audio data can be collected, and the sample target audio data and background audio data can be mixed in a certain proportion to obtain sample mixed audio data, and the proportion of the target sound frequency corresponding to each frequency point in each audio frame during mixing is recorded as the sample mixing ratio data of the sample mixed audio data. The initial model is trained based on the sample mixed audio data and its sample mixing ratio data to obtain the sound extraction model.

[0094] When actually performing sound extraction, the audio data can be input into the sound extraction model for frequency analysis to obtain the mixing ratio data corresponding to the audio data. The mixing ratio data includes the proportion of the target sound frequency corresponding to each frequency point in each audio frame of the audio data. After the audio data is converted to the frequency domain, the value of each frequency point is multiplied by the proportion of the target sound frequency corresponding to the frequency point in the mixing ratio data to obtain the filtered frequency domain data, which is then converted back to the time domain to obtain the target audio data.

[0095] After obtaining relatively pure target audio data, anomaly identification can be performed on the target audio data to determine whether it contains abnormal sounds, thereby improving the accuracy of anomaly identification.

[0096] Based on any of the above embodiments, Figure 3 The second flow chart of the abnormal target detection method provided by the embodiment of the present invention is as follows: Figure 3 As shown, the method includes:

[0097] The camera, microphone and / or infrared sensor are used to synchronously collect image data, audio data and / or thermal image data (including temperature information).

[0098] Perform target detection on the image data to obtain the location of the object to be detected. The object to be detected can be determined according to different scenarios. For example, in the scenario of locating people with respiratory tract infections, the object to be detected is a person; in the scenario of gas leak detection, the object to be detected is a gas pipeline. At the same time, after obtaining the audio data, the sound of the object to be detected is extracted according to its acoustic characteristics to obtain relatively pure target audio data.

[0099] Here, if the sound of the object to be detected can be distinguished from other sounds based on the frequency range, it can be filtered based on the frequency range of the object to be detected to filter out the data outside the frequency range; if the sound of the object to be detected is mixed with other sound frequency bands, it can be extracted through the sound extraction model to separate the sound of the object to be detected and remove the influence of other factors such as the environment. For example, in the scenario of locating people with respiratory infections, human voices can be separated, and in the scenario of detecting gas leaks, the sound emitted by the pipeline can be separated.

[0100] After obtaining relatively pure target audio data belonging to the object to be detected, the target audio data is judged to be normal or abnormal. Here, the preset sound discrimination rules can be used for judgment, such as whether the sound amplitude of the target audio data is greater than a certain decibel value, or it can be judged by the abnormal sound recognition model. If the judgment result is that there is an abnormality, such as a cough or a sound of gas leakage, the sound is located to obtain the position information of the abnormal point of the sound (ie, the suspicious target). Among them, the microphone array technology can be used to calculate the horizontal position, vertical position and distance of the abnormal sound source relative to the microphone, which can be converted into position information in three-dimensional space.

[0101] Similarly, the thermal imaging data can be analyzed to locate the positions of abnormal temperature points in the thermal imaging data. Among them, whether the temperature is abnormal can be determined based on the temperature characteristics of the object to be detected. For example, in the scenario of locating people with respiratory infections, a temperature exceeding 37.5° is considered abnormal, or in the scenario of detecting gas leaks, a temperature exceeding the normal temperature of the gas pipeline is considered abnormal. After determining the abnormal temperature points in the thermal imaging data, they can be located to obtain the position information of the abnormal temperature points (i.e., suspicious targets). For example, the infrared detector can be aligned with the camera to achieve position mapping of each point in the thermal imaging data to the image data, and then the camera calibration parameters can be used to achieve mapping of each point in the thermal imaging data to the three-dimensional spatial position.

[0102] Then, it is determined whether the position of the object to be detected overlaps with the position of the abnormal sound point and / or the abnormal temperature point. Here, when determining whether there is overlap, it can be determined whether the area range of the object to be detected includes the abnormal sound point or the abnormal temperature point. When the object to be detected is small, an envelope range that can surround the object to be detected can also be set according to the position of the object to be detected, and it can be determined whether the envelope range includes the abnormal sound point or the abnormal temperature point.

[0103] If the position of the object to be detected does not coincide with the position of the abnormal sound point or the abnormal temperature point, it can be considered that there is no abnormal target. If the position of the object to be detected coincides with the position of the abnormal sound point or the abnormal temperature point, the suspicious target at the coincident position can be regarded as a suspected defect point. If the scene is to detect gas leaks, the suspected defect point may be a certain position in the pipeline. If the scene is to detect people with respiratory infections, the suspected defect point may be a certain person.

[0104] Use cameras and image tracking technology to track suspicious targets and obtain their location information. Then use microphones and / or infrared detectors to perform multiple directional sound pickups and / or directional temperature measurements at the location to confirm whether the suspicious target is abnormal.

[0105] According to the results of directional sound pickup and / or directional temperature measurement, the suspicious target is detected multiple times to see if there is any abnormality, and it is determined whether an abnormality alarm is needed. Each time an abnormality is detected, if it is determined that the suspicious target has neither temperature abnormality nor sound abnormality based on the audio data and / or thermal imaging data collected this time, then the current detection result is no abnormality. If there is a temperature abnormality and / or temperature abnormality, then the current detection result is that there is an abnormality. Here, an alarm threshold can be set, and an alarm is issued when the number of abnormalities detected exceeds the alarm threshold. The alarm threshold can be set according to actual application requirements. For example, the alarm threshold can be set lower in high-sensitivity scenarios.

[0106] The abnormal target detection device provided by the present invention is described below. The abnormal target detection device described below and the abnormal target detection method described above can be referenced to each other.

[0107] Based on any of the above embodiments, Figure 4 A schematic diagram of the structure of an abnormal target detection device provided by an embodiment of the present invention is shown in FIG. Figure 4 As shown, the device includes: an abnormality identification unit 410 and a detection result determination unit 420.

[0108] The abnormality identification unit 410 is used to identify abnormalities based on the audio data and / or thermal imaging data of the scene to be detected, and determine the location of the suspicious target;

[0109] The detection result determination unit 420 is used to determine the abnormal target detection result based on the position of the object to be detected in the image data of the scene to be detected and the position of the suspicious target.

[0110] The device provided by the embodiment of the present invention performs anomaly recognition based on the audio data and / or thermal imaging data of the scene to be detected, determines the position of the suspicious target, and can preliminarily screen out the position of the suspicious target with abnormalities. Then, combined with the position of the object to be detected in the image data of the scene to be detected and the position of the suspicious target, it can be distinguished whether the suspicious target is the object to be detected or an environmental interferer, thereby weakening the interference of environmental interferers on the detection of abnormal targets and improving the accuracy of detection.

[0111] Based on any of the above embodiments, the detection result determination unit 420 is used to:

[0112] Perform target detection on the image data to obtain the position of the object to be detected;

[0113] Convert the position of the suspicious target into the image coordinate system of the image data;

[0114] If the converted position of the suspicious target coincides with the position of any object to be detected, the suspicious target is determined to be an abnormal target.

[0115] Based on any of the above embodiments, determining that the converted position of the suspicious target coincides with the position of any object to be detected includes:

[0116] Based on the position of the object to be detected, determining an enclosing area of ​​the object to be detected;

[0117] It is determined that the converted position of the suspicious target is within the enclosing area of ​​any object to be detected, and it is determined that the converted position of the suspicious target coincides with the position of the object to be detected.

[0118] Based on any of the above embodiments, the detection result determination unit 420 is used to:

[0119] If it is determined that the position of a suspicious target coincides with the position of any object to be detected, multiple abnormality recognitions are performed on the suspicious target to obtain abnormal target detection results.

[0120] The device provided by the embodiment of the present invention, when determining that the position of a suspicious target coincides with the position of any object to be detected, performs multiple abnormal identifications on the suspicious target to obtain abnormal target detection results, performs more comprehensive detection on the suspicious target, avoids false detection, and further improves the accuracy of abnormal target identification.

[0121] Based on any of the above embodiments, multiple abnormal identifications are performed on suspicious targets to obtain abnormal target detection results, including:

[0122] Performing image tracking on a suspicious target to obtain a tracking position of the suspicious target; wherein the suspicious target is in a moving state;

[0123] Based on the audio data and / or thermal image data at the tracking position of the suspicious target, multiple abnormality identifications are performed on the suspicious target to obtain abnormal target detection results.

[0124] Based on any of the above embodiments, performing abnormality recognition based on audio data of the scene to be detected includes:

[0125] Based on preset sound discrimination rules or abnormal sound recognition models, abnormal sound recognition is performed on audio data;

[0126] The abnormal sound recognition model is trained based on the sample audio data and the sample abnormal sound recognition results of the sample audio data.

[0127] Based on any of the above embodiments, performing abnormality recognition based on audio data of the scene to be detected includes:

[0128] Based on the sound frequency range of the object to be detected, the audio data is filtered to obtain the target audio data;

[0129] Or, based on the sound extraction model, extract the target sound belonging to the object to be detected from the audio data to obtain the target audio data;

[0130] Perform anomaly recognition based on target audio data;

[0131] Among them, the sound extraction model is trained based on the sample mixed audio data and the sample mixed ratio data of the sample mixed audio data; the sample mixed ratio data includes the proportion of the target sound frequency corresponding to each frequency point in the sample mixed audio data.

[0132] Based on any of the above embodiments, Figure 5 A schematic diagram of the structure of an abnormal target detection system provided by an embodiment of the present invention is shown in FIG. Figure 5 As shown, the system includes: a processor 510, an image acquisition device 520, and an audio acquisition device 530 and / or a thermal image acquisition device 540;

[0133] The image acquisition device 520 is used to acquire image data of the scene to be detected and transmit the image data to the processor;

[0134] The audio collection device 530 is used to collect audio data of the scene to be detected and transmit the audio data to the processor;

[0135] The thermal image acquisition device 540 is used to acquire thermal image data of the scene to be detected and transmit the thermal image data to the processor;

[0136] The processor 510 is used to execute the steps of the abnormal target detection method provided in any of the above embodiments.

[0137] The image acquisition device 520 is used to acquire image data of the scene to be detected. For example, a camera may be used to directly take a photo of the scene to be detected, or a video of the scene to be detected may be taken and then the video may be converted into an image sequence.

[0138] The audio collection device 530 is used to collect audio data of the scene to be detected. For example, a single microphone or a microphone array can be used to collect audio data in the scene to be detected.

[0139] The thermal imaging acquisition device 540 is used to acquire thermal imaging data of the scene to be detected. For example, a thermal imaging sensor such as an infrared detector may be used to acquire thermal imaging data within the scene to be detected.

[0140] The image acquisition device 520, the audio acquisition device 530 and the thermal image acquisition device 540 transmit the acquired data to the processor, and the processor can perform the steps of the abnormal target detection method provided in any of the above embodiments based on the image data, audio data and / or thermal image data of the scene to be detected. Specifically, anomaly identification can be performed based on the audio data and / or thermal image data of the scene to be detected to determine the position of the suspicious target, and then the abnormal target detection result can be determined based on the position of the object to be detected in the image data of the scene to be detected and the position of the suspicious target.

[0141] Here, if the position of the suspicious target coincides with the position of any object to be detected, the suspicious target is identified as abnormal multiple times to obtain an abnormal target detection result. Among them, if the suspicious target is in motion, the image acquisition device 520 can be used to track it to obtain the tracking position of the suspicious target, and then the audio acquisition device 530 and / or the thermal imaging acquisition device 540 are used to perform directional sound pickup and / or directional temperature measurement on the tracking position, and multiple abnormal identifications are performed based on the audio data and / or thermal imaging data of the directional sound pickup and / or directional temperature measurement to obtain multiple abnormal identification results. Here, an alarm threshold can be set. If the number of abnormal identification results exceeds the alarm threshold, it can be determined that the abnormal target detection result is the suspicious target, and various alarm forms can also be used for alarm, such as sending a warning message to a control device.

[0142] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the abnormal target detection method, which includes: performing abnormal identification based on the audio data and / or thermal imaging data of the scene to be detected, and determining the position of the suspicious target; determining the abnormal target detection result based on the position of the object to be detected in the image data of the scene to be detected and the position of the suspicious target.

[0143] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0144] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the abnormal target detection method provided by the above-mentioned methods, and the method includes: performing anomaly identification based on audio data and / or thermal imaging data of the scene to be detected, and determining the position of the suspicious target; determining the abnormal target detection result based on the position of the object to be detected in the image data of the scene to be detected, and the position of the suspicious target.

[0145] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the above-mentioned abnormal target detection methods, the methods comprising: performing abnormality identification based on audio data and / or thermal imaging data of the scene to be detected, and determining the position of the suspicious target; determining the abnormal target detection result based on the position of the object to be detected in the image data of the scene to be detected, and the position of the suspicious target.

[0146] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0147] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting abnormal targets, characterized in that: include: Performing anomaly identification based on audio data and / or thermal imaging data of the scene to be detected to determine the location of a suspicious target, wherein the suspicious target is a target with abnormal sound and / or abnormal temperature; Determine an abnormal target detection result based on the position of the object to be detected in the image data of the scene to be detected and the position of the suspicious target, wherein the object to be detected is a target in the scene to be detected that needs to be judged whether it is abnormal; The determining of the abnormal target detection result based on the position of the object to be detected in the image data of the scene to be detected and the position of the suspicious target includes: If it is determined that the position of the suspicious target coincides with the position of any object to be detected, multiple abnormal identifications are performed on the suspicious target to obtain the abnormal target detection result, and an alarm is issued when the abnormal identification results exceeding a preset number indicate that the suspicious target is abnormal; The performing of multiple abnormal identifications on the suspicious target to obtain the abnormal target detection result includes: When the suspicious target is in a stationary state, audio data and / or thermal imaging data at the position of the suspicious target are collected multiple times in a directional manner, and abnormality identification is performed based on the audio data and / or thermal imaging data to obtain the abnormal target detection result; When the suspicious target is in motion, image tracking is performed on the suspicious target to obtain the tracking position of the suspicious target, and audio data and / or thermal imaging data at the tracking position are collected in a directional manner for multiple times. Anomaly identification is performed based on the audio data and / or thermal imaging data to obtain the abnormal target detection result.

2. The abnormal target detection method according to claim 1, characterized in that: The determining of the abnormal target detection result based on the position of the object to be detected in the image data of the scene to be detected and the position of the suspicious target includes: Performing target detection on the image data to obtain the position of the object to be detected; Converting the position of the suspicious object into the image coordinate system of the image data; If it is determined that the converted position of the suspicious target coincides with the position of any object to be detected, the suspicious target is determined to be an abnormal target.

3. The abnormal target detection method according to claim 2, characterized in that: The step of determining that the converted position of the suspicious target coincides with the position of any object to be detected includes: Based on the position of the object to be detected, determining an enclosing area of ​​the object to be detected; It is determined that the converted position of the suspicious target is within the enclosed area of ​​any object to be detected, and then it is determined that the converted position of the suspicious target coincides with the position of any object to be detected.

4. The abnormal target detection method according to any one of claims 1 to 3, characterized in that: Anomaly recognition based on the audio data of the scene to be detected, including: Based on a preset sound discrimination rule or an abnormal sound recognition model, performing abnormal sound recognition on the audio data; The abnormal sound recognition model is trained based on sample audio data and sample abnormal sound recognition results of the sample audio data.

5. The abnormal target detection method according to any one of claims 1 to 3, characterized in that: Anomaly recognition based on the audio data of the scene to be detected, including: Based on the sound frequency range of the object to be detected, filtering the audio data to obtain target audio data; Or, based on the sound extraction model, extract the target sound belonging to the object to be detected from the audio data to obtain the target audio data; Performing anomaly identification based on the target audio data; Among them, the sound extraction model is trained based on sample mixed audio data and sample mixing ratio data of the sample mixed audio data; the sample mixing ratio data includes the proportion of the target sound frequency corresponding to each frequency point in the sample mixed audio data.

6. An abnormal target detection device, characterized in that: include: An anomaly identification unit, used to identify anomalies based on audio data and / or thermal imaging data of the scene to be detected, and determine the location of a suspicious target, wherein the suspicious target is a target with abnormal sound and / or abnormal temperature; A detection result determination unit, configured to determine an abnormal target detection result based on a position of an object to be detected in the image data of the scene to be detected and a position of the suspicious target, wherein the object to be detected is a target in the scene to be detected that needs to be judged as abnormal; The detection result determination unit is specifically used for: If it is determined that the position of the suspicious target coincides with the position of any object to be detected, multiple abnormal identifications are performed on the suspicious target to obtain the abnormal target detection result, and an alarm is issued when the abnormal identification results exceeding a preset number indicate that the suspicious target is abnormal; The performing of multiple abnormal identifications on the suspicious target to obtain the abnormal target detection result includes: When the suspicious target is in a stationary state, audio data and / or thermal imaging data at the position of the suspicious target are collected multiple times in a directional manner, and abnormality identification is performed based on the audio data and / or thermal imaging data to obtain the abnormal target detection result; When the suspicious target is in motion, image tracking is performed on the suspicious target to obtain the tracking position of the suspicious target, and audio data and / or thermal imaging data at the tracking position are collected in a directional manner for multiple times. Anomaly identification is performed based on the audio data and / or thermal imaging data to obtain the abnormal target detection result.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the abnormal target detection method according to any one of claims 1 to 5 are implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the abnormal target detection method according to any one of claims 1 to 5 are implemented.

9. An abnormal target detection system, characterized in that: It includes a processor, an image acquisition device, and an audio acquisition device and / or a thermal image acquisition device; The image acquisition device is used to acquire image data of the scene to be detected and transmit the image data to the processor; The audio acquisition device is used to collect audio data of the scene to be detected and transmit the audio data to the processor; The thermal image acquisition device is used to acquire thermal image data of the scene to be detected and transmit the thermal image data to the processor; The processor is used to execute the steps of the abnormal target detection method according to any one of claims 1 to 5.

Images