Inspection and monitoring method and system
Through the automatic adjustment of audio acquisition and image acquisition device, the problem that the camera in the intelligent IoT monitoring system cannot respond to emergencies in real time is solved, and timely and accurate monitoring of on-site problems is achieved.
Patent Information
- Application Number
- CN202210952894.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-09
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2042-08-09
AI Technical Summary
In the existing intelligent IoT monitoring system, the installation position and angle of the camera need to be manually adjusted, and it is impossible to respond to on-site emergencies in real time.
The audio acquisition device collects environmental audio information, the main control device determines the target object, and controls the image acquisition device to adjust the shooting angle to correspond to the target object, realizing automatic adjustment and real-time monitoring.
Timely and accurate monitoring of emergencies is achieved, the shortcomings of manual camera adjustment are avoided, and the real-time response capability of the monitoring system is improved.
Smart Images

Figure CN115379169B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of Internet of Things, and in particular, to a patrol monitoring method and system. Background Art
[0002] The intelligent Internet of Things (AIoT) is a concept that emerged in 2018. It refers to the system that collects various types of information in real time through various information sensors (generally in monitoring, interaction, and connection scenarios), and uses machine learning to intelligently analyze data in terminal devices, edge domains, or cloud centers, including positioning, comparison, prediction, and scheduling. On the technical level, artificial intelligence enables the Internet of Things to acquire perception and recognition capabilities, and the Internet of Things provides data for training algorithms for artificial intelligence. On the commercial level, the two work together on the real economy, promoting industrial upgrading and experience optimization. In terms of specific types, there are three main categories: intelligent networked devices with perception / interaction capabilities, equipment asset management through machine learning, and systematic solutions with networked devices and AI capabilities.
[0003] Currently, smart IoT gateways are mainly deployed on-site and connected to cameras, and then various conditions on-site are monitored in real time through the captured images.
[0004] However, the monitoring method in the related art is limited by the location of the camera installed on site, and the camera needs to be manually controlled at the pan / tilt to adjust its shooting angle, and cannot respond to various emergencies on site in real time. Summary of the invention
[0005] In order to overcome the problems existing in the related art, the present disclosure provides a patrol monitoring method and system.
[0006] According to a first aspect of an embodiment of the present disclosure, a patrol monitoring method is provided, which is applied to a patrol monitoring system, wherein the patrol monitoring system includes an image acquisition device, an audio acquisition device, and a main control device, and the method includes:
[0007] Collecting audio information of the current environment through the audio collection device;
[0008] Determine the target object from the current environment according to the audio information by the main control device, and control the shooting angle of the image acquisition device to correspond to the target object;
[0009] Acquire image information of the target object by the image acquisition device;
[0010] The main control device determines monitoring result information according to the image information, and outputs notification information according to the monitoring result information.
[0011] Optionally, determining the target object from the current environment according to the audio information by the main control device includes:
[0012] Performing spectrum analysis on the audio information by the main control device to obtain a target spectrum;
[0013] Comparing a plurality of preset reference spectra with the target spectrum to obtain a comparison result, wherein the plurality of reference spectra correspond one-to-one to the plurality of preset reference objects;
[0014] The target object is determined from the multiple reference objects according to the comparison result.
[0015] Optionally, controlling the shooting angle of the image acquisition device to correspond to the target object includes:
[0016] Determine the position information of the target object relative to the image acquisition device according to the audio information;
[0017] The shooting angle of the image acquisition device is adjusted to face the target object according to the orientation information.
[0018] Optionally, determining the position information of the target object relative to the image acquisition device according to the audio information includes:
[0019] Extracting amplitude information corresponding to the target object from the audio information;
[0020] The position information of the target object relative to the image acquisition device is determined according to the amplitude information.
[0021] Optionally, there are multiple target objects, and the multiple target objects correspond to different priorities, and the controlling the shooting angle of the image acquisition device to correspond to the target objects includes:
[0022] The shooting angle of the image acquisition device is controlled to correspond to the target object with the highest priority among the multiple target objects.
[0023] Optionally, determining monitoring result information according to the image information by the main control device includes:
[0024] extracting initial image features from the image information;
[0025] Determining a target image feature that matches the initial image feature from a plurality of preset reference image features, wherein the plurality of reference image features correspond one-to-one to a plurality of preset reference monitoring information;
[0026] The reference monitoring information corresponding to the target image feature is determined as the monitoring result information.
[0027] Optionally, the plurality of reference monitoring information corresponds to a plurality of preset monitoring department addresses, and the outputting notification information according to the monitoring result information includes:
[0028] According to the monitoring result information, notification information is sent to the monitoring department address corresponding to the monitoring result information.
[0029] According to a second aspect of an embodiment of the present disclosure, a patrol monitoring method is provided, including:
[0030] In response to audio information of the current environment collected by the audio collection device, a target object is determined from the current environment according to the audio information, and a shooting angle of the image collection device is controlled to correspond to the target object;
[0031] In response to the image information of the target object acquired by the image acquisition device, monitoring result information is determined according to the image information, and notification information is output according to the monitoring result information.
[0032] According to a third aspect of the embodiments of the present disclosure, there is provided a patrol monitoring system, comprising: an image acquisition device, an audio acquisition device and a main control device,
[0033] The audio collection device is used to collect audio information of the current environment;
[0034] The main control device is used to determine the target object from the current environment according to the audio information, and control the shooting angle of the image acquisition device to correspond to the target object;
[0035] The image acquisition device is used to acquire image information of the target object;
[0036] The main control device is also used to determine monitoring result information according to the image information, and output notification information according to the monitoring result information.
[0037] Optionally, there are multiple audio acquisition devices, and the multiple audio acquisition devices are arranged in different directions of the image acquisition device.
[0038] The technical solution provided by the embodiments of the present disclosure includes at least the following beneficial effects: audio information of the current environment is collected by an audio collection device; the target object is determined from the current environment according to the audio information by the main control device, and the shooting angle of the image collection device is controlled to correspond to the target object; the image information of the target object is also collected by the image collection device; finally, the monitoring result information is determined by the main control device according to the image information, and notification information is output according to the monitoring result information. Thus, the target object with problems can be found according to the sound collected in the current environment, and then the image collection module is called in time to adjust the shooting angle to collect images of the target object with problems, and the monitoring results are reported after analyzing the collected images, thereby realizing timely and accurate monitoring of the problem situation, and avoiding the problem of being unable to respond to various emergencies on the scene in real time when the scene is monitored by manually adjusting the camera.
[0039] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings:
[0041] Figure 1 It is a flowchart of a patrol monitoring method according to an exemplary embodiment.
[0042] Figure 2 It is a flowchart of a patrol monitoring method according to another exemplary embodiment.
[0043] Figure 3 It is a flowchart of a patrol monitoring method according to another exemplary embodiment.
[0044] Figure 4 It is a structural diagram of a patrol monitoring system according to an exemplary embodiment.
[0045] Figure 5 is a block diagram of an electronic device according to an exemplary embodiment.
[0046] Figure 6 is a block diagram of an electronic device according to another exemplary embodiment. DETAILED DESCRIPTION
[0047] The specific implementation of the present disclosure is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the present disclosure, and is not used to limit the present disclosure.
[0048] In the description of the present disclosure, it should be noted that the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0049] In the description of the present disclosure, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present disclosure can be understood according to specific circumstances.
[0050] The following will be combined with the drawings in this embodiment to clearly and completely describe the technical solution in this embodiment. Obviously, the described embodiment is only a part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.
[0051] Figure 1 A patrol monitoring method is shown according to an exemplary embodiment. The method can be applied to a patrol monitoring system. The patrol monitoring system includes an image acquisition device, an audio acquisition device and a main control device. The main control device can be connected to the image acquisition device and the audio acquisition device respectively. Figure 1 As shown, the method may include:
[0052] 110. Collect audio information of the current environment through the audio collection device.
[0053] In some embodiments, the patrol monitoring system can collect the sound generated by the current environment in real time through the audio collection device to obtain audio information, and send the audio information to the main control device through the audio collection device.
[0054] Optionally, the audio collection device may also collect sound once every preset time interval to save power consumption, wherein the preset time interval may be set according to actual needs and is not limited here.
[0055] 120. Determine a target object from the current environment according to the audio information through the main control device, and control a shooting angle of the image acquisition device to correspond to the target object.
[0056] In some embodiments, the main control device may pre-store reference audio feature information of the sounds emitted by various monitoring objects under different circumstances. After the main control device obtains the audio information of the current scene, it can extract the current audio feature information from the audio information, and then compare the current audio feature information with the above-mentioned multiple reference audio feature information, and then determine which monitoring object emits the sound under which circumstances according to the comparison result, thereby determining the target object. For example, the comparison result of the current audio feature information extracted from the audio information is consistent with the reference audio feature information A, and the reference audio feature information A pre-corresponds to the sound emitted by the motor overload operation, then the main control device can determine the motor as the target object. Optionally, the above-mentioned audio feature information may include but is not limited to amplitude, frequency, etc. As an example, the sounds corresponding to multiple reference audio feature information may include but are not limited to: motor overload operation, abnormal operation sound, fire alarm, accident alarm sound, and sounds in specific occasions.
[0057] In some embodiments, the monitoring system pre-stores the shooting angles of the image acquisition device corresponding to multiple monitoring objects. After determining the target object from the multiple monitoring objects, the main control device can adjust the shooting angle of the image acquisition device to the shooting angle corresponding to the target object, thereby realizing image monitoring of the target object. Optionally, the image acquisition device can include but is not limited to a camera, an industrial camera, etc.
[0058] 130. Collect image information of the target object through the image acquisition device.
[0059] In some embodiments, after adjusting the shooting angle of the image acquisition device to correspond to the target object, the target object can be photographed in real time by the image acquisition device to obtain image information of the target object. It is understandable that when the shooting angle of the image acquisition device is adjusted to correspond to the target object, the shooting angle of the image acquisition device is roughly toward the target object, and a complete image of the target object can be captured. Optionally, the image information can be a video or a picture, which is not limited here.
[0060] 140. Determine monitoring result information according to the image information through the main control device, and output notification information according to the monitoring result information.
[0061] In some embodiments, the monitoring system pre-stores reference image features of multiple monitoring objects under different emergency situations. After collecting the image information of the target object, the monitoring system can extract the target image features from the image information, and then compare the target image features with multiple reference feature images, and then determine the emergency situation according to the comparison results, and generate corresponding monitoring result information according to the emergency situation. For example, the comparison result of the target image feature and the reference image feature A is consistent, and the emergency situation corresponding to the reference image feature A is motor overload operation, then the detection result information of the overload operation of the pointer motor can be generated at this time. Then, the notification information is output according to the detection result information. Optionally, the monitoring system can display the notification information of "motor overload operation" on its display according to the detection result information, or play the notification information of "motor overload operation" through an audio playback device, and can also send the notification information to the relevant departments remotely through a communication device for reminder.
[0062] It can be seen that in this embodiment, the audio information of the current environment is collected by the audio collection device; the target object is determined from the current environment according to the audio information by the main control device, and the shooting angle of the image collection device is controlled to correspond to the target object; the image information of the target object is also collected by the image collection device; finally, the monitoring result information is determined by the main control device according to the image information, and the notification information is output according to the monitoring result information. In this way, the target object with problems can be found according to the sound collected in the current environment, and then the image collection module is called in time to adjust the shooting angle to collect images of the target object with problems, and the monitoring results are reported after analyzing the collected images, thereby realizing timely and accurate monitoring of the problem situation and avoiding the problem of being unable to respond to various emergencies on the scene in real time when the scene is monitored by manually adjusting the camera.
[0063] Figure 2 A patrol monitoring method is shown according to another exemplary embodiment. The method can be applied to a patrol monitoring system, such as Figure 2 As shown, the method may include:
[0064] 210. Collect audio information of the current environment through the audio collection device.
[0065] The specific implementation of step 210 may refer to step 110, so it will not be described here in detail.
[0066] 220. Determine a target object from the current environment according to the audio information through the main control device, and control a shooting angle of the image acquisition device to correspond to the target object.
[0067] In some implementations, specific implementations of step 220 may include:
[0068] The main control device performs spectrum analysis on the audio information to obtain a target spectrum.
[0069] Exemplarily, the collected audio information may be input into a spectrum analyzer to obtain a target spectrum output by the spectrum analyzer.
[0070] The preset multiple reference frequency spectra are compared with the target frequency spectra to obtain a comparison result, and the multiple reference frequency spectra correspond to the preset multiple reference objects one by one.
[0071] The target object is determined from the multiple reference objects according to the comparison result.
[0072] Exemplarily, a plurality of reference spectra may be pre-stored in the monitoring system. As an example, a mapping relationship between the plurality of reference spectra and the plurality of reference objects may be as shown in Table 1.
[0073] Table 1
[0074] Spectrum Monitoring objects Emergency Reference spectrum A Reference object A A situation Reference spectrum B Reference object B B1 situation Reference spectrum C Reference object B B2 situation Reference spectrum D Reference object C C-Situation
[0075] It can be seen that after determining the target spectrum, the target spectrum can be compared with multiple reference spectra in Table 1. For example, if the comparison result is that the target spectrum matches the reference spectrum C, the reference object B can be determined as the target object.
[0076] Optionally, in addition to using frequency spectrum features in the audio information to determine the target object, the monitoring system may also use other audio features in the audio information to determine the target object, which is not limited here.
[0077] In some implementations, specific implementations of step 220 may include:
[0078] 221. Determine the position information of the target object relative to the image acquisition device according to the audio information.
[0079] As an example, there may be multiple audio acquisition devices, and the multiple audio acquisition devices may be arranged in different directions of the image acquisition device to collect sounds from different directions, so that the position information of the target object can be determined based on the sounds from different directions. For example, among the multiple audio acquisition devices, only the audio acquisition device located to the north of the image acquisition device collects the sound, so the position information can be determined as: the target object is in the north direction of the image acquisition device.
[0080] In some implementations, step 221 may include:
[0081] Amplitude information corresponding to the target object is extracted from the audio information; and position information of the target object relative to the image acquisition device is determined based on the amplitude information.
[0082] For example, for the sound received by the monitoring system, the sound amplitude is smaller at 30 degrees east of the image acquisition device, while the sound amplitude is larger at 35 degrees east of the image acquisition device. Then the azimuth information can be determined: the position of the target object at an angle of 35 degrees east of the image acquisition device.
[0083] In this embodiment, the position information of the target object is determined by the amplitude of the sound received in each direction, so that more accurate position information can be obtained.
[0084] It is understandable that, in addition to the amplitude, the position information of the target object can also be determined by other sound features, which are not limited here.
[0085] 222. Adjust the shooting angle of the image acquisition device to face the target object according to the orientation information.
[0086] Continuing with the above example, the main control device can control the shooting angle of the image acquisition device to be adjusted to 35 degrees toward the east, so as to aim at the target object.
[0087] In some implementations, there are multiple target objects, and the multiple target objects correspond to different priorities. The specific implementation of step 220 may include:
[0088] The shooting angle of the image acquisition device is controlled to correspond to the target object with the highest priority among the multiple target objects.
[0089] As an example, if the target objects include an alarm and a motor, and the priority of the alarm is higher than that of the motor, the main control device can control the shooting angle of the image acquisition device to face the position of the alarm. Optionally, the priority of the target object can be customized by the user according to the needs, which is not limited here.
[0090] Taking into account that multiple emergencies may be detected at the same time, in this embodiment, by setting different priorities for multiple target objects and controlling the shooting angle of the above-mentioned image acquisition device to correspond to the target object with the highest priority among the above-mentioned multiple target objects, it can be ensured that important emergencies are monitored first, making monitoring more flexible.
[0091] 230. Collect image information of the target object through the image acquisition device.
[0092] The specific implementation of step 230 may refer to step 130, so it will not be described here in detail.
[0093] 240. Extract initial image features from the above image information.
[0094] Exemplarily, initial image features may be extracted from image information using a pre-trained machine learning model, and the initial image features may include but are not limited to features such as contour, texture, shape, and color.
[0095] 250. Determine a target image feature that matches the initial image feature from a plurality of preset reference image features, wherein the plurality of reference image features correspond one-to-one to a plurality of preset reference monitoring information.
[0096] As an example, if reference image feature B among the multiple reference image features is recorded when the motor is overloaded, and the initial image feature matches the reference image feature B, then the reference image feature B can be determined as the target image feature. Optionally, if the similarity between the initial image feature and the reference image feature B is greater than a preset similarity, then it can be determined that the initial image feature matches the reference image feature B.
[0097] It can be understood that the feature type included in the reference image feature may be consistent or partially consistent with the feature type included in the initial image feature.
[0098] 260. Determine the reference monitoring information corresponding to the target image feature as the monitoring result information.
[0099] Continuing with the above example, since the initial image feature matches the reference image feature B, the reference monitoring information "motor overload operation" corresponding to the reference image feature B can be determined as the monitoring result information.
[0100] 270. Output notification information based on the above monitoring result information.
[0101] In some implementations, the above-mentioned multiple reference monitoring information corresponds to multiple preset monitoring department addresses, and the specific implementation of step 270 may include:
[0102] According to the above monitoring result information, notification information is sent to the monitoring department address corresponding to the above monitoring result information.
[0103] For example, after determining the specific situation based on the image information, the main control device can notify different monitoring departments to deal with the problem based on the address of the detection and monitoring department. For example, if there is an equipment abnormality, a notification message will be sent to the equipment department. If it is a production accident, a notification message will be sent to the production department, so that the emergency can be handled in a timely and effective manner.
[0104] Figure 3A patrol monitoring method is shown according to another exemplary embodiment. The method is applied to the main control device in the patrol monitoring system. Figure 3 As shown, the method may include:
[0105] 310. In response to the audio information of the current environment collected by the audio collection device, determine a target object from the current environment according to the audio information, and control the shooting angle of the image collection device to correspond to the target object.
[0106] 320. In response to the image information of the target object acquired by the image acquisition device, determine monitoring result information according to the image information, and output notification information according to the monitoring result information.
[0107] Among them, the more specific implementation of step 310 to step 320 can refer to the inspection and monitoring method in the above embodiment, so it will not be repeated here.
[0108] Figure 4 A patrol monitoring system is shown according to an exemplary embodiment. Figure 4 As shown, the system may include: an image acquisition device, an audio acquisition device and a main control device, wherein:
[0109] The above-mentioned audio collection device is used to collect audio information of the current environment.
[0110] The main control device is used to determine the target object from the current environment according to the audio information, and control the shooting angle of the image acquisition device to correspond to the target object.
[0111] The image acquisition device is used to acquire image information of the target object.
[0112] The main control device is also used to determine monitoring result information according to the image information, and output notification information according to the monitoring result information.
[0113] Optionally, there are multiple audio acquisition devices, and the multiple audio acquisition devices are arranged in different directions of the image acquisition device.
[0114] As an example, Figure 4 As shown, the audio acquisition device may include an audio acquisition device A, an audio acquisition device B, an audio acquisition device C and an audio acquisition device D. The image acquisition device may be a camera I. The main control device may include a camera control module G, an intelligent analysis module H, a central processing module F and a data transmission module E.
[0115] Among them, the above-mentioned audio collection device can monitor the audio information (hereinafter referred to as sound) in 720 degrees in real time, such as front, back, left, right, up and down. The intelligent analysis module H can filter the collected sound through a noise filter, extract the sound features, and compare them with the sound features in the sample library.
[0116] The camera control module G can control the camera and rotate the camera I to a specific angle.
[0117] Data transmission module E, which can transmit image information and audio information to central processing module F;
[0118] The central processing module F mentioned above can perform policy control and processing. Combined with the intelligent analysis module H, it can identify sound characteristics and sound directions, and can also perform camera rotation control so as to capture the desired clear video for analysis.
[0119] The above-mentioned intelligent analysis module H can pre-record specific audio so that the central processing module can perform analysis and comparison.
[0120] The central processing module F can analyze the sound by spectrum analysis and compare it with the feature quantity stored in the intelligent analysis module H. When the collected sound spectrum is found to be consistent with the stored spectrum, it is necessary to collect images of the object corresponding to the sound. The intelligent analysis module H stores the sound features emitted by the scenes that need to be used. After the sound is sampled and analyzed, the specific sound features are stored in the intelligent analysis module H. Optionally, the sound features are the sound of motor overload operation, abnormal operation, fire alarm, accident alarm and the sound of specific occasions.
[0121] For example, in practical applications, the workflow of the inspection and monitoring system is as follows:
[0122] (1) The audio collection device collects abnormal noise of the equipment to be identified, or the background noise of the working scene to be identified, or other alarm sounds.
[0123] (2) Perform spectrum analysis on the collected sound, draw out the characteristic spectrum, and save the characteristic spectrum in the intelligent analysis module H.
[0124] (3) The patrol monitoring system is started and put into operation; the system begins to monitor the sound and noise within a 360-degree environment.
[0125] (4) When an alarm is sounded on the east side of the monitoring system, the monitoring terminal detects the sound and transmits the sound to the central processing module F through the transmission module E for processing.
[0126] (5) When it is found that the spectrum is consistent with the saved spectrum, the central processing module F sends it to the camera control module G for recording; the camera control module G adjusts the camera I to the angle where the sound is located. Optionally, the camera control module G can also determine a more precise adjustment angle of the camera based on other characteristics of the sound, such as the amplitude.
[0127] (6) Camera I will rotate to a specific position according to the instructions of camera control module G, and then turn on the video recording function of camera I.
[0128] (7) The video captured by camera I enters central processing module F through transmission module E, and then starts the analysis of video intelligent recognition, and makes judgments and recognitions based on the video feature quantities stored in intelligent analysis module H.
[0129] Optionally, the recorded video can be locally identified and analyzed, and the remote control platform can be notified to issue an alarm. If the local cannot be processed, the video can be compressed and transmitted to the remote control platform, which will perform video identification or manual video processing to determine where the problem occurred.
[0130] Optionally, after the video determines the specific situation, the central processing module F can notify different processing units to handle the problem; if it is an equipment abnormality, the equipment department will be notified, and if it is a production accident, the production department will be notified.
[0131] It can be seen that the patrol monitoring system of this embodiment collects sound, extracts features, judges and compares the features of the sound, identifies the direction of the sound, and judges whether the source of the sound is a scene that needs to be monitored. If necessary, the video camera is adjusted to shoot and track at the same time. It can make the patrol inspection more accurate and more purposeful. There is no need to waste too much time on unnecessary places. In addition, the characteristics of sound are often more sensitive and reliable than vision during the monitoring process, so it has a stronger patrol perception.
[0132] Figure 5 FIG. 1 is a block diagram of an electronic device according to an exemplary embodiment. Figure 5 As shown, the electronic device 700 may include: a processor 701 and a memory 702. The electronic device 700 may also include one or more of a multimedia component 703, an input / output (I / O) interface 704, and a communication component 705.
[0133] The processor 701 is used to control the overall operation of the electronic device 700 to complete all or part of the steps in the above-mentioned inspection and monitoring method. The memory 702 is used to store various types of data to support the operation of the electronic device 700, and these data may include, for example, instructions for any application or method used to operate on the electronic device 700, and application-related data, such as contact data, sent and received messages, pictures, audio, video, etc. The memory 702 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, referred to as EPROM), programmable read-only memory (Programmable Read-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic memory, flash memory, magnetic disk or optical disk. The multimedia component 703 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone, which is used to receive external audio signals. The received audio signal may be further stored in the memory 702 or sent through the communication component 705. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 704 provides an interface between the processor 701 and other interface modules, and the other interface modules may be keyboards, mice, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 705 is used for wired or wireless communication between the electronic device 700 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IOT, eMTC, or other 5G, etc., or a combination of one or more of them, is not limited here. Therefore, the corresponding communication component 705 may include: a Wi-Fi module, a Bluetooth module, an NFC module, etc. The input / output (I / O) interface 704 may be connected to an image acquisition device and / or an audio acquisition device. The electronic device may be equivalent to the main control device in the above embodiment.
[0134] In an exemplary embodiment, the electronic device 700 can be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to perform the above-mentioned inspection and monitoring method.
[0135] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by a processor, the steps of the above-mentioned inspection and monitoring method are implemented. For example, the computer-readable storage medium can be the above-mentioned memory 702 including program instructions, and the above-mentioned program instructions can be executed by the processor 701 of the electronic device 700 to complete the above-mentioned inspection and monitoring method.
[0136] Figure 6 1 is a block diagram of an electronic device according to another exemplary embodiment. For example, the electronic device 1900 may be provided as a server. Figure 6 The electronic device 1900 includes a processor 1922, which may be one or more, and a memory 1932 for storing a computer program executable by the processor 1922. The computer program stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processor 1922 may be configured to execute the computer program to perform the above-mentioned inspection and monitoring method.
[0137] In addition, the electronic device 1900 may further include a power supply component 1926 and a communication component 1950, wherein the power supply component 1926 may be configured to perform power management of the electronic device 1900, and the communication component 1950 may be configured to implement communication of the electronic device 1900, for example, wired or wireless communication. In addition, the electronic device 1900 may further include an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, etc.
[0138] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by the processor, the steps of the above-mentioned inspection and monitoring method are implemented. For example, the non-temporary computer-readable storage medium can be the above-mentioned memory 1932 including program instructions, and the above-mentioned program instructions can be executed by the processor 1922 of the electronic device 1900 to complete the above-mentioned inspection and monitoring method.
[0139] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program that can be executed by a programmable device. The computer program has a code portion for executing the above-mentioned patrol monitoring method when executed by the programmable device.
[0140] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings; however, the present disclosure is not limited to the specific details in the above embodiments. Within the technical concept of the present disclosure, a variety of simple modifications can be made to the technical solution of the present disclosure, and these simple modifications all fall within the protection scope of the present disclosure.
[0141] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.
[0142] In addition, various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.
Claims
1. A patrol monitoring method, characterized in that: Applied to a patrol inspection and monitoring system, the patrol inspection and monitoring system includes an image acquisition device, an audio acquisition device and a main control device, and the method includes: Collecting audio information of the current environment through the audio collection device; The main control device determines the target object from the current environment according to the audio information, and controls the shooting angle of the image acquisition device to correspond to the target object. The main control device pre-stores reference audio feature information of the sounds emitted by various monitoring objects in different situations. The sounds corresponding to the reference audio feature information may include but are not limited to the sounds of motor overload operation, abnormal operation, fire alarm and accident alarm, and sounds in specific occasions. Acquire image information of the target object by the image acquisition device; Determining monitoring result information according to the image information by the main control device, and outputting notification information according to the monitoring result information; The determining, by the main control device, monitoring result information according to the image information, and outputting notification information according to the monitoring result information includes: Extracting initial image features from the image information based on a pre-trained machine learning model, the initial image features including but not limited to contour, texture, shape and color; Determining a target image feature that matches the initial image feature from a plurality of preset reference image features, wherein the plurality of reference image features correspond one-to-one to a plurality of preset reference monitoring information, the plurality of reference monitoring information correspond to a plurality of preset monitoring department addresses, and the reference image feature is a feature of the monitored object under different emergency situations; Determine monitoring result information based on the reference monitoring information corresponding to the target image feature; Sending notification information to the monitoring department addresses corresponding to the monitoring result information according to the monitoring result information, so as to notify different monitoring departments to handle the problem; When the monitoring result shows that the device is abnormal, a notification message is sent to the device; When the monitoring result is a production accident, a notification message is sent to the production department.
2. The method according to claim 1, characterized in that The determining the target object from the current environment according to the audio information by the main control device includes: Performing spectrum analysis on the audio information by the main control device to obtain a target spectrum; Comparing a plurality of preset reference spectra with the target spectrum to obtain a comparison result, wherein the plurality of reference spectra correspond one-to-one to the plurality of preset reference objects; The target object is determined from the multiple reference objects according to the comparison result.
3. The method according to claim 1, characterized in that The controlling the shooting angle of the image acquisition device to correspond to the target object comprises: Determine the position information of the target object relative to the image acquisition device according to the audio information; The shooting angle of the image acquisition device is adjusted to face the target object according to the orientation information.
4. The method according to claim 3, characterized in that The step of determining the position information of the target object relative to the image acquisition device according to the audio information comprises: Extracting amplitude information corresponding to the target object from the audio information; The position information of the target object relative to the image acquisition device is determined according to the amplitude information.
5. The method according to any one of claims 1 to 4, characterized in that: There are multiple target objects, and the multiple target objects correspond to different priorities. The controlling the shooting angle of the image acquisition device to correspond to the target objects includes: The shooting angle of the image acquisition device is controlled to correspond to the target object with the highest priority among the multiple target objects.
6. A patrol monitoring method, characterized in that: include: In response to the audio information of the current environment collected by the audio collection device, a target object is determined from the current environment according to the audio information, and the shooting angle of the image collection device is controlled to correspond to the target object. The main control device pre-stores reference audio feature information of the sounds emitted by various monitoring objects in different situations. The sounds corresponding to the reference audio feature information may include but are not limited to the sounds of motor overload operation, abnormal operation, fire alarm and accident alarm, and sounds in specific occasions. In response to the image information of the target object acquired by the image acquisition device, determining monitoring result information according to the image information, and outputting notification information according to the monitoring result information; Determining monitoring result information according to the image information by the main control device, and outputting notification information according to the monitoring result information, including: Extracting initial image features from the image information based on a pre-trained machine learning model, the initial image features including but not limited to contour, texture, shape and color; Determining a target image feature that matches the initial image feature from a plurality of preset reference image features, wherein the plurality of reference image features correspond one-to-one to a plurality of preset reference monitoring information, the plurality of reference monitoring information correspond to a plurality of preset monitoring department addresses, and the reference image feature is a feature of the monitored object under different emergency situations; Determine monitoring result information based on the reference monitoring information corresponding to the target image feature; Sending notification information to the monitoring department addresses corresponding to the monitoring result information according to the monitoring result information, so as to notify different monitoring departments to handle the problem; When the monitoring result shows that the device is abnormal, a notification message is sent to the device; When the monitoring result is a production accident, a notification message is sent to the production department.
7. A patrol monitoring system, characterized in that: include: Image acquisition device, audio acquisition device and main control device, The audio collection device is used to collect audio information of the current environment; The main control device is used to determine the target object from the current environment according to the audio information, and control the shooting angle of the image acquisition device to correspond to the target object. The main control device pre-stores reference audio feature information of the sounds emitted by various monitoring objects in different situations. The sounds corresponding to the reference audio feature information may include but are not limited to the sounds of motor overload operation, abnormal operation, fire alarm and accident alarm, and sounds in specific occasions; The image acquisition device is used to acquire image information of the target object; The main control device is also used to determine monitoring result information according to the image information, and output notification information according to the monitoring result information; Extracting initial image features from the image information based on a pre-trained machine learning model, the initial image features including but not limited to contour, texture, shape and color; determining target image features matching the initial image features from a plurality of preset reference image features, the plurality of reference image features corresponding one-to-one to a plurality of preset reference monitoring information, the plurality of reference monitoring information corresponding to a plurality of preset monitoring department addresses, the reference image features being features of the monitored object under different emergency situations; Determine monitoring result information based on the reference monitoring information corresponding to the target image feature; send notification information to the monitoring department address corresponding to the monitoring result information according to the monitoring result information to notify different monitoring departments to handle the problem; When the monitoring result indicates that the equipment is abnormal, a notification message is sent to the equipment department; when the monitoring result indicates a production accident, a notification message is sent to the production department.
8. The inspection and monitoring system according to claim 7, characterized in that: There are multiple audio acquisition devices, and the multiple audio acquisition devices are arranged in different directions of the image acquisition device.
