Method and system for monitoring environment by using surveillance robot

Abstract

Disclosed is a method for monitoring an environment by using a surveillance robot. The method comprises the steps of: moving the surveillance robot to a surveillance position for monitoring an object to be monitored, located in the environment; obtaining, by the surveillance robot, sound data corresponding to a sound signal generated from the object to be monitored at the surveillance position; and determining whether the object to be monitored is in an abnormal state on the basis of the obtained sound data.

Description

Environmental monitoring method and system using surveillance robots

[0001] The present invention relates to a technology for monitoring abnormal situations such as gas leaks or electric discharges that may occur in the environment using a surveillance robot.

[0002] Robots are used to assist or replace human tasks, and are utilized in diverse fields such as manufacturing, healthcare, and the service industry. In particular, technological innovations in fields such as artificial intelligence, sensor technology, and robotics are making robots smarter and more capable of performing diverse tasks.

[0003] Recently, robots have been actively used in various fields such as environmental monitoring, structural inspection, and disaster response, and their scope of application continues to expand as technology advances.

[0004] Among these, environmental monitoring robots are robots that collect data from the environment using various sensors mounted on the robot, analyze the collected data, and monitor the state of the environment, such as natural disasters and environmental pollution situations. As industrial accidents have increased recently, the need for the environmental monitoring robots described above has further increased.

[0005] In particular, gas leaks from machine parts in factories can lead to explosions or fires, and electrical discharges can expose people to electrical hazards and fire accidents. Therefore, environmental monitoring robots are essential for the safety of workers and for rapid response to dangerous situations.

[0006] The present invention has been conceived in response to the aforementioned needs, and the purpose of the present invention is to propose a method for monitoring and detecting abnormal situations, such as gas leaks or electrical discharges, that may occur in the environment using a surveillance robot.

[0007] In order to achieve the above-described purpose, an environmental monitoring method using a surveillance robot according to an embodiment of the present invention includes a step of the surveillance robot moving to a surveillance position for monitoring a surveillance target object located in an environment, a step of the surveillance robot obtaining acoustic data corresponding to an acoustic signal generated from the surveillance target object at the surveillance position, and a step of determining whether the surveillance target object is in an abnormal state based on the obtained acoustic data.

[0008] And, the abnormal state of the object to be monitored may include at least one of gas leakage and electric discharge.

[0009] In addition, the step of determining may include a step of calculating at least three locations of abnormal sound signals in order of magnitude from the sound data, and a step of analyzing beamformed sound values ​​of the at least three locations calculated to determine whether the abnormal state exists.

[0010] And, the step of calculating at least three positions of the abnormal sound signal may include a step of calculating a first position of the first abnormal sound signal by searching for a position of the first abnormal sound signal from the sound data, a step of calculating a second position of the second abnormal sound signal by searching for a position of the second abnormal sound signal from the sound data after erasing the first abnormal sound signal from the sound data through phase and amplitude compensation, and a step of calculating a third position of the third abnormal sound signal by searching for a position of the third abnormal sound signal from the sound data after erasing the second abnormal sound signal from the sound data through phase and amplitude compensation.

[0011] In addition, a QR code is attached to one area of ​​the object to be monitored, and the surveillance robot may further include a step of obtaining image data corresponding to an optical signal generated from the object to be monitored at the surveillance location, and a step of setting an area of ​​interest for securing at least one of the sound data and the image data based on the QR code when the QR code is recognized at the surveillance location from the image data.

[0012] And, the step of setting the region of interest may be such that the region of interest has a rectangular shape and the QR code is set to be located at the center and one of the corners of the rectangular shape.

[0013] In addition, the method may further include a step of controlling at least one of panning and tilting of the camera so that at least one of the sound data and the image data is acquired in the set region of interest.

[0014] And, the surveillance robot may further include a step of acquiring image data corresponding to an optical signal generated from the target object at the surveillance location, and the step of determining whether the target object is in an abnormal state may include a step of determining that the target object is in an abnormal state when the target object recognized from the image data is an object with a high possibility of being in an abnormal state and audio data is acquired from the object, and a step of estimating the type of the audio data using a learned model to determine whether the target object is in an abnormal state when the target object recognized from the image data is an object with a low possibility of being in an abnormal state and audio data is acquired from the object.

[0015] In addition, the method may further include a step of storing a generated acoustic image based on acoustic data for the entire space of the environment and a step of determining whether an abnormal acoustic signal is progressing or occurring through a cumulative comparison of the stored acoustic images.

[0016] In addition, the step of determining whether the abnormal sound signal has occurred may include a step of aligning the plurality of sound images through matching of the plurality of sound images, and a step of comparing the aligned plurality of sound images to detect at least one of an area where the sound signal has changed from non-occurrence to occurrence and an area where the change in the sound value of the signal has exceeded a preset value.

[0017] In addition, the step of uploading the result data according to the above judgment to the server may be further included.

[0018] In addition, in the moving step, the surveillance robot can autonomously drive along the set driving path to which the surveillance position is assigned.

[0019] Meanwhile, an environmental monitoring system using a surveillance robot according to an embodiment of the present invention for achieving the above-described purpose includes a surveillance robot that moves to a surveillance position for monitoring a surveillance target object located in an environment, and obtains acoustic data corresponding to an acoustic signal generated from the surveillance target object at the surveillance position, and the obtained acoustic data can be used to determine whether the surveillance target object is in an abnormal state.

[0020] Meanwhile, a computer program stored in a computer-readable recording medium according to an embodiment of the present invention for achieving the above-described purpose may include a program code for executing the above-described environmental monitoring method.

[0021] In addition, a computer-readable recording medium according to an embodiment of the present invention for achieving the above-described purpose may record a program for executing the above-described environmental monitoring method.

[0022] According to the present invention, a surveillance robot equipped with an acoustic camera can drive in an actual environment such as a factory and collect acoustic signals from a surveillance target object to monitor, so that abnormal situations such as gas leaks or electrical discharges can be quickly detected, and further, when an abnormal situation occurs, a manager can be quickly notified, so that abnormal situations such as gas leaks or electrical discharges can be quickly resolved.

[0023] In addition, according to the present invention, by determining a gas leak or electric discharge by beamforming sound values ​​of the positions of at least three abnormal acoustic signals in order of magnitude, it is possible to easily and accurately find the position of an abnormal situation of gas leak or electric discharge that may occur simultaneously.

[0024] In addition, according to the present invention, by setting an area of ​​interest for collecting audio data and / or image data using a QR code, the problem of the view area changing each time the surveillance robot is positioned at a surveillance location can be solved.

[0025] In addition, according to the present invention, by determining an abnormal state by setting a high weight for an object with a high possibility of an abnormal state such as a gas leak, such as a gage, among various surveillance target objects included in image data, the accuracy of determining an abnormal state can be increased.

[0026] In addition, according to the present invention, by acquiring acoustic data for the entire space of an environment at a predetermined period and accumulating and comparing the acquired acoustic data for the entire space, it is possible to determine the progress of an abnormal state for the entire space of the environment, etc.

[0027] The effects of the present invention are not limited to those described above, and other effects not mentioned will be clearly recognized by those skilled in the art from the description below.

[0028] Figure 1 is a conceptual diagram showing an environmental monitoring system according to one embodiment of the present invention.

[0029] Figure 2 is a block diagram showing a surveillance robot according to one embodiment of the present invention.

[0030] FIG. 3 is a drawing showing the shape of a surveillance robot according to one embodiment of the present invention.

[0031] Figures 4 and 5 are timing diagrams showing the operation of an environmental monitoring system using a surveillance robot according to one embodiment of the present invention.

[0032] FIGS. 6 and 7 are drawings for explaining a process of determining whether a surveillance target object is in an abnormal state based on beamformed sound values ​​according to one embodiment of the present invention.

[0033] FIGS. 8 and 9 are drawings for explaining a process of setting a region of interest and controlling panning and tilting of a camera according to one embodiment of the present invention.

[0034] FIGS. 10 to 11 are diagrams for explaining a process of determining whether a surveillance target object is in an abnormal state by combining image data-based object recognition and audio data according to one embodiment of the present invention.

[0035] FIGS. 12 and 13 are diagrams for explaining a process of detecting an abnormal area through cumulative comparison of acoustic data according to one embodiment of the present invention.

[0036] Figure 14 is an exemplary diagram showing the configuration of a computing device according to one embodiment of the present invention.

[0037] Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. The following detailed description is provided to facilitate a comprehensive understanding of the methods, devices, and / or systems described herein. However, these are merely examples and the present invention is not limited thereto.

[0038] In describing embodiments of the present invention, if a detailed description of a known technology related to the present invention is judged to unnecessarily obscure the gist of the present invention, the detailed description will be omitted. In addition, the terms described below are terms defined in consideration of their functions in the present invention, and this may vary depending on the intention or custom of the user or operator. Therefore, the definitions should be made based on the contents throughout this specification. The terminology used in the detailed description is only for the purpose of describing embodiments of the present invention and should not be limited in any way. Unless clearly used otherwise, the singular form includes the plural form. In this description, expressions such as "comprises" or "having" are intended to indicate certain features, numbers, steps, operations, elements, parts or combinations thereof, and should not be construed to exclude the presence or possibility of one or more other features, numbers, steps, operations, elements, parts or combinations thereof other than those described.

[0039] Additionally, in describing components of embodiments of the present invention, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only intended to distinguish the components from other components, and the nature, order, or sequence of the components are not limited by the terms.

[0040]

[0041] FIG. 1 is a conceptual diagram illustrating an environmental monitoring system according to an embodiment of the present invention. Referring to FIG. 1, the environmental monitoring system (1000) may include a surveillance robot (100) that drives around an environment (10) and monitors various surveillance target objects located in the environment, and a server (200) that communicates with the surveillance robot (100) and transmits and receives various data with the surveillance robot (100).

[0042] From a functional perspective, the surveillance robot (100) may be composed of an acoustic camera (2000) and a driving robot (3000).

[0043] Here, the environment is a real environment in which the surveillance robot (100) runs, and various objects that are targets of surveillance by the surveillance robot (100) can be installed in the environment.

[0044] For example, the environment can be, but is not limited to, a factory, an underground facility, a structure, or other location where machinery, equipment, and electronic devices are installed. Furthermore, the environment can be implemented as a closed environment, such as a factory, or as an open environment with high-voltage power lines installed.

[0045] In these various environments, the surveillance robot (100) can move to a surveillance position to monitor a surveillance target object located in the environment, and acquire acoustic data corresponding to an acoustic signal generated from the surveillance target object at the surveillance position. In addition, the surveillance robot (100) can acquire image data corresponding to an optical signal generated from the surveillance target object at the surveillance position.

[0046] Here, the object to be monitored is an object located (or installed or equipped) in the environment, and may be various, such as, but not limited to, a flange, a gage, a valve, a flow meter, etc.

[0047] Additionally, the view area according to the field of view (FOV) of the acoustic camera and / or optical camera of the surveillance robot (100) may include at least one surveillance target object. For example, the view area may include multiple surveillance target objects such as a flange, a gauge, and a valve.

[0048] Meanwhile, at least one of the surveillance robot (100) and the server (200) can determine whether the surveillance target object is in an abnormal state based on the acquired acoustic data. In this case, the abnormal state may include at least one of a gas leak and an electrical discharge that cannot be detected by the human eye.

[0049] For example, a gas leak in factory equipment that cannot be detected by the human eye can lead to the risk of explosion or fire. Another example is an electrical discharge that cannot be detected by the human eye, which can expose people to electrical hazards. Accordingly, at least one of the surveillance robot (100) and the server (200) according to the present invention can analyze the acquired acoustic data to determine whether at least one of a gas leak and an electrical discharge is occurring in the surveillance target object.

[0050] The surveillance robot (100) according to the present invention will be described in more detail with reference to FIGS. 2 and 3.

[0051] Fig. 2 is a block diagram showing a surveillance robot according to an embodiment of the present invention. Fig. 3 is a drawing showing the shape of a surveillance robot according to an embodiment of the present invention. Referring to Figs. 2 and 3, the surveillance robot (100) may include all or part of an audio data generation unit (110), an image data generation unit (120), an output unit (130), a communication unit (140), a driving unit (150), a processor (160), and a control and processing unit (170).

[0052] The acoustic data generation unit (110) may be implemented as an acoustic camera composed of an array of multiple microphone sensors. For example, as illustrated in FIG. 3, the multiple microphone sensors constituting the acoustic data generation unit (110) may be arranged in a spiral or random manner toward the outer periphery centered on the lens constituting the image data generation unit (120).

[0053] Here, each of the plurality of microphone sensors constituting the acoustic data generation unit (110) can receive an acoustic signal from an acoustic scene and generate acoustic data based on the received acoustic signal.

[0054] An acoustic scene is a space in the real world where a sound source that generates sound is located, and the environment according to the present invention may be an example of an acoustic scene.

[0055] Here, the acoustic signal of the sound scene may include audible sounds, ultrasound, infrasound, etc. Accordingly, the acoustic data may include audible sound data in the range of 20 Hz to 20 kHz, which is the frequency of sound waves that can be heard by the human ear, and inaudible sound data in a band outside the audible range.

[0056] To this end, the audio data generation unit (110) may include an inaudible audio data generation unit (111) that recognizes an audio signal having a frequency in the inaudible range and generates inaudible audio data corresponding to the inaudible audio signal, and an audible audio data generation unit (112) that recognizes an audio signal having a frequency in the audible range and generates audible audio data corresponding to the audible audio signal.

[0057] Meanwhile, the image data generation unit (120) may be implemented as an optical camera composed of a lens and an image sensor. For example, as illustrated in FIG. 3, the lens constituting the image data generation unit (120) may be positioned at the center of a plurality of microphone sensors constituting the audio data generation unit (110). Here, the image data generation unit (120) may receive an optical signal from an optic scene and generate image data based on the received optical signal.

[0058] Each of the audio data generation unit (110) and the image data generation unit (120) can transmit the generated audio data and image data to the processor (160).

[0059] The processor (160) calculates distances between sensors and acoustic scene points using the coordinates of each sensor constituting the microphone sensor array and the coordinates of a real-world acoustic scene, applies time delay correction to each acoustic signal using the delay distances calculated based on the calculated distances, and calculates the sound source value of each of a plurality of acoustic scene points by adding them up, thereby performing beamforming.

[0060] This processor (160) may be installed inside the acoustic camera (2000). However, the installation location is only an example of one implementation, and it may be implemented as being installed outside the acoustic camera (2000).

[0061]

[0062] Meanwhile, the driving unit (150) is composed of driving wheels, a motor that provides driving force to the driving wheels, etc., and can provide driving force for movement (or driving) of the surveillance robot (100).

[0063] In addition, the communication unit (140) can perform a communication function for data transmission of the surveillance robot (100). According to an example that has not been proposed, the communication unit (140) can be implemented as a module for Bluetooth communication, Zigbee communication, WI-FI communication, etc., and can also be implemented as a configuration that can perform long-distance wireless communication such as LTE communication, 5G communication, etc. Accordingly, the surveillance robot (100) can transmit and receive various data with the server (200) through the communication unit (140).

[0064] Meanwhile, the control and processing unit (170) can control the overall operation of the surveillance robot (100). Specifically, the control unit (170) can control all or part of the audio data generation unit (110), the image data generation unit (120), the output unit (130), the communication unit (140), the driving unit (150), and the processor (160).

[0065] The control and processing unit (170) may transmit a command requesting transmission of sound data at a specific location to the processor (160), or may request the processor (160) to determine whether the monitored object, as determined by the processor (160), is in an abnormal state.

[0066] The control and processing unit (170) can determine whether the object to be monitored is in an abnormal state based on the acoustic data acquired from the acoustic data generation unit (110). For example, the control and processing unit (170) can analyze the acoustic data to determine whether the acoustic signal generated from the object to be monitored is an abnormal acoustic signal such as a gas leak or an electric discharge, and can determine whether the object to be monitored is in an abnormal state based on the determination result.

[0067] Additionally, the control and processing unit (170) can generate an acoustic-optical combined image that combines an acoustic image for an acoustic scene generated according to beamforming and an optical image for an optical scene at the same time as the acoustic scene.

[0068] In addition, the control and processing unit (170) can control the output unit (130) to display whether the object to be monitored is in an abnormal state or display the abnormal position of the object to be monitored. At this time, the control unit (170) can control the output unit (130) to combine an acoustic image for an acoustic scene generated by beamforming processing and an optical image for an optical scene at the same time as the acoustic scene and display the combined image on the output unit (130).

[0069] This control and processing unit (170) may be installed inside the driving robot (3000). However, the installation location is only an example of one implementation, and the control and processing unit (170) may also be installed outside the driving robot (3000).

[0070] Meanwhile, in FIG. 2, the processor (160) and the control and processing unit (170) are described as having distinct functions, but the control and processing unit (170) may be implemented to perform some or all of the functions of the processor (160). In addition, according to another implementation example, the processor (160) may be implemented to perform some or all of the functions of the control and processing unit (170).

[0071] In addition, although FIG. 2 illustrates implementation as a separate module, it may also be implemented as a single module in which the control and processing unit (170) and the processor (160) are integrated.

[0072] The system (1000) of the present invention configured as described above may be configured as a system that connects a surveillance robot (100) and a server (200) as individual computing devices, or may be configured as computing devices installed as services on the same server (200) or surveillance robot (100). Alternatively, each computing device may be executed as a virtual machine in a cloud computing environment.

[0073] The above-described processor (160) and control and processing unit (170) may be an example of an implementation of a computing device.

[0074] Meanwhile, the process of determining whether the above-described surveillance target object is in an abnormal state may be performed by the surveillance robot (100) or by the server (200). This will be described in more detail with reference to FIGS. 4 and 5.

[0075]

[0076] FIG. 4 is a timing diagram illustrating the operation of an environmental monitoring system using a surveillance robot according to an embodiment of the present invention. Referring to FIG. 4, the surveillance robot (100) can determine whether a surveillance target object is in an abnormal state.

[0077] First, the computing device of the surveillance robot (100) can control the driving unit (150) to move to a surveillance position to monitor a surveillance target object located in the environment (S100).

[0078] Here, the moving step (S100) may be a process in which the surveillance robot (100) moves to the surveillance location by autonomous driving or non-autonomous driving.

[0079] For example, a surveillance robot (100) can autonomously drive along a preset driving path to which a surveillance position is assigned. According to one implementation example, a computing device of the surveillance robot (100) can create and store a map of the environment through SLAM (Simultaneous Localization And Map-Building, Simultaneous Localization and Mapping) technology, and calculate the location of the surveillance robot (100) in real time based on the map data to autonomously drive in the environment. According to another example, a driving guide trajectory for guiding the driving of the surveillance robot (100) can be installed in the environment, and the surveillance robot (100) can autonomously drive along the driving guide trajectory.

[0080] As another example, in the case of non-autonomous driving, the surveillance robot (100) can receive a user's remote operation command and drive non-autonomously in the environment.

[0081] In addition, the computing device of the surveillance robot (100) can obtain acoustic data corresponding to an acoustic signal generated from the surveillance target object at the surveillance location (S200). Here, step S200 may further include a step of the surveillance robot (100) obtaining image data corresponding to an optical signal generated from the surveillance target object at the surveillance location.

[0082] In addition, the computing device of the surveillance robot (100) can determine whether the surveillance target object is in an abnormal state based on the acquired acoustic data (S300). Here, steps S200 and S300 will be described later with reference to FIGS. 6 to 13.

[0083] In addition, the computing device of the surveillance robot (100) can control the communication unit (140) to upload the result data according to the determination of abnormality to the server (200) (S400). In this case, the server (200) can store the received result data (S500).

[0084] Here, the result data may include judgment result information indicating whether the surveillance target object is normal or abnormal, and acoustic data acquired from the surveillance target object. In addition, the result data may further include at least one of image data acquired from the surveillance target object, an optical image generated from the image data, an acoustic image generated by beamforming the acoustic data, and an acoustic-optical combined image that combines the optical image and the acoustic image.

[0085]

[0086] FIG. 5 is a timing diagram illustrating the operation of an environmental monitoring system using a fixed camera according to another embodiment of the present invention. Referring to FIG. 5, a server (200) can determine whether a target object to be monitored is in an abnormal state.

[0087] First, the computing device of the surveillance robot (100) can control the driving unit (150) to move to a surveillance position to monitor a surveillance target object located in the environment (S100).

[0088] In addition, the computing device of the surveillance robot (100) can obtain acoustic data corresponding to an acoustic signal generated from the surveillance target object at the surveillance location (S200). Here, step S200 may further include a step of the surveillance robot (100) obtaining image data corresponding to an optical signal generated from the surveillance target object at the surveillance location.

[0089] And, the computing device of the surveillance robot (100) can transmit the acquired acoustic data to the server (200) (S600).

[0090] In this case, the computing device of the server (200) can determine whether the object to be monitored is in an abnormal state based on the acoustic data received from the surveillance robot (100) (S700). Here, steps S200 and S700 will be described later with reference to FIGS. 6 to 13.

[0091] And, the computing device of the server (200) can store the result data according to the determination of whether or not it is in an abnormal state (S400).

[0092] Hereinafter, with reference to FIGS. 6 to 13, the process of determining whether a surveillance target object is in an abnormal state through acoustic data and / or acoustic data analysis will be described in more detail. In the following, the processor may be a computing device of a surveillance robot (100) or a computing device of a server (200).

[0093]

[0094] FIGS. 6 and 7 are drawings for explaining a process of determining whether a surveillance target object is in an abnormal state based on at least three beamformed sound values ​​in order of size according to one embodiment of the present invention.

[0095] Referring to FIG. 6, the surveillance robot (100) moves to a surveillance position to monitor a surveillance target object, and the audio data generation unit (110) of the surveillance robot (100) can generate audio data corresponding to an audio signal generated from the surveillance target object.

[0096] In this case, the computing device can obtain sound data generated by the sound data generation unit (100), calculate at least three locations of abnormal sound signals from the obtained sound data, and analyze the beamformed sound values ​​of at least three locations to determine whether the object to be monitored is in an abnormal state.

[0097] Referring to FIG. 7, the step (S300, S7000) of determining the abnormal state of the above-described surveillance target object may include steps S11 to S14, which will be described later.

[0098] The computing device can search for the location of the first abnormal acoustic signal from the acoustic data and calculate the first location of the first abnormal acoustic signal (S11).

[0099] And, the computing device can erase the first abnormal sound signal from the sound data by compensating for the phase and amplitude of the first abnormal sound signal, and then search for the position of the second abnormal sound signal from the sound data to calculate the second position of the second abnormal sound signal (S12).

[0100] And, the computing device can erase the second abnormal sound signal from the sound data by compensating for the phase and amplitude of the second abnormal sound signal, and then search for the position of the third abnormal sound signal from the sound data to calculate the third position of the third abnormal sound signal (S13).

[0101] Here, the first to third positions may be position values ​​of abnormal acoustic signals in an acoustic image according to beamforming of acoustic data.

[0102] And, the computing device can determine whether the surveillance target object is in an abnormal state by analyzing the frequency characteristics of the beamformed sound values ​​of at least three positions produced by imaging them and using the trained model (S14). Specifically, the computing device can image the spectrogram results of the frequency characteristics of the beamformed sound values ​​of at least three positions produced and input them into the trained model, and the trained model can estimate whether the surveillance target object is in an abnormal state from the spectrogram results. At this time, the trained model may be a neural network model constructed by learning the correlation between the spectrogram results and whether the surveillance target object is in an abnormal state.

[0103] According to the present invention, by determining gas leakage or electric discharge using beamformed sound values ​​of the positions of at least three abnormal acoustic signals, it is possible to easily and accurately locate abnormal situations occurring simultaneously.

[0104] Meanwhile, when the determination of whether the computing device is in an abnormal state is completed, the output unit (130) displays an acoustic-optical combined image (601) in which an acoustic image and an optical image are combined according to beam forming, and the location where an abnormal acoustic signal is detected can be displayed as a visual object (602) in the acoustic-optical combined image (601).

[0105]

[0106] Figures 8 and 9 are drawings for explaining a process of setting a region of interest and controlling a camera according to one embodiment of the present invention.

[0107] Referring to Fig. 8, a QR code (803) may be attached to the object to be monitored. Then, the surveillance robot (100) moves to a surveillance position to monitor the object to be monitored, and the image data generation unit (110) of the surveillance robot (100) may generate image data corresponding to an optical signal generated from the object to be monitored.

[0108] In this case, as shown in FIG. 9, the computing device of the surveillance robot (100) acquires image data (S21), and when a QR code is recognized from the image data at the surveillance location of the surveillance robot (100), the computing device can set a region of interest for securing audio data and / or image data based on the QR code (S22). Here, the region of interest has a rectangular shape, and the computing device can set the region of interest so that the QR code is located at the center and one of the corners of the rectangular shape.

[0109] Referring to FIG. 8, the view area of ​​the sound camera and the video camera acquired after the surveillance robot (100) reaches the initial surveillance position may be a first area (810) of a rectangular image, and the computing device may set the area of ​​interest (802) so that the QR code is located at the left corner of the rectangular shape.

[0110] And, the computing device can control at least one of panning and tilting of the camera so that sound data and / or image data are acquired in the set region of interest (802) (S23).

[0111] According to the present invention, by setting an area of ​​interest for collecting audio data and / or image data using a QR code, the problem of the view area changing each time the surveillance robot (100) is positioned at a surveillance location can be solved.

[0112]

[0113] FIGS. 10 to 11 are diagrams for explaining a process of determining whether a surveillance target object is in an abnormal state by combining image data-based object recognition and audio data according to one embodiment of the present invention.

[0114] Referring to FIG. 10, a surveillance robot (100) moves to a surveillance position to monitor a surveillance target object, an audio data generation unit (110) of the surveillance robot (100) generates audio data corresponding to an audio signal generated from the surveillance target object, and an image data generation unit (110) of the surveillance robot (100) can generate image data corresponding to an optical signal generated from the surveillance target object.

[0115] That is, the computing device can obtain audio data and image data (S32).

[0116] Meanwhile, the steps (S300, S700) for determining whether the above-described drawings 4 to 5 are in an abnormal state may include the following steps S33 to S38.

[0117] Specifically, if the target object recognized from the image data is likely to be in an abnormal state (S33: YES) and audio data is acquired from the object (S34: YES), the computing device can determine that the monitored object is in an abnormal state (S35). For example, if a gauge with a high possibility of a gas leak is recognized among various monitored objects included in the image data and audio data is acquired from the gauge, the computing device can determine that a gas leak has occurred at the gauge.

[0118] In addition, if the target object recognized from the image data is an object with a low possibility of being in an abnormal state (S33:NO) and acoustic data is acquired from the object (S36:YES), the type of acoustic data can be estimated using the learned model to determine whether the target object is in an abnormal state (S37). For example, if a flange with a low possibility of gas leakage is recognized among various surveillance target objects included in the image data and acoustic data is acquired from the flange, the computing device can input the acoustic data acquired from the flange into the learned model to estimate whether the acoustic data is due to a gas leak.

[0119] Additionally, if the target object recognized from the image data is an object with a low probability of being in an abnormal state (S33:NO) and no sound data is acquired from the object (S36:NO), the computing device can determine that the target object is in a normal state (S38).

[0120] According to the present invention, by determining an abnormal state by setting a high weight for an object with a high possibility of an abnormal state such as a gas leak, such as a gage, among various objects to be monitored included in image data, the accuracy of determining an abnormal state can be increased.

[0121]

[0122] Figures 12 and 13 are diagrams illustrating a process for detecting abnormal areas through full-area scanning of an acoustic image according to one embodiment of the present invention. Referring to Figure 12, the surveillance robot (100) can move to a surveillance location where data needs to be obtained from the same location in an environment with multiple facilities. For example, the full surveillance location for monitoring the entire space of the environment may be a point along a path that can monitor multiple facilities simultaneously.

[0123] In addition, the sound data generation unit (110) of the surveillance robot (100) can generate sound data corresponding to sound signals occurring in the entire space of the environment.

[0124] In this case, the computing device of the surveillance robot (100) can acquire acoustic data, generate an acoustic image through beamforming of the acoustic data, and store the acoustic image in units of time (S41).

[0125] In addition, the computing device of the surveillance robot (100) can determine the progression and occurrence of an abnormal acoustic signal through a cumulative comparison of previously stored acoustic images. Specifically, the processor can align a plurality of acoustic images through matching of the plurality of acoustic images (S42).

[0126] In addition, the computing device of the surveillance robot (100) can detect at least one of an area where an acoustic signal has transitioned from non-occurrence to occurrence, and an area where the change in the sound value of the signal has exceeded a preset value, through comparison of a plurality of aligned acoustic images (S42). For example, the computing device of the surveillance robot (100) can divide the entire area of ​​an acoustic image into block units and perform the detection through comparison of each block of a plurality of acoustic images.

[0127] According to the present invention, by acquiring acoustic data for the entire space of an environment at a predetermined period and accumulating and comparing the acquired acoustic data for the entire space, an abnormal state for the entire space of the environment can be determined.

[0128]

[0129] Hereinafter, a specific hardware implementation of a computing device according to the present embodiment will be described with reference to FIG. 14.

[0130] Figure 14 is an exemplary diagram showing the configuration of a computing device according to one embodiment of the present invention.

[0131] At least one of each module constituting the computing device is implemented on a general-purpose computing processor and thus may include a processor (1008), an input / output (I / O) device (1002), a memory (1004), an interface (1006), and a bus (1014). The processor (1008), the input / output device (1002), the memory (1004), and / or the interface (1006) may be coupled to each other via the bus (1014). The bus (1014) corresponds to a path through which data is moved.

[0132] Specifically, the processor (1008) may include at least one of a Central Processing Unit (CPU), a Micro Processor Unit (MPU), a Micro Controller Unit (MCU), a Graphic Processing Unit (GPU), a microprocessor, a digital signal processor, a microcontroller, an application processor (AP), and logic elements capable of performing functions similar thereto.

[0133] The input / output device (1002) may include at least one of a keypad, a keyboard, a touchscreen, and a display device. The memory (1004) may store data and / or programs, etc.

[0134] The interface (1006) may perform a function of transmitting data to or receiving data from a communication network. The interface (1006) may be wired or wireless. For example, the interface (1006) may include an antenna or a wired / wireless transceiver. The memory (1004) may further include high-speed DRAM and / or SRAM, etc., as a volatile operating memory that enhances the operation of the processor (1008) while protecting personal information.

[0135] Additionally, the memory (1004) stores programming and data configurations that provide the functionality of some or all of the modules described herein. For example, it may include logic for performing selected aspects of a method for environmental monitoring using a surveillance robot.

[0136] In addition, a program or application is loaded as a set of commands including each operation for performing the environmental monitoring method using the above-described surveillance robot stored in the memory (1004), and the processor is enabled to perform each operation.

[0137] For example, the actions may include an action of the surveillance robot moving to a surveillance location for monitoring a surveillance target object located in the environment, an action of the surveillance robot obtaining acoustic data corresponding to an acoustic signal generated from the surveillance target object at the surveillance location, and an action of determining whether the surveillance target object is in an abnormal state based on the obtained acoustic data.

[0138]

[0139] The various embodiments described herein may be implemented in a recording medium readable by a computer or similar device, for example, using software, hardware, or a combination thereof.

[0140] In terms of hardware implementation, the embodiments described herein can be implemented using at least one of ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), processors, controllers, micro-controllers, microprocessors, and other electrical units for performing functions. In some cases, the embodiments described herein can be implemented as a control module itself.

[0141] In a software implementation, the procedures and functions described herein, as well as other embodiments, may be implemented as separate software modules. Each of these software modules may perform one or more of the functions and operations described herein. The software code may be implemented as a software application written in a suitable programming language. The software code may be stored in a memory module and executed by a control module.

[0142] The above description is merely an example of the technical idea of ​​the present invention, and those skilled in the art will appreciate that various modifications, changes, and substitutions can be made without departing from the essential characteristics of the present invention.

[0143] Accordingly, the embodiments disclosed in the present invention and the accompanying drawings are intended to illustrate, rather than limit, the technical concept of the present invention, and the scope of the technical concept of the present invention is not limited by these embodiments and the accompanying drawings. The protection scope of the present invention should be interpreted by the following claims, and all technical concepts within the scope equivalent thereto should be interpreted as being included within the scope of the rights of the present invention.

Claims

1. In a method for environmental monitoring using a surveillance robot, A step of the surveillance robot moving to a surveillance position for monitoring a surveillance target object located in the environment; A step in which the surveillance robot acquires acoustic data corresponding to an acoustic signal generated from the surveillance target object at the surveillance location; and An environmental monitoring method comprising a step of determining whether the object to be monitored is in an abnormal state based on the acquired acoustic data.

2. In paragraph 1, The abnormal state of the above surveillance target object is: An environmental monitoring method characterized by including at least one of gas leakage and electric discharge.

3. In paragraph 1, The above judging step is, A step of calculating at least three locations of abnormal sound signals in order of size from the above sound data; and An environmental monitoring method characterized by comprising a step of analyzing beamformed sound values ​​of at least three positions produced above to determine whether an abnormal state exists.

4. In paragraph 3, The step of calculating at least three locations of the above abnormal sound signals in order of size is: A step of searching for the position of a first abnormal acoustic signal from the above acoustic data and calculating the first position of the first abnormal acoustic signal; A step of deleting the first abnormal sound signal from the above sound data through phase and amplitude compensation, and then searching for the position of the second abnormal sound signal from the above sound data to derive the second position of the second abnormal sound signal; and An environmental monitoring method characterized by comprising: a step of deleting the second abnormal acoustic signal from the acoustic data through phase and amplitude compensation, and then searching for the position of the third abnormal acoustic signal from the acoustic data to derive the third position of the third abnormal acoustic signal.

5. In paragraph 1, A QR code is attached to an area of ​​the above surveillance target object, The above surveillance robot acquires image data corresponding to an optical signal generated from the surveillance target object at the surveillance location; and An environmental monitoring method further comprising: a step of setting an area of ​​interest for securing at least one of the sound data and the image data based on the QR code when the QR code is recognized at the surveillance location from the image data; 6. In paragraph 5, The steps for setting the above area of ​​interest are: An environmental monitoring method characterized in that the region of interest is a rectangular shape, and the region of interest is set such that the QR code is located at the center and one of the corners of the rectangular shape.

7. In paragraph 6, An environmental monitoring method, further comprising: a step of controlling at least one of panning and tilting of a camera so that at least one of the sound data and the image data is acquired in the set region of interest.

8. In paragraph 1, The above surveillance robot further includes a step of acquiring image data corresponding to an optical signal generated from the target object at the surveillance location; The step of determining whether the above abnormal condition is present is: A step of determining that the target object recognized from the image data is an object with a high possibility of being in an abnormal state, and when sound data is acquired from the object, the step of determining that the target object is in an abnormal state; and An environmental monitoring method characterized by comprising: a step of determining whether the target object is in an abnormal state by estimating the type of the acoustic data using a learned model when the target object recognized from the image data is an object with a low probability of being in an abnormal state and acoustic data is acquired from the object; 9. In paragraph 1, A step of storing an acoustic image generated based on acoustic data for the entire space of the above environment; and An environmental monitoring method further comprising a step of determining whether an abnormal acoustic signal is progressing or occurring through a cumulative comparison of the above-mentioned stored acoustic images.

10. In paragraph 9, The step of determining whether the above abnormal sound signal has occurred is as follows: A step of aligning a plurality of sound images by matching the plurality of sound images; and An environmental monitoring method characterized by comprising a step of detecting at least one of an area where an acoustic signal has transitioned from non-occurrence to occurrence and an area where a change in the sound value of the signal has exceeded a preset value through comparison of the plurality of aligned acoustic images.

11. In paragraph 1, An environmental monitoring method further comprising a step of uploading result data according to the above judgment to a server.

12. In paragraph 1, The above moving steps are: An environmental monitoring method characterized in that the above surveillance robot autonomously drives along a preset driving path to which the surveillance position is assigned.

13. In an environmental monitoring system using a surveillance robot, A surveillance robot that moves to a surveillance position to monitor a surveillance target object located in an environment and acquires acoustic data corresponding to an acoustic signal generated from the surveillance target object at the surveillance position; An environmental monitoring system characterized in that the acquired acoustic data is used to determine whether the object to be monitored is in an abnormal state.

14. A computer program stored on a computer-readable recording medium including a program code for executing an environmental monitoring method described in any one of paragraphs 1 to 12.

15. A computer-readable recording medium having recorded thereon a program for executing an environmental monitoring method described in at least one of paragraphs 1 to 12.

Images