Apparatus, method, and system for detecting and responding to abnormal situations based on multimodal information

Abstract

A multimodal information-based abnormal situation detection and response device, method, and system are disclosed. A multimodal information-based abnormal situation detection and response device according to one embodiment of the present invention comprises: a multimodal information acquisition unit that acquires image information and sound information from a monitored space; an image analysis unit that calculates a reliability score related to an abnormal situation based on the image information acquired from the multimodal information acquisition unit; a first voice analysis unit that classifies whether a voice is abnormal based on the sound information acquired from the multimodal information acquisition unit; a second voice analysis unit that performs voice recognition and a preset keyword detection on the sound information only when the sound information satisfies a preset condition; a determination unit that determines a risk level by synthesizing the reliability score related to the abnormal situation, the magnitude value of the sound information, the classification result of the first voice analysis unit, and the keyword detection result of the second voice analysis unit; and a response unit that performs at least one response operation among alarm output, lighting control, and notification transmission according to the determination result of the determination unit.

Description

Technology Field

[0001] The present invention relates to multimodal information processing technology, and more specifically, to an apparatus, method, and system that analyzes image information, acoustic information, and infrared-based image information acquired from a monitored space to detect dangerous situations and performs responses such as alarms, lighting control, or notification transmission according to the detected level of danger. In particular, the present invention relates to a technology that can improve the accuracy of detecting dangerous situations and suppress unnecessary alarms caused by false positives by comprehensively determining information from multiple channels in an edge artificial intelligence environment. Background Technology

[0002] Recently, the importance of surveillance technology for rapidly detecting and responding to dangerous situations—such as violence, physical altercations, falls, intrusion, and disturbances—in common spaces like apartment buildings, public housing facilities, hallways, stairwells, and parking lots has been increasing. In particular, in public residential environments, there is a growing demand for intelligent surveillance technology capable of early detection of dangerous situations and immediate on-site response to protect the physical safety and psychological well-being of residents.

[0003] Conventional surveillance technology has primarily evolved into video-based surveillance using cameras or audio-based surveillance using microphones. While video-based surveillance technology can detect situations such as violent acts, falls, and loitering through object detection, person recognition, posture analysis, or behavior classification, relying solely on a single video channel presents a problem in that normal behaviors—such as play, exercise, simple conversation, and daily movement—are mistaken for dangerous situations.

[0004] Meanwhile, acoustic-based surveillance technology can identify dangerous situations secondarily by detecting noise levels, screams, shouting, glass breaking sounds, or specific acoustic events. However, relying solely on acoustic information makes it difficult to accurately determine the location of dangerous behavior, the context of the situation, or the movement status of individuals at the actual site, and false positives may occur due to surrounding environmental noise or echoes.

[0005] Meanwhile, since the accuracy of visible light-based video surveillance can rapidly deteriorate in nighttime or low-light environments, a means to continuously secure visual information is required during both day and night. Accordingly, it is necessary to maintain video channels even in low-light environments by utilizing the infrared night mode of a dual-purpose camera, thereby ensuring the continuity of risk assessment. However, conventionally, technology to organically combine such infrared-based video information, acoustic information, and conditional voice recognition results within a single edge device to assess risk levels and respond has not been sufficiently established.

[0006] Therefore, there is a need for technology that enables real-time analysis of multimodal information, including video, audio, and infrared-based visual information, at edge devices, thereby more accurately detecting dangerous situations while reducing false alarms and allowing for immediate response. Prior art literature

[0007] Korean Registered Patent No. 10-2936330 (March 4, 2026) "Multimodal AI Dangerous Situation Detection System and Method" Korean Registered Patent No. 10-2647328 (March 8, 2024) "Edge-type Surveillance Camera AI Anomaly Detection Control Device and Method" The problem to be solved

[0008] The multimodal information-based abnormal situation detection and response device, method, and system according to one embodiment of the present invention aims to solve the problem of false alarms occurring in existing detection methods relying on a single modality and to provide a technology capable of rapid and accurate response to actual dangerous situations.

[0009] In addition, the purpose is to solve the problem of excessive consumption of computational resources and reduced real-time performance in edge device environments when high-load computations such as speech recognition are performed constantly.

[0010] In addition, it aims to solve the problem where it is difficult to distinguish between everyday situations with similar human behavior and actual dangerous situations using only video-based analysis, and the problem where it is difficult to clearly distinguish between noise and abnormal speech using only acoustic-based analysis.

[0011] In addition, the purpose is to solve the problem of reduced processing efficiency that occurs when the difference in computational characteristics between the neural network processing unit and the central processing unit is not considered when performing AI-based video and audio analysis, and to provide a structure that can efficiently utilize various computational resources.

[0012] Furthermore, the purpose is to suppress unnecessary alarms while maintaining a response to actual dangerous situations by synthesizing multiple analysis results to determine the risk level in multiple stages, rather than judging detection results based on a single standard.

[0013] The problems of the present invention are not limited to the technical problems mentioned above, and other unmentioned technical problems will be clearly understood by those skilled in the art from the description below. means of solving the problem

[0014] A multimodal information-based abnormal situation detection and response device according to one embodiment of the present invention comprises: a multimodal information acquisition unit that acquires image information and sound information from a monitored space; an image analysis unit that calculates a reliability score related to an abnormal situation based on the image information acquired from the multimodal information acquisition unit; a first voice analysis unit that classifies whether a voice is abnormal based on the sound information acquired from the multimodal information acquisition unit; a second voice analysis unit that performs voice recognition and a preset keyword detection on the sound information only when the sound information satisfies a preset condition; a determination unit that determines a risk level by synthesizing the reliability score related to the abnormal situation, the magnitude value of the sound information, the classification result of the first voice analysis unit, and the keyword detection result of the second voice analysis unit; and a response unit that performs at least one response operation among alarm output, lighting control, and notification transmission according to the determination result of the determination unit.

[0015] In an embodiment, the image analysis unit is placed in a first core group of a neural network processing unit (NPU), the first voice analysis unit is placed in a second core group of the neural network processing unit, and the second voice analysis unit and the judgment unit may be placed in a central processing unit (CPU).

[0016] In an embodiment, artificial intelligence models of different modalities may be deployed for division of labor so that the first core group performs detection of object or behavior-related features regarding the image information, and the second core group performs classification of abnormal speech regarding the acoustic information.

[0017] In an embodiment, the second voice analysis unit can perform voice recognition only when the magnitude value of the acoustic information is greater than or equal to a preset first threshold value.

[0018] In an embodiment, the first voice analysis unit can calculate a probability distribution for any one of the abnormal voice, normal voice, and noise based on acoustic features extracted from the acoustic information.

[0019] In an embodiment, the determination unit may determine a first risk level if the reliability score related to the abnormal situation is greater than or equal to a second threshold value, and at least one of the magnitude value of the acoustic information, the classification result of the first voice analysis unit, and the keyword detection result of the second voice analysis unit satisfies a risk condition.

[0020] In an embodiment, the determination unit determines a monitoring state indicating a suspected danger state without generating an external alarm when the reliability score related to the abnormal situation is greater than or equal to the second threshold value, but the magnitude value of the acoustic information, the classification result of the first voice analysis unit, and the keyword detection result of the second voice analysis unit correspond to normal conditions; and determines a caution state when the reliability score related to the abnormal situation is less than the second threshold value and the magnitude value of the acoustic information or the classification result of the first voice analysis unit satisfies danger conditions.

[0021] A multimodal information-based abnormal situation detection and response method according to an embodiment of the present invention is an abnormal situation detection and response method performed by a multimodal information-based abnormal situation detection and response device, comprising: a step of acquiring multimodal information including image information and sound information from a monitored space through a multimodal information acquisition unit; a step of an image analysis unit in a first core group of a neural network processing unit calculating an abnormal situation-related reliability score based on the image information; a step of a first voice analysis unit in a second core group of the neural network processing unit classifying whether a voice is abnormal based on the sound information; a step of a second voice analysis unit in a central processing unit performing voice recognition and preset keyword detection for the sound information only when the sound information satisfies preset conditions; a step of a judgment unit in the central processing unit determining a risk level by synthesizing the abnormal situation-related reliability score, the magnitude value of the sound information, the classification result of the first voice analysis unit, and the keyword detection result of the second voice analysis unit; and a step of a response unit performing at least one response operation among alarm output, lighting control, and notification transmission according to the risk level.

[0022] A multimodal information-based abnormal situation detection and response system according to one embodiment of the present invention comprises: a multimodal information-based abnormal situation detection and response device according to any one of claims 1 to 7; a notification receiving unit that receives a remote notification according to the judgment result of the multimodal information-based abnormal situation detection and response device; and a dashboard server that displays the judgment result or an event log. Effects of the invention

[0023] According to embodiments of the present invention, at least the following effects can be provided.

[0024] According to a multimodal information-based abnormal situation detection and response device, method, and system according to one embodiment of the present invention, since a multimodal-based judgment combining image analysis results and acoustic analysis results is possible, the accuracy of abnormal situation detection can be improved and false alarms can be effectively reduced compared to cases relying on a single modality.

[0025] In addition, by distinguishing between a first voice analysis unit and a second voice analysis unit for the acoustic channel and performing a first analysis based on acoustic patterns and a second analysis based on semantics in stages, it is possible to reflect not only acoustic anomalies such as screaming and shouting but also semantic danger signs such as profanity, threatening utterances, and requests for rescue. Accordingly, the judgment unit can perform more precise complex judgments and increase the reliability of detection regarding actual danger situations.

[0026] In addition, by applying a structure that performs speech recognition and keyword detection only when preset conditions are met, high-load speech recognition operations can be selectively performed, enabling efficient use of computational resources and maintaining real-time processing performance even in edge device environments.

[0027] In addition, by configuring image analysis and first voice analysis to be distributed across multiple cores of the neural network processing unit, and second voice analysis and judgment to be performed by the central processing unit, fixed-shape-based operations and variable-length-based operations can be appropriately separated. Accordingly, work distribution suitable for the characteristics of each computational resource is achieved, and the processing efficiency and stability of the entire system can be improved.

[0028] The effects according to the present invention are not limited to those exemplified above, and various effects may be derived from the description in this specification. Brief explanation of the drawing

[0029] FIG. 1 is a schematic diagram of a multimodal information-based abnormal situation detection and response system according to an embodiment of the present invention. FIG. 2 is a schematic diagram of a multimodal information-based abnormal situation detection and response device according to an embodiment of the present invention. FIG. 3 is a diagram schematically illustrating the division of labor structure between a neural network processing unit and a central processing unit in a multimodal information-based anomaly detection and response device according to an embodiment of the present invention. FIG. 4 is a schematic diagram illustrating the input value and risk level determination result of a determination unit according to an embodiment of the present invention. FIG. 5 is a schematic flowchart of a multimodal information-based abnormal situation detection and response method according to one embodiment of the present invention. Specific details for implementing the invention

[0030] The present invention is capable of various modifications and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. The effects and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the drawings. However, the present invention is not limited to the embodiments disclosed below but can be implemented in various forms and should be understood to include all modifications, equivalents, and substitutions that fall within the spirit and scope of the present invention.

[0031] Prior to the explanation, the terms used in the detailed description are explained. In the following embodiments, terms such as "first," "second," etc., are used not in a limiting sense, but for the purpose of distinguishing one component from another. Therefore, it is understood that the first component mentioned below may be the second component within the technical scope of the present invention. Furthermore, singular expressions include plural expressions unless the context clearly indicates otherwise. Additionally, terms such as "include" or "have" mean that the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification exist, and do not preclude the possibility that one or more other features or components may be added.

[0032] Additionally, the size of components in the drawings may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are depicted arbitrarily for convenience of explanation, so the present invention is not necessarily limited to what is illustrated.

[0033] Hereinafter, an embodiment according to the present invention will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, identical or corresponding components are given the same reference numerals, and redundant descriptions thereof are omitted.

[0034] A multimodal information-based abnormal situation detection and response device, method, and system according to one embodiment of the present invention detects an abnormal situation based on multimodal information including image information, sound information, and infrared-based visual information, and performs a response operation according to the detection result. In particular, by utilizing multiple computational resources in a division of labor within an edge device to perform image analysis, voice pattern analysis, conditional voice recognition, and complex judgment, the device, method, and system enable rapid response while reducing false alarms.

[0035] In conventional technology, an edge-server structure was common in which integrated judgment of video and audio analysis results or high-load computations were performed on a server. In contrast, one embodiment of the present invention is configured to perform video analysis, voice analysis, and risk level determination within an edge device comprising a neural network processing unit and a central processing unit, thereby enabling independent detection and response to abnormal situations without relying on an external server.

[0036] In an embodiment of the present invention, the multimodal information-based abnormal situation detection and response device is identically included in the system, and the device is described together with the system.

[0037] Referring to FIG. 1, a multimodal information-based abnormal situation detection and response system (1000) according to one embodiment of the present invention includes a multimodal information-based abnormal situation detection and response device (100), a control server (200), an administrator terminal (300), an event log storage unit (400), and an alarm output unit (500). In one embodiment, the multimodal information-based abnormal situation detection and response system (1000) may be configured to perform abnormal situation detection for a plurality of monitored spaces based on multimodal information provided from a plurality of multimodal information acquisition units (110).

[0038] Referring to FIG. 2, the multimodal information-based abnormal situation detection and response device (100) includes a multimodal information acquisition unit (110), an image analysis unit (120), a first voice analysis unit (130), a second voice analysis unit (140), a judgment unit (150), and a response unit (160). The multimodal information-based abnormal situation detection and response device (100) may be in the form of an edge device.

[0039] The multimodal information acquisition unit (110) acquires video information and sound information from a monitored space and provides it to the composition of the video analysis unit (120), the first voice analysis unit (130), the second voice analysis unit (140), etc. for subsequent analysis, thereby forming the input terminal of the multimodal information-based abnormal situation detection and response device (100).

[0040] The multimodal information acquisition unit (110) may include a video acquisition means for acquiring video information and a sound acquisition means for acquiring sound information, and such video acquisition means and sound acquisition means may be integrated into a single hardware device or configured as multiple separate devices. For example, the video acquisition means and sound acquisition means may be a device in which a camera and a microphone are formed as a single unit, or a camera and a microphone may be installed as separate devices and connected to the multimodal information-based abnormal situation detection and response device (100) via wired or wireless communication.

[0041] The image acquisition means acquires images of the monitored space in frame units, and the acquired images can be converted into digital image data having a predetermined resolution and frame rate and transmitted to the image analysis unit (120). The image acquisition means may include an infrared-based image acquisition function for monitoring in low-light environments as well as images in the visible light range. For example, the image acquisition means may be configured as a day / night dual-purpose camera including an infrared illumination device in the near-infrared wavelength range including an 850nm near-infrared LED, and may automatically switch to an infrared-based image acquisition mode when the ambient illumination decreases below a certain level. By including an infrared-based image acquisition function in this way, visual information regarding the monitored space can be continuously secured not only in daytime environments but also in nighttime or low-light environments.

[0042] The sound acquisition means can collect sound generated in the monitored space and convert it into a digital sound signal. The sound acquisition means may include a microphone, and the collected sound signal may be digitized at a constant sampling frequency and transmitted to the first voice analysis unit (130) and the second voice analysis unit (140). The sound acquisition means may be configured to have appropriate sensitivity and dynamic range so as to maintain the characteristics of screams, shouts, or other abnormal sounds even in situations where environmental noise is present.

[0043] In the specification, "abnormal sound" means abnormal sound or speech related to an abnormal situation in the monitored space, and may include, for example, speech including screaming, shouting, or profanity, speech requesting rescue, threatening speech, or abnormal sound equivalent thereto.

[0044] The multimodal information acquisition unit (110) can be configured to acquire video information and audio information in a temporally related form. For example, video frames and audio signals can be synchronized to correspond to the same time interval, thereby allowing the results produced by the video analysis unit (120) and the voice analysis unit, respectively, to reflect the same situation.

[0045] Additionally, the multimodal information acquisition unit (110) may provide basic data for calculating the magnitude value of the acoustic information. The magnitude value of the acoustic information may be generated by a central processing unit or a low-power core group performing RMS calculation on the PCM acoustic signal provided from the multimodal information acquisition unit. Since the magnitude value calculated from the acoustic signal is used as a condition for determining whether to perform voice recognition in the second voice analysis unit (140), reliably acquiring the acoustic signal in the multimodal information acquisition unit (110) prevents unnecessary voice recognition and enables the efficient use of computational resources. In particular, by performing voice recognition only when an acoustic level above a certain threshold is detected, it is not necessary to perform high-load voice recognition computations at all times, thereby reducing resource consumption of the edge device and ensuring real-time performance.

[0046] The image analysis unit (120) is configured to calculate a reliability score related to an abnormal situation based on image information provided from the multimodal information acquisition unit (110), and is responsible for making an artificial intelligence-based judgment on the visual channel in the multimodal information-based abnormal situation detection and response device (100). The image analysis unit (120) can detect or extract features related to an abnormal situation occurring in the monitored space, and generate a reliability score related to the abnormal situation based on the results and provide it to the judgment unit (150). In the specification, "abnormal situation" may include objects or actions related to violence, physical fighting, falling, wandering, etc.

[0047] The image analysis unit (120) may be implemented on an edge device, in which case the edge device may be a hardware platform that implements a multimodal information-based anomaly detection and response device (100). For example, the edge device may be a single-board computing device or an embedded system including a neural network processing unit (NPU) and a central processing unit (CPU), and with reference to FIG. 3, the image analysis unit (120) may be deployed and operated in a first core group of a neural network processing unit inside such an edge device. In this specification, 'deployment' means that a specific function is configured or assigned to be performed on a specific computational resource. That is, it means that a specific function is configured or assigned to be performed on a specific computational resource, such as a core group of a neural network processing unit or a central processing unit.

[0048] In an embodiment, the neural network processing unit may be an NPU comprising three or more independent cores. Additionally, the edge device may be a system-on-chip comprising an NPU comprising three or more independent cores and a CPU comprising eight or more cores. As an example, the edge device may include a system-on-chip having an RK3588S2 or a similar structure. In one embodiment, the neural network processing unit may be an NPU having a structure that supports dynamic operators or a structure capable of executing a mixture of Transformer-family models and CNN-family models. Additionally, the edge device is not limited to a specific processor architecture and may be implemented on an edge platform adopting various processor architectures.

[0049] By placing the image analysis unit (120) in the first core group of the neural network processing unit, the neural network processing unit, rather than the central processing unit, can handle image inference with a large amount of computation, and the central processing unit can allocate computational resources to other functions or control functions. Accordingly, image processing, voice processing, and judgment processing can be performed in parallel or asynchronously within the edge device, thereby improving the real-time capability of the entire device. In addition, by allowing the neural network processing unit to handle image inference with a large amount of computation and enabling the central processing unit to allocate computational resources to other functions or control functions described later, it has the effect of distributing the computational load of the entire multimodal information-based abnormal situation detection and response device (100).

[0050] The fact that the image analysis unit (120) is placed in the first core group of the neural network processing unit is not merely a choice of placement. Since image information is continuously input in frame units and detection of object or behavior-related features and reliability calculation must be performed repeatedly for each frame, a relatively large amount of computation and continuous inference processing are required. By having image inference performed in a neural network processing unit specialized for artificial intelligence computation, computational efficiency can be increased compared to the case where the central processing unit directly handles image inference, and a division of labor structure with other components can be formed more clearly. In particular, when the multimodal information-based anomaly detection and response device (100) is operated within a single edge device, the configuration of placing the image analysis unit (120) in the neural network processing unit is advantageous for enabling AI-based anomaly detection to be performed independently on-site.

[0051] The image analysis unit (120) can detect object or behavior-related features from image information using an artificial intelligence model. In another embodiment, at least some of the functions of the image analysis unit (120) and the first voice analysis unit (130) may be performed by a multimodal integrated artificial intelligence model that processes image information and sound information in an integrated manner. For example, the image analysis unit (120) can detect features related to people, contact between people, changes in posture, movement paths, fallen state, wandering state, or threat situations using a detection-type artificial intelligence model. In one embodiment, the image analysis unit (120) may use an RT-DETR series model, an RT-DETR R50 model, an RT-DETRv2 series model, or an object detection-based artificial intelligence model that performs substantially the same functions. Such artificial intelligence models may be configured to detect object or behavior-related features such as people within the monitored space, contact between people, sudden changes in posture, fallen state, prolonged stay, or repeated movement.

[0052] The artificial intelligence model of the video analysis unit (120) can perform classification or detection of multiple classes, and in one embodiment, may include classes such as Violence, Non-Violence, Falling, and Loitering. The Violence class may represent a state related to violence, physical struggle, or threatening contact; the Non-Violence class may represent a normal state or a non-dangerous state; the Falling class may represent a state related to falling or sudden collapse of posture; and the Loitering class may represent a state related to loitering or prolonged stay. The class configuration is not limited thereto and may vary depending on the installation location, surveillance purpose, or training data configuration.

[0053] The artificial intelligence model used by the video analysis unit (120) can be generated through pre-training, additional training, or fine-tuning. For example, the artificial intelligence model may use weights pre-trained with a general object recognition dataset, such as the COCO 2017 dataset, as initial values. Subsequently, it may be further trained or fine-tuned using a video dataset specialized for abnormal situations that may occur in public residential common spaces. In one embodiment, the artificial intelligence model of the video analysis unit (120) may be trained using frame images extracted from the Smart-City CCTV Violence Detection dataset and the Real-Life Violence Situations dataset. When the artificial intelligence model is configured to possess both general object recognition capabilities and field-specific abnormal situation recognition capabilities, it can calculate an abnormal situation reliability score that is more suitable for the actual environment of the monitored space. Accordingly, the video analysis unit (120) can have recognition performance specialized for abnormal situations such as violence, falling, and loitering, in addition to general object detection performance, and can generate an abnormal situation reliability score suitable for the installation environment more precisely.

[0054] The image analysis unit (120) can convert the original image frame provided by the multimodal information acquisition unit (110) into an input format suitable for artificial intelligence inference. For example, the original image frame may be an RGB image with a resolution of 1920×1080 or an infrared-based grayscale image, and the image analysis unit (120) can generate an input tensor of a predetermined format by performing resizing, normalization, and channel alignment on the frame. In one embodiment, the input tensor may be in the format uint8[1,640,640,3]. When an infrared-based grayscale image is input, the grayscale frame may be expanded into a 3-channel format or converted to match the model input shape and applied to the same or similar artificial intelligence model. Through such input shape normalization, the neural network processing unit can perform stable and high-speed inference based on a fixed tensor shape. In one embodiment, the image information provided to the image analysis unit (120) may further undergo artificial intelligence-based image preprocessing performed by a neural network processing unit or a processing unit associated therewith to improve the quality of low-light environments or infrared-based images.

[0055] In addition, when the multimodal information acquisition unit (110) provides infrared-based visual information, the image analysis unit (120) can calculate an abnormal situation reliability score using the same artificial intelligence model or a model equivalent thereto for both visible light images and infrared-based images. That is, an infrared-based image acquired in night mode of a day-night dual-purpose camera can also be input into the image analysis unit (120) and directly used to calculate an abnormal situation reliability score. Accordingly, the image channel is not interrupted even in an environment close to complete darkness, and the abnormal situation detection function can be maintained even in night or low-light environments.

[0056] The image analysis unit (120) can output object location information and class-specific reliability information as inference results of an artificial intelligence model. For example, the output of the image analysis unit (120) may consist of an array containing bounding box coordinates for a number of detected objects N, and an array containing multiple class reliability scores for N objects. In one embodiment, the bounding box coordinates may be in the format float32[N,4] including center coordinates and size, and the class-specific reliability information may be in the format float32[N,C], where C is the number of classes including Violence, NonViolence, Falling, and Loitering. The image analysis unit (120) can post-process these inference results to generate reliability scores related to abnormal situations.

[0057] The reliability score related to anomalies can be calculated based on reliability values ​​for one or more classes. The reliability information for each class may be reliability values ​​calculated by an artificial intelligence model for each of multiple classes, and the reliability score related to anomalies may be a representative judgment value calculated by selecting at least one of the reliability information for each class or by combining multiple reliability information for each class. In one embodiment, the reliability score related to anomalies may be the maximum reliability value for the Violence class. For example, if multiple objects are detected, the maximum value among the Violence class reliability values ​​for each object may be set as the reliability score related to anomalies. In another embodiment, the reliability score related to anomalies may be calculated by considering not only the reliability value of the Violence class but also the reliability values ​​of the Falling class or Loitering class. For example, in an installation environment where fall detection is important, the reliability value of the Falling class may be used as the reliability score related to anomalies, either alone or in a weighted combination. In addition, anomaly-related confidence scores can be calculated using the maximum value, weighted sum, average value, or rule-based post-processing results of confidence values ​​for multiple classes.

[0058] The method of calculating the reliability score related to abnormal situations can be adjusted according to the monitoring purpose and environment. For example, in places where the detection of violence or falling is important, such as hallways or stairwells in apartment buildings, greater weight can be given to the Violence class and Falling class, and in places where the detection of loitering or intrusion is important, such as parking lots or outer passages, greater weight can be given to the Loitering class or intrusion-related class. Accordingly, the video analysis unit (120) can calculate the reliability score related to abnormal situations based on abnormal situation judgment criteria suitable for the installation environment, rather than simply using a specific class score as is, and can contribute to reducing false alarms and improving detection accuracy for each site.

[0059] The artificial intelligence model of the video analysis unit (120) can be retrained or updated to suit the characteristics of the space being monitored. For example, since the patterns of normal behavior and abnormal situations may differ depending on the installation location, such as a hallway in an apartment building, a stairwell, a parking lot, or an area adjacent to a playground, the artificial intelligence model of the video analysis unit (120) can be further trained or fine-tuned based on field data. Accordingly, it becomes possible to generate a reliability score related to abnormal situations that reflects the unique lighting, camera angle, population density, and movement patterns of the site, which can lead to a reduction in false positive rates and an improvement in detection performance. These effects are achieved by a configuration in which the video analysis unit (120) uses a learnable artificial intelligence model rather than a simple rule-based processing unit.

[0060] The artificial intelligence model of the image analysis unit (120) can be quantized and executed in a neural network processing unit. For example, the completed model can be converted into ONNX format and then converted again into an inference format suitable for the neural network processing unit. In one embodiment, the RT-DETR-based model can be quantized into INT8 and executed in a neural network processing unit. By using such a quantized artificial intelligence model, the memory usage and inference time required by the edge device can be reduced, and the image analysis unit (120) can perform continuous frame inference more stably. In addition, by executing the quantized image artificial intelligence model in the first core group of the neural network processing unit, other operations can be distributed to other core groups or the central processing unit, thereby enabling a division of labor structure between the neural network processing unit and the central processing unit to be implemented more effectively.

[0061] By operating the image analysis unit (120) in the first core group of the neural network processing unit, artificial intelligence inference can be performed in parallel with the other core group used by the first voice analysis unit (130). Accordingly, image analysis and acoustic-based abnormal voice classification can be performed simultaneously or asynchronously in different core groups, and the central processing unit can focus on conditional voice recognition and final judgment. Due to this division of computational resources structure, the image analysis unit (120) can reduce bottlenecks in the entire device while maintaining continuous frame inference, and consequently, the responsiveness and stability of abnormal situation detection can be improved. In particular, by the image analysis unit (120) continuously calculating reliability scores related to abnormal situations and transmitting them to the judgment unit (150) along with the results of the first voice analysis unit (130) and the second voice analysis unit (140), it becomes possible to respond quickly to signs of danger while reducing false alarms caused by image-only judgment.

[0062] The first voice analysis unit (130) is configured to classify whether a voice is abnormal based on acoustic information provided from the multimodal information acquisition unit (110), and is responsible for the primary artificial intelligence-based judgment on the acoustic channel in the multimodal information-based abnormal situation detection and response device (100).

[0063] The first voice analysis unit (130) can detect characteristics of abnormal voices that are distinguishable from screams, shouts, normal voices, or background noise from sounds occurring in the monitored space, and provide the results to the judgment unit (150). The first voice analysis unit (130) can also be implemented on an edge device. The first voice analysis unit (130) can be deployed and operated in the second core group of a neural network processing unit inside such an edge device. By deploying the first voice analysis unit (130) in the second core group of the neural network processing unit, the neural network processing unit, rather than the central processing unit, can perform artificial intelligence inference regarding sound features, and the central processing unit can allocate computational resources to other functions such as calculating sound magnitude values, voice recognition control, or final judgment, which will be described later. Accordingly, video analysis and sound-based abnormal voice classification can be performed in parallel in different core groups, and the real-time capability of the entire device can be improved.

[0064] The first voice analysis unit (130) can distinguish between abnormal voices, normal voices, or background noise from acoustic information using an artificial intelligence model. In one embodiment, the first voice analysis unit (130) may use a lightweight artificial intelligence model based on acoustic features, for example, a convolutional neural network structure. As a specific example, it may have a structure including Conv2D, batch normalization, pooling, global average pooling, a fully connected layer, and a softmax layer. The artificial intelligence model used by the first voice analysis unit (130) may be configured to classify whether a voice is abnormal by reflecting both the time-axis change and the frequency-axis characteristics of the acoustic information. For example, the artificial intelligence model may learn features such as a sudden increase in energy associated with screaming or shouting, concentration in specific frequency bands, and spectral changes accompanying emotional vocalization, thereby detecting abnormal voice patterns that are distinguishable from normal conversation, laughter, breathing sounds, footsteps, or general environmental noise.

[0065] The artificial intelligence model used by the first voice analysis unit (130) can be trained based on a pre-configured acoustic dataset. In one embodiment, the artificial intelligence model can be generated based on training data reconstructed by combining an environmental sound dataset and an emotional voice dataset. For example, samples including crying, footsteps, laughter, wind, rain, engine sounds, train sounds, etc., can be acquired from the environmental sound dataset, and samples including angry or fearful utterances can be acquired from the emotional voice dataset, and then reconstructed to fit multiple classes of abnormal voice, normal voice, and noise. In an embodiment, the artificial intelligence model of the first voice analysis unit (130) can be trained based on the ESC-50 dataset and the RAVDESS dataset, and can be configured to include a class corresponding to screaming or shouting, a class corresponding to normal speech, and a class corresponding to background noise.

[0066] The artificial intelligence model of the first voice analysis unit (130) can generate probability distributions for a plurality of classification classes. In one embodiment, the artificial intelligence model can output probability values ​​for abnormal voice, normal voice, and noise. More specifically, abnormal voice may be defined as a concept encompassing screams, shouts, or similar dangerous sound signs, normal voice may include normal conversation, laughter, breathing sounds, or everyday vocalizations, and noise may include unintentional environmental sounds or background noise. As an example, the first voice analysis unit (130) can generate probability distributions for three classes corresponding to scream, normal, and noise. Accordingly, the judgment unit (150) can quantitatively determine the degree of probability that the output of the first voice analysis unit (130) is abnormal voice and can perform probability-based complex judgment rather than simple binary judgment.

[0067] The first voice analysis unit (130) can convert the raw acoustic signal provided from the multimodal information acquisition unit (110) into a feature format suitable for artificial intelligence inference. For example, the raw acoustic signal may be digital audio data in PCM format, and the first voice analysis unit (130) may divide the acoustic signal of a certain time interval into frames and then generate a feature vector including Mel frequency cepstrum coefficients and their change amounts. The acoustic information may be, for example, a mono PCM signal having a sampling frequency of 16 kHz, and the first voice analysis unit (130) may extract 39-dimensional features including MFCC 13th order, Delta, and Delta-Delta from the acoustic signal of a 0.5-second or 1-second interval. The acoustic features thus generated may be composed of an input tensor of, for example, the format float32[1,39,32,1]. Through this feature extraction process, the time domain information of the raw acoustic signal is converted into a fixed-shape tensor containing frequency domain and rate of change information, so the neural network processing unit can perform fast and stable artificial intelligence inference based on the fixed shape. In other words, the first voice analysis unit (130) does not directly process the variable-length raw acoustic signal, but rather converts it into a fixed-shape acoustic feature tensor and inputs it into an artificial intelligence model, thereby forming a structure suitable for real-time inference on an edge device.

[0068] The first voice analysis unit (130) can output a probability distribution for each class as an inference result of an artificial intelligence model. For example, the output of the first voice analysis unit (130) may be a vector containing multiple probability values ​​corresponding to abnormal voice, normal voice, and noise. In one embodiment, the first voice analysis unit (130) may output a probability distribution in the format float32[1,3], and an example output may be in the form of {scream: 0.91, normal: 0.07, noise: 0.02}. The judgment unit (150) may directly use the probability value corresponding to abnormal voice from this output and, if necessary, interpret the classification result of the first voice analysis unit (130) by comparing it with the probability values ​​of normal voice and noise. For example, if the probability value corresponding to abnormal voice is the largest and is greater than or equal to a preset threshold, it may be determined that abnormal voice has been detected, and if the probability value of normal voice or noise is dominant, it may be determined that the acoustic channel is in a normal state. In this way, by the first voice analysis unit (130) outputting a probability distribution, the judgment unit (150) can perform a composite judgment that more finely reflects the state of the acoustic channel.

[0069] The classification criteria or threshold of the first voice analysis unit (130) can be adjusted according to the purpose of monitoring and the installation environment. For example, in environments such as underground parking lots with significant reverberation, the threshold for determining abnormal voice can be set high to reduce false alarms caused by background noise and reflected sound, while in spaces where sound is transmitted directly, such as narrow corridors or stairwells, a relatively low threshold can be used. Additionally, in spaces where children's daily loud speech frequently occurs, such as areas adjacent to playgrounds, the criteria for distinguishing between normal voice and abnormal voice can be set more strictly. Accordingly, the first voice analysis unit (130) can reduce false alarms and improve detection accuracy by applying classification criteria suitable for the acoustic environment of the installation location and the purpose of monitoring, rather than simply using a fixed threshold.

[0070] The artificial intelligence model of the first voice analysis unit (130) can be retrained or updated to suit the characteristics of the monitored space. For example, since the characteristics of background noise, residents' speech patterns, and environmental sound composition may vary depending on the installation location, such as a corridor of an apartment building, a stairwell, a parking lot, or an area adjacent to a playground, the artificial intelligence model of the first voice analysis unit (130) can be further trained or fine-tuned based on acoustic data collected after on-site installation. In the embodiment, the characteristics of the site can be reflected using a transfer learning method while maintaining existing weights by utilizing scream, shouting, general conversation, and environmental noise data collected on-site. Accordingly, the first voice analysis unit (130) can secure abnormal voice classification performance that reflects the site's unique reverberation, noise source, and microphone placement conditions, which can lead to a reduction in false positive rates and an improvement in the reliability of the acoustic channel.

[0071] The artificial intelligence model of the first voice analysis unit (130) can be quantized and executed in a neural network processing unit. For example, the completed model can be converted into TFLite format or ONNX format, and then converted again into an inference format suitable for the neural network processing unit. In an embodiment, the artificial intelligence model of the first voice analysis unit (130) can be quantized into INT8 and executed in a neural network processing unit, and the model size can be maintained at a very small level. By using such a quantized lightweight artificial intelligence model, the memory usage and inference time required by the edge device can be reduced, and the first voice analysis unit (130) can stably perform inference on acoustic windows that repeat at short time intervals. In addition, by executing the quantized acoustic artificial intelligence model in the second core group of the neural network processing unit, computational resources can be separated from the first core group used by the image analysis unit (120), and the central processing unit can focus on calculating acoustic magnitude, conditional voice recognition control, and final judgment. Accordingly, the division of labor structure between the neural network processing unit and the central processing unit can be implemented more effectively.

[0072] By operating in the second core group of the neural network processing unit, the first voice analysis unit (130) can perform artificial intelligence inference in parallel with the first core group used by the image analysis unit (120). Accordingly, visual channel-based abnormal situation analysis and acoustic channel-based abnormal voice classification can be performed simultaneously or asynchronously, and the central processing unit can focus on voice recognition control and final risk level determination according to preset conditions. Due to this division of computational resources structure, the first voice analysis unit (130) can reduce bottlenecks in the entire device while maintaining continuous acoustic monitoring, and consequently, improve the responsiveness and stability of abnormal situation detection. Furthermore, by the first voice analysis unit (130) continuously generating a probability distribution regarding whether the voice is abnormal and providing it to the judgment unit (150), the judgment unit (150) can combine the reliability score of the image channel and the classification result of the acoustic channel to reduce false alarms while enabling a more rapid response to actual danger signs.

[0073] The second voice analysis unit (140) is configured to perform voice recognition and preset keyword detection only when preset conditions are met for the acoustic information provided by the multimodal information acquisition unit (110), and is responsible for a secondary semantic-based judgment on the acoustic channel. The second voice analysis unit (140) is distinguished from the first voice analysis unit (130), which classifies whether the voice is abnormal based on the acoustic pattern itself, by performing STT (Speech-to-Text) based voice recognition on the acoustic information and detecting preset keywords from the voice recognition results. Accordingly, the second voice analysis unit (140) can provide the determination unit (150) with the existence of keywords related to profanity, threatening speech, distress calls, or abnormal situations that are difficult to identify solely by acoustic abnormalities such as screaming or shouting.

[0074] The second voice analysis unit (140) can also be implemented on an edge device and can be placed and operated within a central processing unit inside the edge device. In particular, the second voice analysis unit (140) can be executed in a high-performance core group of the central processing unit, and accordingly, can perform operations such as continuous decoding, token generation, and text post-processing required for voice recognition. By placing the second voice analysis unit (140) in the central processing unit in this way, variable-length voice recognition operations, which are distinct from the fixed-shape-based inference structure of the neural network processing unit, can be processed more stably, and the neural network processing unit can be divided to focus on image analysis and the first voice analysis.

[0075] The second voice analysis unit (140) can be activated only when acoustic information satisfies preset conditions. For example, the second voice analysis unit (140) can be controlled to perform voice recognition only when the magnitude value of acoustic information acquired from the multimodal information acquisition unit (110) is greater than or equal to a preset first threshold value. As an example, if the magnitude value of the acoustic information is 75 dB or higher, the second voice analysis unit (140) can be activated because there is a possibility that a shout or abnormal speech has started. Due to this conditional activation structure, the second voice analysis unit (140) does not perform voice recognition operations at all times, but can operate selectively only when there is a high probability that speech related to an abnormal situation actually exists. As a result, the computational load of the central processing unit can be maintained at an intermittent level, and the resource efficiency and real-time capability of the entire device can be improved.

[0076] The second voice analysis unit (140) can convert acoustic information into text using a speech recognition model. In one embodiment, the second voice analysis unit (140) may use a transformer-based speech recognition model or an equivalent automatic speech recognition model. For example, the second voice analysis unit (140) may use a faster-whisper family model, a whisper family model, a Vosk family model, or a speech recognition model that performs substantially the same function. In an embodiment, the second voice analysis unit (140) may use a faster-whisper tiny model, which can be implemented as a lightweight speech recognition engine for converting speech segments into text. Since such a speech recognition model has a processing length that varies depending on the input length and may include an autoregressive structure in which a decoder sequentially generates tokens, it is suitable for execution on a central processing unit. Accordingly, a division of roles is possible in which the neural network processing unit performs fixed-shape-based image analysis and acoustic pattern classification, and the central processing unit performs variable-shape-based speech recognition.

[0077] The second voice analysis unit (140) can use a section of acoustic information provided by the multimodal information acquisition unit (110) that satisfies a preset condition as a voice recognition input. For example, the multimodal information acquisition unit (110) can store acoustic information after a specific point in time in a buffer, and the second voice analysis unit (140) can acquire an acoustic signal of a certain time interval based on a point in time when the magnitude value of the acoustic information is greater than or equal to a first threshold value and input it into a voice recognition model. For example, the second voice analysis unit (140) can use an acoustic buffer of 10 seconds in length as a voice recognition target. The input acoustic signal may be a PCM signal in float32 format having a sampling frequency of 16 kHz, and normalization or buffer alignment may be performed to match the format required by the voice recognition model. In this way, by using a sound segment of a certain length as input, the second voice analysis unit (140) can secure a context including speech content before and after the occurrence of an abnormal situation, and can perform semantic analysis based on continuous speech rather than fragmentary speech.

[0078] The second voice analysis unit (140) can output text information as a voice recognition result. For example, the second voice analysis unit (140) can generate a string restored from an input sound segment and additionally output a preset keyword detection result together. The output of the second voice analysis unit (140) may be a structure including, for example, text, keywords, and count. text may be a string generated as a voice recognition result, keywords may be a list of preset keywords detected within the string, and count may be the number of detected keywords.

[0079] The second voice analysis unit (140) can detect preset keywords from the voice recognition result text. The preset keywords may be defined to reflect the utterance content related to an abnormal situation. For example, the preset keywords may include vocabulary representing profanity, threatening utterance, distress request words, aggressive expressions, or other utterances related to an abnormal situation. The preset keywords may be constructed using a Korean profanity dictionary or a set of words equivalent thereto, and may be detected through regular expression matching or string comparison. In this way, by the second voice analysis unit (140) performing text-based keyword detection, it can provide the judgment unit (150) with semantic danger signs that are difficult to identify solely through acoustic pattern analysis handled by the first voice analysis unit (130), and the judgment unit (150) can determine the abnormal situation more precisely based on the utterance content itself as well as the intensity or pattern of the sound.

[0080] The voice recognition criteria or keyword detection criteria of the second voice analysis unit (140) can be adjusted according to the monitoring purpose and installation environment. For example, since frequently used speech expressions, living noise levels, and acoustic reverberation conditions may vary depending on the installation location, such as a hallway, stairwell, parking lot, or area adjacent to a playground, the voice recognition activation conditions, input buffer length, or keyword dictionary configuration may be adjusted according to the environment. In addition, in a specific installation environment, profanity detection may be prioritized, while in another installation environment, the detection of rescue request speech or threat expressions may be focused. Accordingly, the second voice analysis unit (140) can apply semantic analysis criteria suitable for the characteristics of the installation environment to reduce false alarms and more effectively capture speech related to actual abnormal situations.

[0081] The speech recognition model or keyword detection logic used by the second voice analysis unit (140) can be retrained, replaced, or updated. For example, language adaptation of the speech recognition model can be performed by reflecting field acoustic data collected after installation, resident speech patterns, or speech data when actual abnormal situations occur, or a pre-configured keyword dictionary can be added or modified. When configured to enable updates that reflect field characteristics in this way, the second voice analysis unit (140) can better capture regional differences in expression, pronunciation variations, abbreviated expressions, or diversity in rescue request expressions, even for the same profanity. Accordingly, the second voice analysis unit (140) can maintain semantic-based analysis performance suitable for the installation environment, and continuous performance improvement is possible in a surveillance system that operates for a long period.

[0082] The second voice analysis unit (140) can operate complementarily with the first voice analysis unit (130). While the first voice analysis unit (130) rapidly classifies abnormal voice patterns, such as screams or shouts, from acoustic features of short time intervals, the second voice analysis unit (140) performs semantic analysis on relatively long acoustic intervals only when preset conditions are met. Accordingly, the first voice analysis unit (130) is responsible for a low-latency primary detection function, and the second voice analysis unit (140) is responsible for a high-load but semantically rich secondary verification function. Due to this dual voice analysis structure, the multimodal information-based abnormal situation detection and response device (100) can utilize both pattern-based and semantic-based danger signs in the acoustic channel, and the judgment unit (150) can perform a more sophisticated composite judgment by combining with the abnormal situation-related reliability score of the image analysis unit (120). In particular, even if the video channel suggests danger, an alert can be withheld if there are no semantic danger signs in the audio channel; conversely, if strong semantic danger signs are detected in the audio channel, it can lead to a rapid danger assessment, thereby allowing for the reduction of false alarms while maintaining response speed.

[0083] The judgment unit (150) is configured to determine the risk level of an abnormal situation by synthesizing the analysis results provided from the image analysis unit (120), the first voice analysis unit (130), and the second voice analysis unit (140), and to interpret the information of multiple modalities in an integrated manner.

[0084] Referring to FIG. 4, the judgment unit (150) can determine in stages how much risk a situation occurring in a monitored space has by considering together the reliability score related to an abnormal situation provided by the image analysis unit (120), the magnitude value of sound information provided by the multimodal information acquisition unit (110) or calculated from sound information, the classification result of the first voice analysis unit (130), and the keyword detection result of the second voice analysis unit (140).

[0085] The decision unit (150) can also be implemented on an edge device. The decision unit (150) can operate within a central processing unit inside such an edge device. By placing the decision unit (150) in the central processing unit, while the neural network processing unit performs artificial intelligence inference through the image analysis unit (120) and the first voice analysis unit (130), the central processing unit can receive the semantic-based analysis results of the second voice analysis unit (140) and the quantitative judgment values ​​of each channel to determine the final risk level. When the decision unit (150) is configured to operate in the central processing unit in this way, a division of labor structure between the neural network processing unit and the central processing unit is clearly formed, and the function of integrating the inference results of each channel into rule-based or condition-based logic can be reliably performed.

[0086] The judgment unit (150) can receive a plurality of input values ​​and determine the level of risk. In one embodiment, the judgment unit (150) may use an abnormal situation related reliability score provided by the image analysis unit (120), a sound information magnitude value, an abnormal voice related probability or classification result provided by the first voice analysis unit (130), and a keyword detection result provided by the second voice analysis unit (140) as input values. As an example, the values ​​input to the judgment unit (150) may have a structure such as, for example, v_score, dB value, scream_prob, and stt_keywords. Here, the reliability score related to the abnormal situation may be a quantitative judgment value of the abnormal situation calculated from the video channel, and the magnitude value of the sound information may be a value representing the intensity of the sound event, and the classification result of the first voice analysis unit (130) may be whether there is an abnormal voice or a probability distribution thereon, and the keyword detection result of the second voice analysis unit (140) may include whether there is a keyword related to profanity, threatening expressions or distress request utterances, the type or number of such keywords.

[0087] The judgment unit (150) can perform a composite judgment by considering that the nature of each input value is different. The reliability score related to abnormal situations reflects the risk potential based on object or behavior-related features captured in the visual channel, the magnitude value of the acoustic information reflects the physical intensity of the acoustic event, the classification result of the first voice analysis unit (130) reflects whether there is an abnormality in terms of the acoustic pattern, and the keyword detection result of the second voice analysis unit (140) reflects whether there is a semantic risk in the utterance content. Therefore, by combining multiple judgment values ​​that interpret the same situation from different perspectives, the judgment unit (150) can more accurately determine abnormal situations that are difficult to capture with only one channel. For example, even if the video channel shows a high probability of violence or falling, if the audio channel does not show any signs of danger, it may not be immediately judged as the highest risk level; conversely, even if the risk score of the video channel is not high, if strong abnormal voice or threatening utterance is detected in the audio channel, it may be judged as a state of caution. With such a complex judgment structure, the judgment unit (150) can reduce false positives or omissions that may occur in single modality-based detection.

[0088] The judgment unit (150) can determine the risk level in multiple stages based on preset conditions or thresholds. In one embodiment, the judgment unit (150) can determine the first risk level if the reliability score related to the abnormal situation is greater than or equal to the second threshold, and at least one of the magnitude value of the acoustic information, the classification result of the first voice analysis unit (130), and the keyword detection result of the second voice analysis unit (140) satisfies the risk condition. In the embodiment, this first risk level may correspond to DANGER (first risk level). For example, if the reliability score related to violence in the video channel is high, and at the same time, a shout or abnormal voice of a certain magnitude or greater is detected, or a threatening keyword is detected, the judgment unit (150) can determine the first risk level by judging that there is a high probability that an actual abnormal situation has occurred. With such a configuration, the judgment unit (150) ensures that the highest level of alarm condition is met only when visual danger signs and acoustic danger signs complement each other, thereby suppressing excessive alarms caused by simple video false positives.

[0089] The judgment unit (150) may determine a surveillance state indicating a suspected danger state without generating an external alarm when the reliability score related to an abnormal situation is above the second threshold but the danger condition of the acoustic channel is not satisfied. In the embodiment, this surveillance state may correspond to WATCH (surveillance state). For example, if the video analysis unit (120) detects an act similar to violence or physical fighting and calculates a high reliability score related to an abnormal situation, but the magnitude value of the acoustic information is low, the first voice analysis unit (130) determines normal voice or noise as dominant, and the second voice analysis unit (140) does not detect threatening keywords, the judgment unit (150) may determine a surveillance state without immediately determining a first danger level. This surveillance state is a state for deferring the handling of cases where danger signs are captured only in the video channel, and can contribute to distinguishing between actual violent situations such as play, exercise, moving, daily physical contact, etc. Therefore, by having the judgment unit (150) separately define the monitoring state, it has the effect of structurally suppressing false alarms based on image-only judgment.

[0090] The judgment unit (150) may determine a state of caution if the reliability score related to the abnormal situation is below the second threshold value, but the magnitude value of the sound information or the classification result of the first voice analysis unit (130) satisfies the danger condition. In the embodiment, this state of caution may correspond to an ALERT. For example, if no clear violence or abnormal behavior is captured in the video channel, but an abnormal sound corresponding to shouting or screaming is strongly detected, or if the magnitude value of the sound information is above a certain level, the judgment unit (150) may determine that there is a possibility that a potential abnormal situation has occurred and determine a state of caution. In this way, the judgment unit (150) can preemptively detect the possibility of danger based on the sound channel even when the judgment of the visual channel is limited due to insufficient video information, nighttime environment, obscuration, or limited viewing angle. Accordingly, the possibility of missing an actual abnormal situation can be reduced, and further deterioration can be prevented through minimal response.

[0091] The determination unit (150) can determine a normal state when all input values ​​correspond to normal conditions. In an embodiment, this normal state may correspond to NORMAL. For example, if the reliability score related to abnormal situations is low, the magnitude value of the acoustic information is at an everyday level, the first voice analysis unit (130) classifies it as normal voice or noise, and the second voice analysis unit (140) does not detect keywords related to abnormal situations, the determination unit (150) can determine a normal state. Accordingly, the device can maintain a normal monitoring state while minimizing unnecessary alarms, notification transmissions, or recordings. This definition of a normal state contributes to increasing the consistency of subsequent measures corresponding to each state by clearly defining the boundary with the monitoring state or the attention state.

[0092] The judgment unit (150) can generate judgment results to induce different responses according to the risk level. For example, if it is determined to be at the first risk level, the response unit (160) can be controlled to perform voice guidance output, lighting control, and remote notification transmission. If it is determined to be in a monitoring state, it can be controlled not to generate an external alarm, but to perform event log storage, dashboard display, video frame storage, enhanced tracking, or subsequent risk level transition monitoring. If it is determined to be in a caution state, it can be controlled to perform limited responses such as warning lights or lighting control while monitoring for additional sound or video changes, and if it is determined to be in a normal state, it can be controlled to maintain a normal monitoring state. In this way, by the judgment unit (150) classifying and outputting risk levels in multiple stages, the response unit (160) can perform differentiated actions according to the severity of the situation and simultaneously reduce excessive and insufficient responses.

[0093] The judgment unit (150) can be configured to allow the threshold value or judgment criteria for determining the risk level to be adjusted according to the installation environment. For example, since the patterns of normal behavior and the characteristics of environmental noise may differ depending on the installation location, such as a corridor in an apartment building, a stairwell, a parking lot, or an area adjacent to a playground, the second threshold value for the reliability score related to abnormal situations, the criteria for the magnitude value of acoustic information, the criteria for judging abnormal voices, or the criteria for reflecting keywords can be adjusted according to the environment. In an area adjacent to a playground, the threshold value for the reliability score related to abnormal situations of the video channel can be set high to reduce false alarms, and in a space with a large reverberation, the criteria for the magnitude value of acoustic information can be adjusted to reduce false alarms caused by sound reflection. Accordingly, the judgment unit (150) can apply judgment logic suitable for the installation environment rather than simple fixed rules, and can increase detection accuracy and operational reliability for each site.

[0094] The judgment unit (150) may generate event logs or detailed judgment data. For example, the judgment unit (150) may generate detailed data including the score of the video channel, the size value and classification result of the audio channel, the keyword detection result, time information, etc., along with the judgment result. In an embodiment, the judgment result may be in the form of a structure including level and detail, and the detail may include the video analysis result, the audio analysis result, the second voice analysis result, and a timestamp. Such detailed judgment data may be provided to the event log storage unit (400), the control server (200), or the administrator terminal (300), and may be used as basic data for post-analysis, threshold adjustment, or field-specific retraining.

[0095] A division of labor structure between a neural network processor and a central processing unit can be completed by a structure in which a judgment unit (150) operates in the central processing unit and integrates the results of an image analysis unit (120), a first voice analysis unit (130), and a second voice analysis unit (140). The image analysis unit (120) and the first voice analysis unit (130) perform artificial intelligence inference in different core groups of the neural network processor, while the second voice analysis unit (140) and the judgment unit (150) perform semantic analysis and final judgment in the central processing unit. With this configuration, the neural network processor handles the first-order artificial intelligence inference of the visual channel and the acoustic channel, and the central processing unit handles complex condition judgment and state transition control, thereby enabling functional division suitable for the characteristics of each computational resource. As a result, the multimodal information-based abnormal situation detection and response device (100) can simultaneously achieve high real-time performance, a low false alarm rate, and stable abnormal situation judgment even in an edge device environment.

[0096] The response unit (160) is configured to perform response actions on the monitored space according to the risk level provided by the judgment unit (150), and forms an execution unit that converts the judgment result from the multimodal information-based abnormal situation detection and response device (100) into an actual alarm, lighting control, notification transmission, or recording action. The response unit (160) receives risk level or status information generated by the judgment unit (150) and can perform different responses depending on the severity of the abnormal situation. Accordingly, the response unit (160) does not simply generate alarms uniformly, but performs differentiated measures in response to judgment results classified as a first risk level, monitoring status, caution status, and normal status, thereby enabling rapid and active response when necessary while reducing excessive alarms.

[0097] The response unit (160) can operate through a central processing unit inside the edge device or an output control interface connected thereto. By configuring the response unit (160) around the central processing unit or the output control interface, the voice output device, lighting device, remote notification transmission function, event recording function, or screen display function can be integratedly controlled based on the risk level determined by the judgment unit (150). When the response unit (160) operates in conjunction with the judgment unit (150) in this manner, the artificial intelligence inference result performed by the neural network processing unit does not end as a simple analysis value but can be directly connected to on-site response or remote notification.

[0098] The response unit (160) can perform active response when the risk level is determined to be the first risk level. For example, if the determination unit (150) determines the first risk level by judging that the reliability score related to the abnormal situation is above the standard and that at least one of the magnitude value of the sound information, the classification result of the first voice analysis unit (130), or the keyword detection result of the second voice analysis unit (140) satisfies the risk condition, the response unit (160) can perform voice guidance output, lighting control, and remote notification transmission. In the embodiment, the first risk level may correspond to DANGER, and the response unit (160) can perform voice guidance or warning message output through a speaker, turning on a searchlight or warning light, and transmitting a push notification to an administrator terminal (300) or a control server (200). The voice guidance output performed by the response unit (160) may be an operation to convey the fact of the occurrence of an abnormal situation or a warning message to the monitored space. The lighting control performed by the response unit (160) may be an operation for visual warning or securing illumination for the monitored space. For example, the response unit (160) may turn on or control the flashing of searchlights, warning lights, or other lighting devices. The remote notification transmission performed by the response unit (160) may be an operation for transmitting information related to abnormal situations to an administrator terminal (300), a control server (200), or an external system. For example, the response unit (160) may generate a notification message including the result of the first risk level or caution state determination by the judgment unit (150), the time of event occurrence, the risk level, the video analysis result, the sound analysis result, or the keyword detection result, and transmit it to the administrator terminal (300) or the control server (200). Remote notification transmission may be performed via push notification, message transmission, network packet transmission, or a similar method. Accordingly, not only on-site response but also the intervention of remote monitoring personnel becomes possible, and centralized management can be achieved even in an environment where multiple devices are installed.In particular, if the response unit (160) is configured to have different notification transmission status or notification content depending on the multiple states of the judgment unit (150), it is possible to reduce unnecessary notifications while still delivering information suitable for the actual risk level.

[0099] The response unit (160) may not generate an external alarm when determined to be in a monitoring state, but may perform auxiliary response or recording-related operations. A monitoring state may apply when a sign of danger is detected in the video channel but the audio channel does not sufficiently support it. In such cases, the response unit (160) may not perform active external alarms, such as using a speaker, lighting device, or remote notification transmission. Instead, the response unit (160) may record information related to the monitoring state in the event log storage unit (400), provide data to display the monitoring state to the control server (200) or administrator terminal (300), store video frames of the monitored area, or generate control signals for tracking enhancement. For example, it may store video snapshots or buffered video from before and after the time the monitoring state occurred, display the attention observation state on the dashboard, and continuously monitor changes in the audio channel or video analysis score thereafter.

[0100] The response unit (160) may perform limited response when determined to be in a state of caution. For example, if the judgment unit (150) determines that the reliability score related to abnormal situations is below the standard but the magnitude value of the sound information or the classification result of the first voice analysis unit (130) satisfies the danger condition and determines to be in a state of caution, the response unit (160) may perform lighting control and not perform voice guidance output or remote notification transmission. In the embodiment, such a state of caution may correspond to an ALERT and may only perform the turning on of a searchlight or lighting device.

[0101] The response unit (160) may be configured to maintain a normal monitoring state when determined to be in a normal state. For example, if the determination unit (150) determines that all input values ​​correspond to normal conditions, the response unit (160) may not perform voice guidance output, lighting control, or remote notification transmission. However, even in a normal state, the normal monitoring function of the device may be maintained, and a standby state may be maintained in preparation for the possibility of the next abnormal situation occurring. Accordingly, the response unit (160) can continuously maintain the monitoring function of the entire device without generating unnecessary output or network traffic.

[0102] The response unit (160) can provide detailed event information by being linked with the event log storage unit (400), the control server (200), or the administrator terminal (300). For example, the response unit (160) can control the storage of detailed event data, including the risk level generated by the judgment unit (150), the video analysis result, the magnitude value of the sound information, the classification result of the first voice analysis unit (130), the keyword detection result of the second voice analysis unit (140), and time information, in the event log storage unit (400), or transmit it to the control server (200), or display it on the administrator terminal (300). With this configuration, the circumstances at the time of the occurrence of the abnormal situation, the response history, and the basis for judgment can be managed together and can be utilized for post-analysis, response to complaints, threshold adjustment, or securing additional learning data.

[0103] The response unit (160) may selectively combine and perform multiple response means according to the risk level determination result of the judgment unit (150). For example, at the first risk level, voice guidance output, lighting control, and remote notification transmission are all performed; in the monitoring state, event recording and screen display are performed; in the caution state, only lighting control is performed; and in the normal state, no external response is performed.

[0104] By a structure in which the response unit (160) is linked with the judgment unit (150) to perform multi-stage response, the multimodal information-based abnormal situation detection and response device (100) can form a series of closed-loop safety response systems that include analysis results and response results, going beyond the division of labor structure between the neural network processing unit and the central processing unit. That is, the image analysis unit (120) and the first voice analysis unit (130) infer abnormal signs in the neural network processing unit, and the second voice analysis unit (140) and the judgment unit (150) perform semantic analysis and final judgment in the central processing unit, and then the response unit (160) executes the results as on-site warnings, remote notifications, record storage, or lighting control, thereby enabling detection, judgment, and response to be continuously realized within a single edge device. With this configuration, the multimodal information-based abnormal situation detection and response device (100) can enable real-time abnormal situation response while reducing dependency on external servers, and can perform active on-site measures when necessary while reducing false alarms.

[0105] The control server (200) is configured to receive, store, manage, and display judgment results or event information provided by the multimodal information-based abnormal situation detection and response device (100), and is a central management unit that integrally manages the operating status of multiple devices and the history of abnormal situations in the multimodal information-based abnormal situation detection and response system (1000).

[0106] The control server (200) can receive the risk level, detailed judgment data, time of event occurrence, video analysis result, sound analysis result, or keyword detection result generated by the multimodal information-based abnormal situation detection and response device (100) through the judgment unit (150). The control server (200) can store the judgment result or event log. For example, the control server (200) can store the risk level, detailed judgment data, time of event occurrence, installation location information, device identification information, response history, or snapshot information in a database format.

[0107] Additionally, the control server (200) can visually display judgment results or event logs. For example, the control server (200) can display a normal state, monitoring state, caution state, or first risk level in the form of colors, icons, text, or a chronological list of events through a dashboard function.

[0108] The control server (200) can be connected via a network to a multimodal information-based abnormal situation detection and response device (100) or an administrator terminal (300). For example, the control server (200) can be connected to multiple multimodal information-based abnormal situation detection and response devices (100) via a wired or wireless communication network to receive device-specific events and can provide a dashboard screen, event list, or alarm history to the administrator terminal (300). Accordingly, the control server (200) can perform relay and integrated management roles between the field device and the administrator terminal (300), enabling centralized operation of the entire abnormal situation detection and response system.

[0109] The administrator terminal (300) is configured to receive judgment results or alarm information provided by a multimodal information-based abnormal situation detection and response device (100) or a control server (200), and forms a user interface in the multimodal information-based abnormal situation detection and response system (1000) for an administrator to remotely check whether an abnormal situation has occurred. The administrator terminal (300) may be a smartphone, tablet, laptop, desktop computer, or an information processing device equivalent thereto. The administrator terminal (300) may receive push notifications, messages, or warning screens when a first risk level occurs, and may also receive or view information regarding a monitoring status, a caution status, or a normal status as needed.

[0110] The administrator terminal (300) is not limited to simply receiving notifications, but can also check detailed event information in conjunction with the control server (200). For example, the administrator terminal (300) can display the time of occurrence of an abnormal situation, the level of risk, video analysis results, sound analysis results, keyword detection results, event logs, or stored snapshot information.

[0111] The event log storage unit (400) is configured to store judgment results and additional information provided from a multimodal information-based abnormal situation detection and response device (100) or a control server (200), and forms a record management unit that structures and preserves abnormal situation-related history in a multimodal information-based abnormal situation detection and response system (1000). The event log storage unit (400) can store risk levels, detailed judgment data, time of event occurrence, installation location information, device identification information, video snapshot information, response execution history, or state transition history.

[0112] The event log storage unit (400) can perform an important function in distinguishing between a monitoring state and a normal state. For example, in a monitoring state, even if no external alarm occurs, the event log storage unit (400) can store judgment results, detailed analysis values, and snapshot information before and after the point in time. Accordingly, cases where risk signs appeared only in the video channel or cases that subsequently transitioned to a first risk level can be analyzed retrospectively, and data necessary for adjusting the threshold of the judgment unit (150), retraining the artificial intelligence model, or improving operational policies can be secured. On the other hand, in a normal state, unnecessary data accumulation can be reduced by not storing event logs.

[0113] The event log storage unit (400) may be implemented as an integral part of the control server (200) or as a separate storage device. For example, the event log storage unit (400) may be implemented as a database module inside the control server (200), or as an independent storage device connected to the control server (200) via a network.

[0114] The alarm output unit (500) is configured to output an alarm at the site or remotely according to the control of the multimodal information-based abnormal situation detection and response device (100) or the control server (200), and forms an external output unit that converts the judgment result from the multimodal information-based abnormal situation detection and response system (1000) into an audiovisual or electronic warning. The alarm output unit (500) may include a speaker, a voice guidance device, a searchlight, a warning light, a lighting device, or an equivalent output device. The alarm output unit (500) may perform a voice warning and a lighting warning when a first risk level occurs, may perform only a lighting warning in an attention state, and may not perform an active alarm in a monitoring state. The voice alarm performed by the alarm output unit (500) may have a function to immediately notify the site user of the occurrence of an abnormal situation.

[0115] The alarm output unit (500) may be built into the configuration of the multimodal information-based abnormal situation detection and response device (100), or it may be installed as a separate external component separated from the multimodal information-based abnormal situation detection and response device (100) in the multimodal information-based abnormal situation detection and response system (1000). For example, if a speaker, voice guidance device, searchlight, warning light, or lighting device is integrally provided in the edge device body, the alarm output unit (500) may be implemented as part of the multimodal information-based abnormal situation detection and response device (100). On the other hand, if a speaker or lighting device is installed at a separate location in the monitored space and controlled via a wired or wireless method, the alarm output unit (500) may be implemented as an external component of the multimodal information-based abnormal situation detection and response system (1000). In this case, the response unit (160) may function as a functional component that controls the operation of the alarm output unit (500) according to the judgment result of the judgment unit (150), and the alarm output unit (500) may function as a physical output means that performs actual voice output, lighting output, or warning light output according to the control of the response unit (160). Accordingly, the alarm output unit (500) may be included in the multimodal information-based abnormal situation detection and response device (100) depending on the implementation form, or it may be placed as a separate component of the multimodal information-based abnormal situation detection and response system (1000).

[0116] The control server (200), administrator terminal (300), event log storage unit (400), and alarm output unit (500) can be linked with the multimodal information-based abnormal situation detection and response device (100) to form the entire system. For example, the multimodal information-based abnormal situation detection and response device (100) can analyze multimodal information at the site to generate a judgment result, the control server (200) can manage the judgment result and event log in an integrated manner, the administrator terminal (300) can receive alarm or status information, the event log storage unit (400) can store history, and the alarm output unit (500) can perform a site alarm based on the judgment result.

[0118] Next, a method for detecting and responding to abnormal situations based on multimodal information according to an embodiment of the present invention will be described. In the embodiment, the method for detecting and responding to abnormal situations based on multimodal information is performed by a device (100) for detecting and responding to abnormal situations based on multimodal information. Accordingly, descriptions of the same content as the device for detecting and responding to abnormal situations described above are omitted.

[0119] Referring to FIG. 5, a multimodal information-based abnormal situation detection and response method (S1000) according to one embodiment of the present invention includes a multimodal information acquisition step (S100), a reliability score calculation step (S200), an abnormal voice classification step (S300), a voice recognition and keyword detection step (S400), a risk level determination step (S500), and a response action execution step (S600).

[0120] The multimodal information acquisition step (S100) is a step in which the multimodal information acquisition unit (110) acquires video information and audio information from the monitored space. The multimodal information acquisition unit (110) can collect visual information and audio information of the monitored space through a single device including a camera and a microphone or through multiple devices functionally combined. Video information can be acquired, for example, in the form of visible light images or infrared-based images, and audio information can be acquired in the form of digital audio signals. Since infrared-based visual information can be acquired together in nighttime or low-light environments, abnormal situation detection can be continued regardless of time of day and changes in illumination.

[0121] The reliability score calculation step (S200) is a step in which the image analysis unit (120) calculates a reliability score related to an abnormal situation based on the image information acquired in the multimodal information acquisition step (S100). The image analysis unit (120) can detect object or behavior-related features in the first core group of the neural network processing unit after preprocessing the image information into a format suitable for artificial intelligence inference. For example, the image analysis unit (120) can detect features related to a person, contact between people, change in posture, falling state, wandering state, or threat situation, and output reliability values ​​for multiple classes. Subsequently, the image analysis unit (120) can calculate a reliability score related to an abnormal situation by directly using the value of the class related to the abnormal situation among these class-specific reliability values, or by using the maximum value of multiple classes, a weighted sum, or post-processing results. By performing the reliability score calculation step (S200) in this manner, the risk probability captured in the visual channel is converted into a quantitative value and can be combined with the acoustic-based judgment of the subsequent step. As a result, more sophisticated complex judgments become possible compared to simple video surveillance.

[0122] The abnormal voice classification step (S300) is a step in which the first voice analysis unit (130) classifies whether the voice is abnormal based on the acoustic information acquired in the multimodal information acquisition step (S100). The first voice analysis unit (130) processes the acoustic information in units of a certain time interval and can extract acoustic features including Mel frequency cepstrum coefficients and change amounts from the acoustic signal. Subsequently, the first voice analysis unit (130) can calculate the probability distribution for abnormal voice, normal voice, and noise using a lightweight artificial intelligence model in the second core group of the neural network processing unit. Through the abnormal voice classification step (S300), screams, shouting, or other dangerous sound signs equivalent thereto can be distinguished from normal conversation, laughter, or environmental noise, and the state of the acoustic channel can be provided in a quantitative or probabilistic form. Accordingly, the judgment unit (150) can consider not only the video analysis results but also the abnormality of the acoustic pattern itself, thus contributing to reducing false alarms caused by video-only judgment. In addition, since the first voice analysis unit (130) operates in the second core group of the neural network processing unit, it can be performed in real time in parallel with the reliability score calculation step (S200).

[0123] The voice recognition and keyword detection step (S400) is a step in which the second voice analysis unit (140) performs voice recognition on acoustic information only when the acoustic information satisfies a preset condition, and detects a preset keyword from the voice recognition result. For example, if the magnitude value of the acoustic information is greater than or equal to a preset first threshold, the second voice analysis unit (140) can buffer an acoustic segment of a certain length based on that point in time and perform voice recognition in the central processing unit. The second voice analysis unit (140) can generate text as a result of voice recognition and detect profanity, threat expressions, distress requests, or other keywords related to abnormal situations from the generated text. In this way, the voice recognition and keyword detection step (S400) is a step of performing semantic-based analysis on the acoustic channel and plays a complementary role to the acoustic pattern-based classification performed by the first voice analysis unit (130). In particular, since the speech recognition and keyword detection step (S400) is performed only when preset conditions are met, high-load speech recognition operations do not need to be performed constantly, and the computational resources of the central processing unit can be used efficiently. Therefore, the speech recognition and keyword detection step (S400) can function as a step that enables the practical implementation of semantic analysis functions in an edge device environment.

[0124] The risk level determination step (S500) is a step in which the determination unit (150) determines the risk level by synthesizing the reliability score related to the abnormal situation, the magnitude value of the acoustic information, the classification result of the first voice analysis unit (130), and the keyword detection result of the second voice analysis unit (140). The determination unit (150) may consider together the possibility of an abnormal situation captured in the visual channel, the intensity of the acoustic event, whether there is an abnormality in the dimension of the acoustic pattern, and whether there is a semantic risk in the dimension of the utterance content. For example, if the reliability score related to the abnormal situation is greater than or equal to a preset second threshold value, and at the same time at least one of the magnitude value of the acoustic information, the classification result of the first voice analysis unit (130), or the keyword detection result of the second voice analysis unit (140) satisfies the risk condition, the determination unit (150) may determine the first risk level. In addition, if the reliability score related to abnormal situations is above the second threshold but the danger condition of the acoustic channel is not satisfied, it can be determined to be in a monitoring state, and even if the reliability score related to abnormal situations is below the second threshold, if the magnitude value of the acoustic information or the classification result of the first voice analysis unit (130) satisfies the danger condition, it can be determined to be in a caution state. If all input values ​​correspond to normal conditions, it can be determined to be in a normal state.

[0125] The response action execution step (S600) is a step in which the response unit (160) performs at least one response action among alarm output, lighting control, and notification transmission according to the determination result of the risk level determination step (S500). For example, if the first risk level is determined in the risk level determination step (S500), the response unit (160) may perform voice guidance output, lighting control, and remote notification transmission. If it is determined to be in a monitoring state, it may not generate an external alarm, but may perform event log storage, dashboard display, snapshot storage, or tracking enhancement. If it is determined to be in a caution state, it may perform only limited responses such as lighting control, and if it is determined to be in a normal state, it may maintain a normal monitoring state without separate external responses.

[0126] The multimodal information acquisition step (S100), reliability score calculation step (S200), abnormal voice classification step (S300), speech recognition and keyword detection step (S400), risk level determination step (S500), and response action execution step (S600) may be performed sequentially, but are not limited thereto, and some steps may be performed in parallel or asynchronously. For example, the reliability score calculation step (S200) and the abnormal voice classification step (S300) may be performed in parallel on different core groups of the neural network processing unit, and the speech recognition and keyword detection step (S400) may be performed selectively only when a preset condition is met. Additionally, the risk level determination step (S500) may be performed repeatedly by periodically receiving the latest results of the video channel and audio channel, and the response action execution step (S600) may be executed dynamically according to changes in the determination result. In this way, the multimodal information-based abnormal situation detection and response method (S1000) combines artificial intelligence-based analysis of multiple modalities with a central processing unit-based complex judgment and performs differential follow-up measures corresponding to the risk level, thereby enabling real-time performance, computational efficiency, and detection reliability even in an edge device environment.

[0127] As described above, according to the present embodiment, a multimodal information-based abnormal situation detection and response device, method, and system are provided that detect an abnormal situation based on multimodal information including image information, sound information, and infrared-based visual information, and perform a response action according to the detection result, and in particular, by utilizing multiple computational resources in a division of labor within an edge device to perform image analysis, voice pattern analysis, conditional voice recognition, and complex judgment, thereby enabling rapid response while reducing false alarms.

[0128] The use of all examples or exemplary terms (e.g., etc.) in the present invention is merely for the purpose of describing the invention in detail, and the scope of the invention is not limited by such examples or exemplary terms unless limited by the claims. Furthermore, a person skilled in the art will understand that various modifications, combinations, and changes may be made according to design conditions and factors within the scope of the claims or equivalents to which they are added.

[0129] Accordingly, the scope of the present invention is not limited to the embodiments described above, and all scopes equivalent to or equivalently modified from the claims set forth below, as well as the claims described below, shall be considered to fall within the scope of the concept of the present invention. Explanation of the symbols

[0130] 1000: Multimodal Information-Based Anomaly Detection and Response System 100: Multimodal Information-Based Anomaly Detection and Response Device 110: Multimodal Information Acquisition Unit 120: Image Analysis Unit 130: 1st voice analysis unit 140: 2nd voice analysis unit 150 : Judgment Section 160 : Correspondence Section 200: Control Server 300: Administrator Terminal 400: Event log storage unit 500: Alarm output unit S1000: Multimodal Information-Based Anomaly Detection and Response Method S100: Multimodal information acquisition step S200: Reliability score calculation step S300: Abnormal speech classification step S400: Speech recognition and keyword detection step S500: Risk level determination step S600: Response action execution step

Claims

Claim 1 A multimodal information acquisition unit that acquires video information and audio information from a monitored space; an image analysis unit that calculates a reliability score related to an abnormal situation based on the video information acquired from the multimodal information acquisition unit; a first voice analysis unit that classifies whether a voice is abnormal based on the audio information acquired from the multimodal information acquisition unit; a second voice analysis unit that performs voice recognition and preset keyword detection on the audio information only when the audio information satisfies preset conditions; a determination unit that determines a risk level by synthesizing the reliability score related to the abnormal situation, the magnitude value of the audio information, the classification result of the first voice analysis unit, and the keyword detection result of the second voice analysis unit; and a response unit that performs at least one corresponding operation among alarm output, lighting control, and notification transmission according to the determination result of the determination unit.The image analysis unit is placed in a first core group of a neural network processing unit, the first voice analysis unit is placed in a second core group of the neural network processing unit, the second voice analysis unit and the judgment unit are placed in a central processing unit, and artificial intelligence models of different modalities are divided and assigned to perform detection of object or behavior-related features for the image information and to perform classification of abnormal voices for the acoustic information for the second core group, the second voice analysis unit performs voice recognition and preset keyword detection for the acoustic information only when the magnitude value of the acoustic information is greater than or equal to a preset first threshold, and the judgment unit determines a first risk level when the reliability score related to the abnormal situation is greater than or equal to a second threshold and at least one of the magnitude value of the acoustic information, the classification result of the first voice analysis unit, and the keyword detection result of the second voice analysis unit satisfies a risk condition, and when the reliability score related to the abnormal situation is greater than or equal to the second threshold and the magnitude value of the acoustic information, the classification result of the first voice analysis unit, and the keyword detection result of the second voice analysis unit are all under normal conditions A multimodal information-based abnormal situation detection and response device that, in applicable cases, determines a monitoring state indicating a suspected danger state without generating an external alarm, and determines a caution state when the reliability score related to the abnormal situation is less than the second threshold value and the magnitude value of the acoustic information or the classification result of the first voice analysis unit satisfies the danger condition. Claim 2 delete Claim 3 delete Claim 4 delete Claim 5 A multimodal information-based abnormal situation detection and response device according to claim 1, wherein the first voice analysis unit calculates a probability distribution for any one of the abnormal voice, normal voice, and noise based on acoustic features extracted from the acoustic information. Claim 6 delete Claim 7 delete Claim 8 A method for detecting and responding to an abnormal situation performed by a multimodal information-based abnormal situation detection and response device, comprising: a step of acquiring multimodal information including image information and sound information from a monitored space through a multimodal information acquisition unit; a step in which an image analysis unit in a first core group of a neural network processing unit calculates a reliability score related to an abnormal situation based on the image information; a step in which a first voice analysis unit in a second core group of the neural network processing unit classifies whether a voice is abnormal based on the sound information; a step in which a second voice analysis unit in a central processing unit performs voice recognition and preset keyword detection for the sound information only when the sound information satisfies preset conditions; and a step in which a judgment unit in the central processing unit determines a risk level by synthesizing the reliability score related to the abnormal situation, the magnitude value of the sound information, the classification result of the first voice analysis unit, and the keyword detection result of the second voice analysis unit. The method comprises the step of performing at least one response operation among alarm output, lighting control, and notification transmission according to the above risk level, wherein the step of calculating the reliability score related to the abnormal situation includes the step of detecting object or behavior-related features for the image information in the first core group, the step of classifying whether the voice is abnormal includes the step of classifying abnormal voice for the acoustic information in the second core group, wherein artificial intelligence models of different modalities are divided and assigned to the first core group and the second core group, the step of performing voice recognition and preset keyword detection is performed by the central processing unit only when the magnitude value of the acoustic information is greater than or equal to a preset first threshold, and the step of determining the risk level includes the step of determining the first risk level when the reliability score related to the abnormal situation is greater than or equal to a second threshold, and at least one of the magnitude value of the acoustic information, the classification result of the first voice analysis unit, and the keyword detection result of the second voice analysis unit satisfies the risk condition.A multimodal information-based abnormal situation detection and response method comprising: a step of determining a monitoring state indicating a suspected danger state without generating an external alarm when the reliability score related to the abnormal situation is greater than or equal to the second threshold, and the magnitude value of the acoustic information, the classification result of the first voice analysis unit, and the keyword detection result of the second voice analysis unit all correspond to normal conditions; and a step of determining a caution state when the reliability score related to the abnormal situation is less than the second threshold, and the magnitude value of the acoustic information or the classification result of the first voice analysis unit satisfies a danger condition; Claim 9 A multimodal information-based abnormal situation detection and response system comprising: a multimodal information-based abnormal situation detection and response device of claim 1 or claim 5; a notification receiver that receives a remote notification according to a judgment result of the multimodal information-based abnormal situation detection and response device; and a dashboard server that displays the judgment result or an event log.

Images