A situation information integration analysis system using multi-modal information and a situation information integration analysis method using the same

Abstract

The present invention relates to a situation information integration analysis system using multimodal information, and by analyzing video data acquired from a CCTV camera and audio data acquired from a microphone in a temporally aligned manner, it enables situation recognition, including dangerous events in visual blind spots that are difficult to detect through video-only analysis. The system of the present invention includes an edge device that analyzes video and audio data to generate comprehensive event information and transmits an RTSP-based video stream and event-related metadata, and a main VMS that receives this information and performs monitoring and automatic response. The main VMS provides an alert for a dangerous event based on the event-related metadata and automatically controls the field of view of a PTZ camera to the direction or location where the event occurs. Accordingly, the present invention utilizes acoustic information to compensate for visual blind spots and has the effect of enabling real-time monitoring and rapid response to dangerous events.

Description

Technology Field

[0001] The present invention relates to a situation information integration analysis system and method, and more specifically, to a situation information integration analysis system using multimodal information and a situation information integration analysis method using the same.

[0002] This invention is the result of the Regional Innovation Center University Support System (RISE) carried out in 2025 with funding from the Ministry of Education and Daegu Metropolitan City and support from the Daegu RISE Center (2025-RISE-03-001) Background Technology

[0003] Recently, numerous CCTVs (Camera Closed-Circuit Televisions) have been installed and operated on major roads, streets, and public spaces for the purposes of crime prevention, traffic management, and ensuring safety. These CCTV-based control systems are utilized to detect abnormal behavior or dangerous situations by collecting and analyzing video information in real time, and have established themselves as a core element of urban safety infrastructure.

[0004] However, conventional CCTV control systems rely primarily on video data-centric analysis methods, resulting in a structural limitation where they fail to comprehensively utilize sensory information beyond video data. In particular, despite the fact that audio data can be collected through microphones installed alongside or placed in close proximity to CCTV cameras, multimodal-based control technology that integrates and analyzes both video and audio information has not been sufficiently implemented.

[0005] Due to the absence of such multimodal integrated analysis, the following problems arise in existing systems. First, current monitoring systems are heavily focused on video-centric analysis, failing to effectively utilize acoustic information such as screams, explosions, glass breaking, and arguments. Consequently, there are limitations in recognizing dangerous situations occurring in areas that are physically outside the coverage range of CCTV cameras.

[0006] Furthermore, there is the issue of visual blind spots. For instance, in areas not directly covered by CCTV cameras—such as behind buildings, narrow alleys, or spaces obscured by vehicles or structures—it is difficult to detect dangerous situations, like assaults or accidents, based solely on video footage. Conversely, while acoustic signals like screams or sounds of destruction can be transmitted to the surroundings even in these areas, conventional technology fails to effectively incorporate this acoustic information into control center decisions.

[0007] Furthermore, the problem of information fragmentation arises. Even if abnormal behavior is detected through video analysis, it is difficult to accurately grasp the context of the incident unless the auditory information causing the behavior—such as arguments or threatening shouting—is analyzed alongside it. Consequently, control center personnel must assess the situation based on limited information, leading to reduced accuracy and speed in the response.

[0008] Therefore, a new technological approach is required to overcome the limitations of existing CCTV control systems that rely on video information, and to complement visual blind spots and improve the accuracy of situational awareness by integrally analyzing multiple sensory data, including acoustic information. Prior art literature

[0009] Patent Document 1: Korean Published Patent No. 10-2025-0071213 (Publication Date: May 21, 2025) Patent Document 2: Korean Published Patent No. 10-2009-0070325 (Publication Date: July 1, 2009) Patent Document 3: Korean Registered Patent No. 10-2731104 (Publication Date: November 15, 2024) Patent Document 4: Korean Published Patent No. 10-2025-0101827 (Publication Date: July 4, 2025) The problem to be solved

[0010] The present invention, devised in response to the aforementioned necessity, aims to provide a situation information integration analysis system using multimodal information and a situation information analysis method using the same, which complements the visual limitations of existing CCTV control systems by collecting and analyzing video and audio data in real time and sophisticatedly fusing them, thereby enabling a comprehensive understanding of the context of an event and, furthermore, a more rapid and effective response to dangerous situations. means of solving the problem

[0011] To achieve the above objective, a situation information integration analysis system using multimodal information according to an embodiment of the present invention comprises: an edge device installed adjacent to a CCTV camera and a microphone, receiving video data from the CCTV camera and receiving audio data from the microphone, generating event-related metadata from the video data and audio data, and transmitting the video data as an RTSP-based video stream; and a main VMS that receives the event-related metadata and the RTSP-based video stream from the edge device and performs a control function. The edge device is configured to extract video features and audio features, respectively, and combine them using a feature-level fusion method to generate comprehensive event information including audio-based danger events occurring in visual blind spots that are difficult to detect through video-only analysis. The main VMS is configured to provide an alert for danger events based on the comprehensive event information and to automatically control the field of view of a PTZ (Pan-Tilt-Zoom) camera in the direction of the danger event occurrence based on location information or direction information included in the event-related metadata, thereby enabling real-time response by compensating for visual blind spots using audio information.

[0012] In this case, the edge device includes a data collection unit that aligns the image data and the sound data on the same time axis based on timestamps assigned to the image data and the sound data, respectively.

[0013] In this case, the data collection unit further comprises: a dynamic automatic calibration module that automatically estimates and corrects relative spatial coordinates, time offsets, or clock errors between each edge device based on ambient sounds included in acoustic data collected from a plurality of edge devices installed at mutually spaced locations, without a separate calibration signal or dedicated calibration procedure; a semantic masking-based privacy-preserving audio processing unit that does not store or transmit voice signals among acoustic data in their original state, converts voice into text information and emotion or urgency indicators, discards the original voice, and generates reconstructed synthetic voice or de-voiced audio information based on said text information and emotion or urgency indicators; and an optical-acoustic composite synchronization module that performs device identification, pairing, or clock synchronization between the CCTV camera and the microphone by the CCTV camera visually or acoustically receiving a visible light communication (VLC) signal or an inaudible frequency signal output from an active marker provided in a microphone or microphone array.

[0014] In this case, the comprehensive event information is configured to determine a risk event only if, after converting the spatial coordinates of the event estimated through image analysis and the sound source direction or location vector estimated through sound analysis into the same coordinate system, the geometric relationship value regarding at least one of the distance, angle, or intersection between the spatial coordinates and the sound source direction or location vector is within a preset standard range.

[0015] Meanwhile, the edge device includes a data preprocessing unit that performs resolution adjustment, noise removal, or illumination correction for image data, and environmental noise removal or abnormal sound candidate segment detection for acoustic data.

[0016] Meanwhile, the edge device includes a multimodal feature fusion unit that dynamically adjusts the combined weight of the image feature and the acoustic feature according to the reliability of the image feature or the reliability of the acoustic feature.

[0017] Meanwhile, the edge device includes an event generation unit that determines the event as a visual blind spot-based risk event if the sound-based risk level is greater than or equal to the threshold value, even if the reliability of the image-based judgment is below the threshold value.

[0018] The edge device includes a communication unit that transmits video data to the RTSP-based video stream and transmits event-related metadata through a communication channel independent of the video stream.

[0019] Meanwhile, the main VMS determines the priority of PTZ camera control based on the risk or reliability of the events when multiple risk events occur simultaneously, and corrects the PTZ control command to bypass the privacy protection area.

[0020] A method for integrating situational information using multimodal information according to another embodiment of the present invention comprises: receiving video data from a CCTV camera and receiving sound data from a microphone; aligning the video data and the sound data on the same time axis based on timestamps assigned to the video data and the sound data, respectively; extracting video features related to the presence, behavior, or scene change of an object from the time-aligned video data and extracting sound features related to a scream, impact sound, or breakage sound from the time-aligned sound data; adaptively fusing the video features and the sound features at the feature-level in a time-aligned state to generate comprehensive event information; determining whether a danger event has occurred, the type of the event, and the level of risk based on the comprehensive event information, and providing an alert for the danger event according to the determination result; and automatically controlling the pan, tilt, and zoom of a PTZ (Pan-Tilt-Zoom) camera according to the location information or direction information included in the comprehensive event information or event-related metadata generated based thereon, thereby switching the camera's field of view to the direction or location where the danger event occurred.

[0021] In this case, the step of adaptively fusing at the feature level is,

[0022] When reduced illumination, occlusion, or limited field of view occurs in the image data, the method includes a step of weighted combining the image feature and the acoustic feature to mutually complementarily adjust the combined weight of the image feature and the acoustic feature so that the contribution of the acoustic feature is increased, thereby including acoustic-based risk events occurring in visual blind spots that are difficult to detect through image-only analysis in the comprehensive event information.

[0023] Meanwhile, the step of providing an alert for the above-mentioned risk event includes the step of automatically linking video data corresponding to a predetermined time interval before and after the time of event occurrence based on the time of event occurrence included in the comprehensive event information or event-related metadata generated based thereon, and displaying it on a control screen; and the step of automatically controlling the field of view of the PTZ camera includes the step of calculating pan, tilt, and zoom control values ​​of the PTZ camera based on location information or direction information included in the event-related metadata, and determining the priority of the PTZ camera control according to the risk level or judgment reliability of the event when multiple risk events occur simultaneously, and executing them sequentially. Effects of the invention

[0024] According to the present invention, by analyzing acoustic data collected through a microphone linked to CCTV video together with video data using a feature-level fusion method, visual blind spots that were difficult to perceive in existing CCTV standalone control systems can be effectively compensated for.

[0025] In addition, by integratively analyzing video and audio information, it becomes possible to grasp the context of an event by considering both behavior and sound simultaneously, thereby improving the accuracy of judgment regarding dangerous situations and the situational awareness capabilities of control personnel.

[0026] Furthermore, by automatically controlling the PTZ camera's field of view toward the direction of dangerous events based on multimodal analysis results, real-time responsiveness is enhanced by enabling the active tracking and verification of incidents occurring in blind spots.

[0027] In addition, by providing a sound map that visually displays risk levels based on the spatial distribution of acoustic data, comprehensive threat detection and rapid control response become possible.

[0028] Consequently, the present invention can implement an advanced control system that provides technical effects such as eliminating blind spots, improving situational awareness accuracy, and enhancing real-time response capabilities. Brief explanation of the drawing

[0029] FIG. 1 is a block diagram exemplarily illustrating a situation information integration analysis system (10) using multimodal information according to one embodiment of the present invention, FIG. 2 is a drawing illustrating an edge device (100) according to an embodiment of the present invention, FIG. 3 is a drawing for exemplarily explaining the operation of a main VMS (200) according to an embodiment of the present invention, FIG. 4 is a flowchart illustrating an exemplary method for integrating situational information using multimodal information according to another embodiment of the present invention, and, FIG. 5 is a timing diagram illustrating the overall operation of a situation information integration analysis system using multimodal information according to another embodiment of the present invention. Specific details for implementing the invention

[0030] Preferred embodiments of the present invention will be described below with reference to the drawings. The embodiments of the present invention described below may be modified in various ways within the scope of the technical concept of the present invention, and it is obvious that the scope of the rights of the present invention extends to such modified embodiments.

[0031] FIG. 1 is a block diagram illustrating, in an exemplary manner, a situation information integration analysis system (10) using multimodal information according to an embodiment of the present invention. Referring to FIG. 1, the system (10) can be functionally divided into an edge layer and a central control layer. An edge device (100) is placed in the edge layer, and a main VMS (200) is placed in the central control layer.

[0032] The edge device (100) is installed adjacent to the CCTV camera (1001) and microphone (1002) and is configured to collect and analyze visual and auditory information occurring in the same space under the same time standard. This is intended to minimize time delays or event discrepancies that may occur when the video sensor and the acoustic sensor are installed physically separated.

[0033] Accordingly, it is possible to recognize the situation by taking sound information into account even in situations where video analysis is difficult due to sound-based events occurring outside the field of view of the CCTV camera (1001), or due to reduced lighting or obstruction.

[0034] The edge device (100) transmits video data to the main VMS (200) in the form of a video stream based on RTSP (Real-Time Streaming Protocol), and at the same time, it generates semantic results of events derived by comprehensively analyzing video and audio information in the form of event-related metadata and transmits them through a separate communication channel. That is, in the present invention, the edge device (100) functions not as a simple sensor relay device, but as an intelligent analysis node that performs event recognition and semantic interpretation.

[0035] The main VMS (200) combines the video stream transmitted from the edge device (100) with event-related metadata and provides it to the control screen. When a dangerous event is detected, it provides an alarm or automatically controls the field of view of the PTZ camera (1003) to the direction or location where the event occurred, thereby enabling a preemptive response before the intervention of the control officer (1004).

[0036] FIG. 2 is a diagram illustrating an edge device (100) according to an embodiment of the present invention. The edge device (100) illustrated in FIG. 2 functionally distinguishes a series of processing steps from the input of video and audio data to the generation and transmission of event-related metadata through multimodal analysis. The edge device (100) according to an embodiment of the present invention is configured to functionally include a data collection unit (110), a data preprocessing unit (120), a multimodal feature fusion unit (130), an event generation unit (140), and a communication unit (150).

[0037] The data collection unit (110) is configured to collect video data and sound data from the CCTV camera (1001) and microphone (1002), respectively, and to generate a temporally aligned input data set so that the data acquired from different sensors reflects the same event.

[0038] The data collection unit (110) includes a video collection function that receives video frames or video streams from a CCTV camera (1001) and a sound collection function that receives sound signals from a microphone (1002). At this time, video data can be input in frame units or stream units, and sound data can be acquired at a constant sampling rate and divided into window units of a length suitable for analysis.

[0039] Additionally, the data collection unit (110) includes a time synchronization function that aligns the two data on the same time axis based on timestamps assigned to the video data and audio data, respectively. Through this, the video frame and the audio signal are matched to represent events at the same point in time, and the accuracy of the multimodal fusion analysis performed in the subsequent step is ensured.

[0040] With the configuration of the data collection unit (110) as described above, a foundation is established to analyze the event in a temporally consistent manner, even in situations where video recognition is difficult due to sound-based events occurring outside the field of view of the CCTV camera (1001) or due to obstruction or reduced illumination.

[0041] The data collection unit (110) includes a dynamic automatic calibration module (111) that automatically estimates and corrects relative spatial coordinates, time offsets, or clock errors between each edge device based on environmental sound included in acoustic data collected from a plurality of edge devices.

[0042] Here, ambient sound may include non-periodic or irregular acoustic events that occur naturally in the installation environment, such as door closing sounds, car horn sounds, impact sounds, and footstep sounds, rather than calibration signals that are artificially generated separately. When such ambient sound is detected by a plurality of edge devices, the dynamic automatic calibration module (111) of the present invention calculates the relative distance relationship, direction relationship, or clock error between edge devices by analyzing the difference in arrival time, frequency characteristics, or correlation of the acoustic signals that have reached each edge device.

[0043] In particular, the dynamic automatic calibration module (111) according to the present invention performs the correction operation by utilizing acoustic data collected during normal system operation, without outputting a separate calibration signal or performing a dedicated calibration procedure. Accordingly, even if clock drift or minute changes in device position occur due to changes in the installation environment or long-term operation, the spatial and temporal consistency of the system can be continuously maintained.

[0044] The data collection unit (110) further includes a semantic masking-based privacy-preserving audio processing module (112) that selectively preserves only the semantic information of the voice without storing or transmitting the voice signal in its original state among the acoustic data. Specifically, the audio processing module (112) can perform speech-to-text (STT) processing to identify a section containing a voice signal from the input acoustic data and then convert the voice signal into text information. In addition, it can calculate an emotional state or urgency index by analyzing the intonation, speed, intensity, etc. of the voice along with the converted text. Subsequently, the audio processing module (112) prevents biometric information, such as the speaker's voice, from leaking to the outside by discarding or irreversibly deleting the original waveform data of the voice signal. The audio information provided to the control system is limited to synthesized speech (Text-to-Speech) or de-voiced audio information reconstructed based on the text information and the emotional or urgency index.

[0045] Accordingly, control personnel can audibly perceive the content and urgency of the incident, while personally identifiable information contained in the original audio is technically removed, thereby satisfying privacy protection requirements under relevant laws.

[0046] The data collection unit (110) further includes an optical-acoustic combined synchronization module (113) that performs synchronization between devices using a signal output from an active marker equipped in a microphone or microphone array. Here, the active marker may be configured to output at least one of an optical signal in a visible light communication (VLC) method or an acoustic signal in an inaudible frequency band. The CCTV camera analyzes a captured video frame or a received acoustic signal to visually or acoustically detect the signal output from the active marker.

[0047] The optical-acoustic combined synchronization module (113) uses device identification information or timing information included in the detected signal to perform device identification, automatic pairing, or clock synchronization between the CCTV camera and the microphone. Through this, time alignment between video data and audio data can be secured more precisely without relying on a network-based time synchronization method.

[0048] The data preprocessing unit (120) is configured to perform a preprocessing function for converting image data and sound data transmitted from the data collection unit (110) into a form suitable for feature extraction and event judgment.

[0049] The data preprocessing unit (120) can perform at least one of resolution adjustment, noise removal, illumination correction, or region of interest (ROI) setting on the image data. For example, the data preprocessing unit (120) can perform region of interest-based preprocessing to reduce the analysis load by lowering the resolution of the entire image, while maintaining the area where objects mainly appear at a relatively high quality. In addition, the data preprocessing unit (120) may apply illumination correction or contrast correction to ensure the stability of image feature extraction even in nighttime or backlit environments.

[0050] The data preprocessing unit (120) performs at least one of environmental noise removal, abnormal sound candidate section detection, or frequency band filtering on the acoustic data. Through this, continuous background noise such as wind noise or machine noise can be mitigated, and acoustic patterns related to events such as screaming, impact sounds, and breakage sounds can be relatively emphasized.

[0051] By configuring the data preprocessing unit (120) in this way, unnecessary variation elements included in the raw sensor data are removed, thereby improving the reliability of multimodal feature extraction and fusion analysis in subsequent steps.

[0052] The multimodal feature fusion unit (130) is configured to extract semantic features from image data and sound data that have passed through the data preprocessing unit (120), and to combine them at the feature-level to generate fusion features that reflect the event context.

[0053] The multimodal feature fusion unit (130) includes a video feature extraction function that extracts video features related to events, such as the presence of an object, human behavior, and scene changes, from video data. These video features are not limited to a single form and can be produced in various forms of expression depending on the purpose of analysis and the implementation method. For example, video features can be expressed as object detection results indicating the presence and location of a person, vehicle, or other object of interest within the input video, and in this case, may include bounding box coordinates, an object class identifier, and detection reliability for each object.

[0054] Furthermore, video features can be expressed in the form of behavior classification probabilities related to events such as intrusion, loitering, falling, and violent acts through the analysis of consecutive video frames or specific time intervals. These behavior classification probabilities are not limited to binary judgments for a single class but are calculated in the form of probability distributions for each of multiple behavior classes, thereby reflecting the uncertainty or complexity of the event.

[0055] Furthermore, image features can be generated in the form of embedding vectors that numerically represent the visual patterns of image data in a low- or high-dimensional space. These embedding vectors can be generated corresponding to individual frames, sequences of frames over a certain period, or specific object regions, and can be utilized in a form suitable for calculating distances between similar scenes or actions, clustering, or multimodal fusion in subsequent steps.

[0056] As such, image features can be expressed from at least one of the object, behavior, and scene levels, and multiple feature representations can be used in parallel or in combination, rather than being limited to a single type of feature. Through this, more sophisticated situational awareness becomes possible that reflects the context and temporal continuity of events, going beyond the simple determination of whether an object exists.

[0057] Additionally, the multimodal feature fusion unit (130) includes an acoustic feature extraction function that extracts acoustic features related to dangerous events, such as screams, impact sounds, and breakage sounds, from acoustic data. Acoustic feature extraction is not limited to a method of directly using raw acoustic signals, but can be performed by converting the acoustic signals into the time-frequency domain and then calculating features that reflect semantic patterns.

[0058] For example, acoustic features can be represented based on a mel-spectrogram generated by performing a Short-Time Fourier Transform (STFT) or an equivalent transformation on an input acoustic signal. Since these mel-spectrogram-based features include a frequency band distribution that reflects human auditory characteristics, they can effectively represent dangerous events where energy changes are prominent in specific frequency bands, such as screams or breaking sounds.

[0059] In addition, acoustic features can be calculated in the form of Mel frequency cepstrum coefficients (MFCC) or equivalent statistical acoustic feature vectors. In this case, the acoustic feature extraction function can reflect not only the acoustic characteristics of a single moment but also changes in acoustic patterns over time by including the average value, variance value, or dynamic change amount within a certain time interval. This enables feature representation that is advantageous for distinguishing between instantaneous noise and actual dangerous events.

[0060] Furthermore, acoustic features can be expressed in the form of event-specific probability distributions calculated through an acoustic event classification model. For example, the acoustic feature extraction function can calculate the probability of occurrence for each of multiple predefined risk event classes, such as screams, impact sounds, and breakage sounds, and output them as vector-based acoustic features. When these probability-based features are combined with image features in a subsequent step, they offer the advantage of quantitatively reflecting the uncertainty or complexity of an event.

[0061] In particular, the acoustic feature in the multimodal feature fusion unit (130) of the present invention is technically distinct in that it is not merely a simple acoustic recognition result, but an intermediate representation designed on the premise of combining with image features. That is, the acoustic feature is generated in a form having the same or similar temporal resolution so that it can be directly combined with time-synchronized image features, and can be utilized in a fusion logic that relatively increases the contribution of the acoustic feature in situations where image reliability is low.

[0062] As such, acoustic features can be implemented as at least one of frequency-based representations, statistical feature vectors, and probability distribution forms, and these various representation methods are utilized in a form optimized for multimodal fusion analysis to complement visual blind spots, rather than for standalone acoustic recognition.

[0063] As described above, the extracted image features and sound features are combined at the feature-level in a state where they are matched to correspond to the same time interval based on the time information matched in the data collection stage. At this time, the multimodal feature fusion unit (130) is not limited to a method of simply connecting image features and sound features in parallel, but can be configured to perform adaptive fusion that dynamically adjusts the combination weight by reflecting the reliability of each modality or the importance according to the event type.

[0064] For example, in situations where the reliability of video features is assessed as low due to reduced illumination, occlusion, or motion blur, fusion can be achieved by lowering the weight of the video feature and relatively increasing the weight of the acoustic feature. Conversely, in environments with high background noise, the reliability of acoustic features is assessed as low, so the contribution of video features can be adjusted to be relatively increased. Additionally, if the discriminability of acoustic information is relatively high depending on the type of event (e.g., sounds of breakage, impact, screaming, etc.), weights can be set so that acoustic features are reflected preferentially during the fusion process.

[0065] Through such adaptive feature-level fusion, acoustic information can function as the primary basis for judgment rather than a supplementary means, even when video information is incomplete or the camera's field of view is limited. As a result, it becomes possible to recognize dangerous events even in visual blind spots that were difficult to detect in conventional video-centric control systems, and more stable and robust situational awareness is realized by utilizing multimodal information.

[0066] The event generation unit (140) is configured to determine whether a risk event has occurred by using the multimodal fusion feature generated by the multimodal feature fusion unit (130) as input, quantitatively calculate the type and risk level of the event, and then generate the result as structured event-related metadata. At this time, the event generation unit (140) does not stop at simple binary event detection, but comprehensively considers the reliability information included in the fusion feature combining image and sound features and the temporal context of the event to derive a judgment result that can be directly utilized for decision-making and automatic control.

[0067] Specifically, the event generation unit (140) determines whether a risk event exists based on multimodal fusion features and classifies whether the event corresponds to one of a plurality of predefined risk event types, such as screaming, impact, or damage. In addition, by calculating the severity and confidence of judgment together for each event, the importance and uncertainty of the event can be expressed numerically. This quantitative evaluation can be used as a criterion for determining alarm priority or automatic response in subsequent steps.

[0068] In particular, the event generation unit (140) may be configured to determine the event as a visual blind spot-based risk event if the sound-based risk level is greater than or equal to a predetermined threshold value, even if the reliability of the image-based judgment is evaluated to be below a threshold value due to factors such as reduced illumination, obstruction, or limited visibility. This corresponds to a core judgment logic that overcomes the limitations of existing control systems that rely solely on image analysis and enables the use of sound information as the primary basis for judgment.

[0069] The event generation unit (140) structures the above judgment result into event-related metadata including an event identifier (event_id), time of occurrence (timestamp), event type (event_type), severity, judgment confidence, and location or direction information of the event occurrence. The generated event-related metadata is configured to be used as key input information for providing alarms in the main VMS (200), managing event history, and automatic field of view control of the PTZ camera, thereby allowing the multimodal analysis result to be directly linked to real-time monitoring and automatic response.

[0070] The communication unit (150) transmits video data and event-related metadata generated from the edge device (100) to the main VMS (200) and is configured to maintain the reliability of event information transmission even in the event of network state changes or failure situations. Considering that the transmission characteristics and purposes of use of the video data and event-related metadata are different, the communication unit (150) has a structure that transmits them separately using different communication methods.

[0071] Specifically, the communication unit (150) includes a video streaming function that transmits video data to the main VMS (200) in the form of an RTSP-based video stream. Since RTSP is a protocol suitable for setting up and controlling stream sessions, the main VMS (200) can reliably receive the video stream and perform playback, pause, reconnection, or quality adjustment as needed. Accordingly, the video data is transmitted in a manner optimized for providing continuous visual information and is utilized for displaying control screens and checking the status.

[0072] Meanwhile, the communication unit (150) transmits event-related metadata generated by the event generation unit (140) in real time through a communication channel independent of the video stream. The event-related metadata is control-oriented information including the type, risk level, reliability, location, or direction information of the risk event, and unlike video data, low latency and transmission reliability are particularly important. Therefore, the communication unit (150) transmits the event-related metadata using a message-based communication method so that the main VMS (200) can receive it immediately and use it for providing alarms or automatic control of the PTZ camera.

[0073] In addition, the communication unit (150) includes a failure response function to minimize the loss of event-related metadata even when a network failure occurs. For example, the communication unit (150) sequentially stores event-related metadata in a temporary storage area, and when communication is restored, it retransmits the stored metadata, thereby ensuring that records and response information regarding dangerous events are not omitted. Through this structure, the continuity of event recognition and control functions is ensured even in short-term network disconnections or congested environments.

[0074] In this way, the communication unit (150) functions as a key component that effectively links the multimodal analysis results to real-time monitoring and automatic response in the main VMS (200) by separating and processing the continuous transmission of video data and the rapid and reliable transmission of event-related metadata.

[0076] FIG. 3 is a diagram illustrating the operation of a main VMS (200) according to an embodiment of the present invention. Referring to FIG. 3, the main VMS (200) receives video streams and event-related metadata generated from edge devices (100) and functions as a central control unit that performs monitoring of risk events, provides alarms, and automatically controls PTZ cameras. In the present invention, the main VMS (200) is not a simple video display device, but is configured as a central node for real-time decision-making and automatic response based on multimodal analysis results.

[0077] The video receiving unit (210) receives an RTSP-based video stream transmitted from the edge device (100) and displays it on the control screen. At this time, since the RTSP-based stream provides continuous time information with low latency, the control personnel can check the situation at the site in near real-time.

[0078] Additionally, the video receiving unit (210) includes a stream session management function so that even if network delay or temporary disconnection occurs, the continuity of the monitoring screen can be maintained through stream reconnection or quality adjustment. In this way, the video receiving unit (210) performs the role of providing a means of visual verification of “what is actually visible.”

[0079] The metadata receiving unit (220) receives event-related metadata transmitted from the edge device and parses it into a format for internal processing within the main VMS. Since the event-related metadata is not a simple notification signal but semantic and control-oriented information including event type, risk level, reliability, time of occurrence, location or direction information, the metadata receiving unit (220) structurally interprets it and transmits it to a subsequent processing step. That is, the metadata receiving unit (220) performs the role of providing decision-making input information regarding “what event occurred, with what level of risk, and where.”

[0080] The event processing unit (230) determines whether a dangerous event has occurred based on the event-related metadata transmitted from the metadata receiving unit (220) and generates an alert for the event. At this time, the event processing unit (230) can determine the level or priority of the alert using the risk and reliability calculated by the event generating unit, thereby selectively highlighting events that require the attention of the control personnel. Additionally, the event processing unit (230) is configured to automatically link video segments before and after the time of the event occurrence and display them on the control screen. For example, by providing the situation before the event occurrence and the subsequent development together, the control personnel can intuitively grasp the context before and after the event rather than just a single moment of video. In this way, the event processing unit (230) functions as a judgment and presentation layer that converts event-related metadata into actual control actions.

[0081] The PTZ control command generation unit (240) calculates Pan, Tilt, and Zoom control values ​​of the PTZ camera (1003) using location information or direction information included in event-related metadata. Through this, the field of view of the PTZ camera (1003) can be automatically switched to the direction or location where a dangerous event is detected. At this time, the calibration unit (241) provides parameters to correct the difference between the direction information generated by the edge device (100) and the actual installation direction and coordinate system of the PTZ camera (1003). For example, even if the microphone-based direction information and the physical rotation coordinate system of the camera do not match, the calculated PTZ control value is aligned with the event location in actual space by applying the transformation relationship corrected by the calibration unit (241). With this configuration, the present invention does not merely provide information that “an event has occurred,” but enables immediate field of view response to the spatial direction where the event occurred.

[0082] The priority-based PTZ scheduler (250) is configured to determine priority based on the risk or reliability of events when multiple risk events occur simultaneously or in close proximity, and to execute PTZ control commands sequentially accordingly. This allows the event with the highest risk to be prioritized for monitoring. Meanwhile, the safety control unit (251) modifies the PTZ control commands to limit the rotation range of the PTZ camera (1003) or to bypass a pre-set privacy protection area. This is intended to prevent the possibility of excessive rotation, equipment collision, or infringement of personal information that may occur during the automatic control process. In this way, the priority-based PTZ scheduler (250) and the safety control unit (251) play a role in ensuring that automatic PTZ control can be performed stably and legally even in an actual operating environment.

[0084] FIG. 4 is a flowchart illustrating an exemplary method for integrating situational information using multimodal information according to another embodiment of the present invention. Referring to FIG. 4, the method for integrating situational information using multimodal information is performed step by step along the time axis, and each step forms a functional flow for recognizing a danger event and performing an automatic response by combining image information and audio information. Hereinafter, the method for integrating situational information according to one embodiment of the present invention is described by dividing it into steps S410 to S460.

[0085] First, in step S410, video data is received from the CCTV camera (1001) and audio data is received from the microphone (1002). The video data can be input in the form of frames or a continuous video stream, and the audio data is input as an audio signal acquired at a constant sampling rate. At this time, the video data and audio data are input information of different modalities that reflect events occurring in the same space, and are used as basic data for fusion analysis in subsequent steps.

[0086] Next, in step S420, the two data are aligned on the same time axis based on the timestamps assigned to the image data and audio data, respectively. That is, the image frame and the audio signal are temporally matched to represent events that occurred at the same time. This temporal alignment process is intended to prevent analysis errors caused by acquisition delays or time errors between the image sensor and the audio sensor, and serves as a prerequisite step for ensuring the accuracy of the subsequent multimodal feature extraction and fusion.

[0087] Next, in step S430, image features are extracted from temporally aligned image data, and acoustic features are extracted from temporally aligned acoustic data. Image features can be extracted as features reflecting visual information related to the event, such as the presence or absence of objects, human behavior, and scene changes. Meanwhile, acoustic features are extracted as features reflecting acoustic patterns related to dangerous events, such as screams, impact sounds, and sounds of destruction. The image and acoustic features extracted in this step contain semantic information of different modalities; while they may have limitations in event recognition on their own, they can express the event context more richly through fusion in subsequent steps.

[0088] Next, in step S440, comprehensive event information is generated by adaptively fusing the aforementioned image and acoustic features at the feature level while they are temporally aligned. At this time, if the image data experiences reduced illumination, occlusion, or limited visibility, weighted fusing may be performed to mutually complementarily adjust the combined weight of image and acoustic features, thereby increasing the contribution of the acoustic features. Through this adaptive fusing, acoustic-based risk events occurring in visual blind spots that are difficult to detect through image-only analysis can also be included in the comprehensive event information. Consequently, the comprehensive event information is generated as data that more stably reflects the occurrence and context of an event compared to single-modal analysis.

[0089] Subsequently, in step S450, the occurrence of a risk event, the event type, and the risk level are determined based on the comprehensive event information, and an alert regarding the risk event is provided according to the determination result. At this time, the event type may be classified as any one of predefined risk events, such as screaming, impact, or damage, and the risk level may be calculated as an indicator representing the severity of the incident. Furthermore, if a risk event is determined, video data corresponding to predetermined time intervals before and after the time of the event occurrence is automatically linked and displayed on the control screen, thereby enabling control personnel to intuitively grasp the context before and after the incident.

[0090] Finally, in step S460, the pan, tilt, and zoom of the PTZ camera (1003) are automatically controlled using location information or direction information included in the comprehensive event information or event-related metadata generated based thereon. Through this, the field of view of the PTZ camera is automatically switched to the direction or location where the dangerous event occurred. In addition, if multiple dangerous events occur simultaneously, the priority of PTZ camera control can be determined based on the risk level or judgment reliability of the events, and control commands can be executed sequentially. This automatic field of view control enables a rapid visual response prior to manual operation by a control officer.

[0091] According to the situation information integration analysis method using multimodal information configured in this way, visual blind spots can be effectively compensated for by aligning video data and audio data temporally and fusing them at the feature level, and real-time situation recognition and response performance can be improved through the provision of warnings for dangerous events and automatic control of the PTZ camera (1003).

[0093] FIG. 5 is a timing diagram illustrating the overall operation of a situation information integration analysis system using multimodal information according to another embodiment of the present invention. Referring to FIG. 5, the signal processing and control flow in the situation information integration analysis system (10) using multimodal information proceeds step by step along the time axis, and each step can be divided into internal processing of the edge device, communication between the edge device and the main VMS, and control and automatic control processing in the main VMS.

[0094] First, in step T0, a video frame or video stream is input from the CCTV camera (1001) to the edge device (100), and simultaneously, an audio signal is input from the microphone (1002). At this time, the video data and audio data are acquired from different sensors, but each data is assigned a timestamp indicating the time of acquisition and processed to reflect the same event in subsequent steps. The input of video data and audio data occurs substantially simultaneously and is used as basic data for analyzing events that occurred in the same space in a temporally consistent manner.

[0095] Subsequently, in the T1 stage, the edge device (100) aligns the input video data and audio data on the same time axis. That is, based on the timestamp assigned to each data, the video frame and the audio signal are matched to represent an event at the same point in time. In parallel with this, video preprocessing such as resolution adjustment, noise removal, and illumination correction is performed on the video data, and audio preprocessing such as environmental noise removal and abnormal sound candidate segment detection is performed on the audio data. Processing up to this stage is performed internally within the edge device (100) independently of the main VMS (200) and corresponds to a preparatory stage to ensure the accuracy of subsequent multimodal analysis.

[0096] Next, in the T2 stage, the edge device (100) extracts image features related to the presence of objects, human behavior, scene changes, etc. from the preprocessed image data, and extracts acoustic features related to screams, impact sounds, breakage sounds, etc. from the preprocessed acoustic data. The extracted image features and acoustic features are combined at the feature level while being temporally aligned, and in this process, if reduced illumination, occlusion, or limited visibility occurs, weighted combination can be performed to increase the proportion of combined acoustic features. Through this adaptive fusion, event recognition becomes possible even in visual blind spots that are difficult to recognize through image-only analysis, and comprehensive event information is generated at this stage.

[0097] In the T3 stage, event-related metadata is generated based on the comprehensive event information. The event-related metadata includes information on whether an event has occurred, the type of event, the risk level, the time of occurrence, and the location or direction of the event occurrence, and is organized into a structured information form for monitoring and automatic control in the main VMS (200).

[0098] Subsequently, the T4 stage serves as a communication branching point, where signal transmission from the edge device (100) to the main VMS (200) proceeds in parallel via two paths. In one path, video data is transmitted continuously in the form of an RTSP-based video stream, while in the other path, event-related metadata is transmitted discontinuously at the time of event occurrence through separate message-based communication. Although these two signal streams correspond to the same event in terms of time, they are processed independently in terms of logic.

[0099] In the T5 stage, the main VMS (200) receives an RTSP-based video stream transmitted from the edge device (100) and displays it on the control screen, and simultaneously receives event-related metadata and parses it into an internal processing format. The video stream is displayed continuously, and the event-related metadata functions as a trigger to initiate subsequent control processing.

[0100] In the subsequent T6 step, the main VMS (200) analyzes the received event-related metadata to confirm the risk event and generates an alert for the event. Additionally, by automatically linking the video segments before and after the time of the event occurrence and displaying them on the control screen, the control personnel can intuitively grasp the context of the incident.

[0101] In the T7 step, the main VMS (200) analyzes location information or direction information included in event-related metadata to determine whether to control the PTZ camera (1003). If multiple risk events occur simultaneously, the priority is determined based on the risk level or reliability of the events, and the scope and order of automatic control are adjusted by checking whether there is a privacy protection area.

[0102] Finally, in the T8 step, the main VMS (200) calculates the pan, tilt, and zoom control values ​​of the PTZ camera (1003), applies the calibration values, and then transmits the PTZ control command to the PTZ camera (1003). Accordingly, the field of view of the PTZ camera (1003) is automatically switched to the direction or location where the dangerous event occurred, and the automatic response is completed before manual operation by the control officer.

[0103] Through such a signal flow and control processing structure, the present invention can simultaneously realize rapid alerting and automatic field of view control based on event-related metadata while maintaining the continuity of the video stream, and enables real-time situational awareness and response in various environments, including visual blind spots. Explanation of the symbols

[0104] 10: Context Information Integration Analysis System Using Multimodal Information 100 : Edge device 110: Data Collection Unit 120 : Data Preprocessing Section 130 : Multimodal feature fusion part 140 : Event creation section 150 : Communications Department 200 : Main VMS 210 : Video receiver 220 : Metadata receiver 230 : Event handling section 240 : PTZ control command generation unit 250 : PTZ Scheduler 1001 : CCTV camera 1002 : Microphone 1003 : PTZ camera

Claims

Claim 1 An edge device installed adjacent to a CCTV camera and a microphone, receiving video data from the CCTV camera and audio data from the microphone, generating event-related metadata by analyzing the video data and audio data in time, and transmitting the video data as a Real-Time Streaming Protocol (RTSP)-based video stream;The system includes a main VMS that receives the RTSP-based video stream and event-related metadata transmitted from the edge device and performs a control function. The edge device is configured to generate comprehensive event information including an acoustic-based risk event occurring in a visual blind spot that is difficult to detect through video-only analysis, by extracting video features from video data and acoustic features from acoustic data, and then adaptively fusing the video features and the acoustic features at the feature-level in a temporally aligned state. The main VMS is configured to provide an alert for the risk event based on the comprehensive event information and to automatically control the field of view of a PTZ (Pan-Tilt-Zoom) camera to the direction or location where the risk event occurred based on location information or direction information included in the event-related metadata, thereby compensating for the visual blind spot using acoustic information and enabling real-time response to the risk event. The edge device includes a data collection unit that aligns the video data and the acoustic data on the same time axis based on timestamps assigned to the video data and acoustic data, respectively. The data collection unit includes a plurality of edge devices installed at mutually spaced locations. A dynamic automatic calibration module that automatically estimates and corrects relative spatial coordinates, time offsets, or clock errors between edge devices based on ambient sounds included in acoustic data collected from devices, without a separate calibration signal or dedicated calibration procedure; and a semantic masking-based privacy-preserving audio processing unit that does not store or transmit voice signals among acoustic data in their original state, but converts the voice into text information and emotion or urgency indicators, discards the original voice, and generates reconstructed synthetic voice or de-voiced audio information based on the text information and emotion or urgency indicators.A situation information integration analysis system using multimodal information, further comprising: an optical-acoustic composite synchronization module that performs device identification, pairing, or clock synchronization between the CCTV camera and the microphone by visually or acoustically receiving a visible light communication (VLC) signal or an inaudible frequency signal output from an active marker provided in a microphone or microphone array. Claim 2 delete Claim 3 delete Claim 4 A situation information integration analysis system using multimodal information according to claim 1, characterized in that the comprehensive event information is configured such that, after converting the spatial coordinates of an event estimated through image analysis and the sound source direction or location vector estimated through sound analysis into the same coordinate system, the geometric relationship value regarding at least one of the distance, angle, or intersection between the spatial coordinates and the sound source direction or location vector is within a preset standard range, the event is determined to be a dangerous event. Claim 5 A situation information integration analysis system using multimodal information, wherein, in claim 1, the edge device comprises a data preprocessing unit that performs resolution adjustment, noise removal, or illumination correction for image data, and environmental noise removal or abnormal sound candidate section detection for acoustic data. Claim 6 A situation information integration analysis system using multimodal information, wherein, in claim 1, the edge device includes a multimodal feature fusion unit that dynamically adjusts the combined weight of the image feature and the acoustic feature according to the reliability of the image feature or the reliability of the acoustic feature. Claim 7 A situation information integration analysis system using multimodal information, wherein, in claim 1, the edge device includes an event generation unit that determines the event as a visual blind spot-based risk event when the sound-based risk level is greater than or equal to the reference value, even if the reliability of the image-based judgment is less than or equal to the reference value. Claim 8 A situation information integration analysis system using multimodal information, wherein, in claim 1, the edge device includes a communication unit that transmits video data to the RTSP-based video stream and transmits event-related metadata through a communication channel independent of the video stream. Claim 9 A situation information integration analysis system using multimodal information according to claim 1, wherein the main VMS determines the priority of PTZ camera control according to the risk or reliability of the events when multiple risk events occur simultaneously, and corrects PTZ control commands to avoid a privacy protection area. Claim 10 A method for integrated analysis of situational information using multimodal information comprises: receiving video data from a CCTV camera and receiving audio data from a microphone; aligning the video data and the audio data on the same time axis based on timestamps assigned to the video data and the audio data, respectively; extracting video features related to the presence, behavior, or scene change of an object from the temporally aligned video data, and extracting audio features related to screams, impact sounds, or breakage sounds from the temporally aligned audio data; adaptively fusing the video features and the audio features at the feature-level in a temporally aligned state to generate comprehensive event information; determining whether a risk event has occurred, the type of event, and the level of risk based on the comprehensive event information, and providing an alert for the risk event according to the determination result. The method includes a step of automatically controlling the pan, tilt, and zoom of a PTZ (Pan-Tilt-Zoom) camera according to the comprehensive event information or the location information or direction information included in the event-related metadata generated based thereon, thereby switching the camera's field of view to the direction or location where the danger event occurred; the matching step comprises a dynamic automatic calibration step that automatically estimates and corrects relative spatial coordinates, time offsets, or clock errors between each edge device based on ambient sounds included in acoustic data collected from a plurality of edge devices installed at mutually spaced locations, without a separate calibration signal or dedicated calibration procedure; and a step of semantic masking-based privacy-preserving audio processing that does not store or transmit voice signals among the acoustic data in their original state, converts the voice into text information and emotion or urgency indicators, discards the original voice, and generates reconstructed synthetic voice or de-voiced audio information based on the text information and emotion or urgency indicators.A method for integrating situational information using multimodal information, further comprising an optical-acoustic composite synchronization step in which the CCTV camera visually or acoustically receives a visible light communication (VLC) signal or an inaudible frequency signal output from an active marker provided in a microphone or microphone array, thereby performing device identification, pairing, or clock synchronization between the CCTV camera and the microphone. Claim 11 A method for integrating situational information using multimodal information, wherein the step of adaptively fusing at the feature-level includes, when low illumination, occlusion, or limited field of view occurs in the image data, a step of weighted combining the image feature and the acoustic feature to mutually complementarily adjust the combined weight of the image feature and the acoustic feature so as to increase the contribution of the acoustic feature, thereby including acoustic-based risk events occurring in visual blind spots that are difficult to detect by image-only analysis in the comprehensive event information. Claim 12 In claim 10, the step of providing an alert for the above-mentioned risk event includes the step of automatically linking video data corresponding to a predetermined time interval before and after the time of event occurrence based on the time of event occurrence included in the comprehensive event information or event-related metadata generated based thereon, and displaying it on a control screen; and the step of automatically controlling the field of view of the above-mentioned PTZ camera includes the step of calculating Pan, Tilt, and Zoom control values ​​of the PTZ camera based on location information or direction information included in the event-related metadata, and determining the priority of the PTZ camera control according to the risk level or judgment reliability of the events when multiple risk events occur simultaneously, and executing them sequentially, characterized in that the method for integrated analysis of situation information using multimodal information.

Images