Method and System for Evaluating Risk Scores and Risk Ratings Based on Composite Situations in Video Data

The method and system leverage AI models to extract and analyze multiple visual features from video data, inferring complex situations and calculating risk scores, addressing limitations of conventional image analysis by enhancing adaptability and reliability in risk assessment.

KR103043636B1Active Publication Date: 2026-11-11ICTWAY
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
KR1020260015384
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Filing Date
2026-01-26
Publication Date
2026-11-11
Estimated Expiration
2046-01-26

AI Technical Summary

Technical Problem

Conventional image analysis technologies struggle to comprehensively interpret complex situations involving multiple objects, actions, and spatiotemporal information, relying on predefined rules and fixed features, limiting real-time performance, reliability, scalability, and adaptability to new situations.

Method used

A method and system using artificial intelligence models to extract multiple visual features from video data, infer complex situations through semantic and contextual analysis, and calculate risk scores and grades, adaptable to various environments without predefined rules, with automatic generation of structured text for risk assessment.

Benefits of technology

Enhances situation understanding, improves usability and adaptability, ensures transparency and reliability, and supports efficient response prioritization by providing structured text outputs for risk assessment in real-time applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112026010852558-PAT00001_ABST
    Figure 112026010852558-PAT00001_ABST
Patent Text Reader

Abstract

The present invention relates to a method and system for calculating a risk score and a risk grade by analyzing complex situations from image data. More specifically, the invention relates to a risk assessment technology that extracts multiple visual features, such as objects, actions, backgrounds, and environmental information, from input images of a multi-CCTV system, infers complex situations including interactions between objects and spatiotemporal information using said visual features through an artificial intelligence model, and calculates a risk score and a risk rating based on the inference results.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The present invention relates to a method and system for calculating a risk score and risk rating based on complex situations within video data. More specifically, it relates to a risk assessment technology that extracts multiple visual features, such as objects, behaviors, backgrounds, and environmental information, from input video of a multi-CCTV system, infers a complex situation including interactions between objects and spatiotemporal information using said visual features through an artificial intelligence model, and calculates a risk score and risk rating based on the inference results. Background Technology

[0003] Conventional image analysis technologies have evolved primarily around techniques for identifying single elements or individual situations occurring within an image, such as object detection, action recognition, and event detection. These technologies are optimized for analyzing local features, such as the presence of specific objects, abnormal behavior, or changes in movement, and are mostly designed to operate according to predefined purposes or specific rules.

[0004] However, existing methods have structural limitations in interpreting situations where multiple objects and actions coexist simultaneously within a video, and where spatiotemporal information is complexly combined, consisting of interactions such as changes in the relative positions of objects, approaches, departures, or intersections resulting from changes in movement paths, surrounding environmental factors such as illumination, congestion, and spatial structure, as well as changes in the state of objects and the duration of actions observed over time.

[0005] Furthermore, conventional technology has limitations in accurately understanding and explaining complex situations within images, often relying on direct human judgment or rule-based systems to calculate risk levels. However, since these rule-based methods are dependent on predefined conditions and scenarios, they struggle to respond flexibly to new objects, behaviors, or environmental changes, and suffer from the limitation that the complexity of system maintenance and management increases as the number of rules grows.

[0006] Furthermore, conventional statistical-based approaches rely on fixed features and model structures, making it difficult to adequately reflect high-dimensional semantic information such as spatiotemporal context or interactions between objects. This limits the ability to generalize to unexpected situations and creates fundamental limitations in implementing automated risk assessment in terms of real-time performance, reliability, scalability, and diversity.

[0007] The matters described above as background technology are intended only to enhance understanding of the background of the present invention and should not be construed as an acknowledgment that they constitute prior art already known to those skilled in the art. The problem to be solved

[0009] The present invention has been devised to solve the problems of the aforementioned prior art, and aims to provide a method and system for calculating risk scores and risk grades based on complex situations that enable faster understanding of complex situations by comprehensively interpreting various visual features within an input image for complex situations that use multiple objects, actions, surrounding environmental factors, and spatiotemporal information as inputs, and by inferring semantically and contextually through the fusion of relationships between visual features into complex relationships between objects, actions, interactions between objects, environmental information, and spatiotemporal information.

[0010] In addition, another objective of the present invention is to provide a method and system that can support a user in intuitively understanding the risk level of a situation and efficiently determining response priorities by calculating a risk score as a numerical value based on the inference results of a complex situation and classifying the risk score into multiple risk grades according to predefined criteria or thresholds.

[0011] In addition, another objective of the present invention is to provide a method and system having an artificial intelligence-based analysis structure that can flexibly adapt to various environments and application fields without relying on predefined rules or domain-specific scenarios.

[0012] In addition, another objective of the present invention is to provide a method and system that significantly improves efficiency and usability in subsequent processing steps, such as monitoring log recording, automatic generation of situation reports, and database linkage, by automatically generating inference results and risk assessment information regarding complex situations in a structured text format that can be automatically read by external analysis devices, etc.

[0013] The technical problems that the present invention aims to solve are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by those skilled in the art from the description of the present invention. means of solving the problem

[0015] According to one aspect of the present invention for solving the problems of the aforementioned prior art, a method for calculating a risk score and risk grade based on a complex situation within video data is provided, comprising: a video input step in which a video of a multi-CCTV system is input to a visual feature extraction module; a step in which the visual feature extraction module analyzes the input video and extracts visual features including object, action, and environmental information within the input video; a step in which a situation inference module derives interaction between objects and spatiotemporal information based on the visual features, semantically and contextually infers a complex situation formed by combining objects, actions, interaction between objects, environmental information, and spatiotemporal information, and generates situation-aware inference information including a natural language-based description of the complex situation; a step in which a risk assessment module identifies risk factors based on semantic and contextual content included in the situation inference information and calculates a risk score and risk grade based on the risk factors; and a step in which a text generation module generates judgment result information including the situation inference information, risk score, and risk grade in a structured text format.

[0016] In the present invention, the visual feature extraction step, the situation inference information generation step, or the risk score and risk grade calculation step is preferably performed using one or more models among a large-scale language model (LLM), a visual language model (VLM), or a multimodal inference model (MRM), which take an input image or visual feature and an instruction prompt as input.

[0017] In the present invention, the visual feature extraction step preferably involves an image language model performing object recognition, behavior recognition, and surrounding environment recognition through a single inference process using features related to objects, behaviors, and environment information based on an input image and a command prompt.

[0018] In the present invention, the visual feature extraction step preferably uses a zero-shot or few-shot based instruction prompt to extract information about objects, behaviors, or environments that are not previously defined, thereby adaptively interpreting new, undefined situations.

[0019] In the present invention, the situation inference information generation step preferably generates situation inference information for a complex situation that reflects the interaction between objects, the meaning of actions, and the spatiotemporal context, by using visual features extracted from an input image and a Chain-of-Thought prompt that instructs the sequential execution of steps such as object identification, distance analysis between objects, and behavioral intention inference.

[0020] In the present invention, the step of calculating the risk score and risk grade preferably comprises: a step of calculating a normalized value for each of the plurality of risk factors included in the complex situation, and calculating a risk score by combining the normalized values ​​for each risk factor with a predetermined weight using a mathematical operation method; and a step of determining a risk grade by mapping the risk score to one of a plurality of threshold intervals that are predefined or determined according to setting information.

[0021] In the present invention, the risk factors preferably include proximity between objects within a complex situation, duration of behavior, rate of change of behavior, environmental risk, and time sensitivity.

[0022] In the present invention, the risk score is calculated as a continuous value in the range of 0 to 100, the weights are set so that their sum is 1, and it is preferable that the risk score is calculated by a weighted sum operation.

[0023] The present invention preferably further includes a step of adjusting the risk level when the risk score is maintained above a predetermined threshold value for a predetermined period of time or longer, when the risk score rises above a predetermined rate of change over time, when the same type of risk factor occurs repeatedly, when the security level of the area where the image was captured is relatively high, when the difference in illumination changes rapidly, or when the sunset time has elapsed.

[0024] In the present invention, the structured text generation step preferably generates judgment result information including reasoning about the complex situation, supporting grounds for reasoning about the complex situation, risk score, and risk grade as structured text, and outputs it in a structured text format such as JSON, XML, CSV, or YAML.

[0025] According to another aspect of the present invention for solving the problems of the aforementioned prior art, a system for calculating risk scores and risk grades based on complex situations within video data is provided, comprising: a processor configured to analyze objects, actions, interactions between objects, environmental information, and spatiotemporal information using an artificial intelligence model from video collected from a multi-CCTV system; wherein the processor comprises: a visual feature extraction module that extracts visual features including objects, actions, and environmental information from video input from a multi-CCTV system; a situation inference module that derives interactions between objects and spatiotemporal information using the visual features as input, semantically and contextually infers a complex situation formed by combining objects, actions, interactions between objects, environmental information, and spatiotemporal information, and derives situation inference information including a natural language-based explanation of the complex situation; a risk evaluation module that receives the situation inference information, identifies risk factors based on semantic and contextual content included in the situation inference information, and calculates a risk score and a risk grade based on the risk factors; and a text generation module that generates judgment result information derived based on the situation inference information, risk score, and risk grade in a structured text format.

[0026] In the present invention, the visual feature extraction module, the situation inference module, and the risk assessment module preferably include one or more models among a large-scale language model (LLM), a visual language model (VLM), or a multimodal inference model (MRM) that process an input image or visual feature and an instruction prompt as input.

[0027] In the present invention, the processor further includes an inference orchestration engine that performs call order control, input-output mapping, state management, error handling, and prompt configuration for a large-scale language model (LLM), a visual language model (VLM), and a multimodal inference model (MRM); it is preferable to configure a single analysis pipeline using the inference orchestration engine and to cross-validate step-by-step inference results by applying a hierarchical inference model. Effects of the invention

[0029] According to the present invention, since multiple visual features such as objects, actions, backgrounds, and environmental information are extracted from an input image and complex situations can be inferred semantically and contextually based thereon, the limitations of situation understanding inherent in conventional single-event-based image analysis technology can be effectively overcome. Consequently, there is an effect of more precisely interpreting multi-layered situations in which interactions between objects, spatiotemporal relationships, and environmental conditions are complexly combined.

[0030] According to the present invention, since it is possible to simultaneously provide supporting evidence for the semantic and contextual interpretation / inference of complex situations based on an artificial intelligence model, it has the effect of ensuring transparency, reliability, and explainability of the analysis process.

[0031] According to the present invention, by utilizing various artificial intelligence models such as large-scale language models (LLM), visual language models (VLM), and multimodal inference models (MRM), and by securing zero-shot or few-shot-based generalization capabilities, it is possible to adaptively respond to new objects, behaviors, and environmental changes without relying on predefined rules or scenarios specific to a particular application field, and accordingly, the universality and scalability of the system are significantly improved.

[0032] According to the present invention, since risk scores and risk grades can be automatically calculated using inference results (situation inference information) regarding complex situations, it is possible to overcome the limitations of existing statistical or rule-based methods and to perform risk assessment by reflecting semantic and contextual information of complex situations in the risk score calculation process. This reduces the burden of risk judgment on the user and has the effect of allowing for the more systematic setting of priorities for risk response.

[0033] According to the present invention, judgment result information derived based on complex situation descriptions, risk scores and risk grades, situation inference information, etc., can be automatically generated in natural language or structured text such as JSON or XML, thereby significantly improving usability in various post-processing modules such as control log recording, report automation, database linkage, integration with external analysis systems, and alarm system triggers.

[0034] According to the present invention, since the entire process from image-based complex situation analysis to risk assessment, text generation, and post-processing is automated, it provides high effectiveness in various application fields requiring real-time response, such as control environments, industrial safety, facility monitoring, traffic management, and smart city security. In particular, it has the effect of simultaneously ensuring the speed, consistency, and accuracy of analysis while reducing reliance on human resources. Brief explanation of the drawing

[0036] FIG. 1 is a flowchart of a method for calculating a risk score and risk grade based on complex situations according to an embodiment of the present invention. FIG. 2 is a configuration diagram of a complex situation-based risk score and risk grade calculation system according to an embodiment of the present invention. FIG. 3 is an exemplary diagram showing the operation of an inference orchestration engine according to an embodiment of the present invention. FIG. 4 is an example diagram of the configuration of an inference orchestration engine according to an embodiment of the present invention. FIGS. 5 and 6 are exemplary diagrams of a report including situation inference information, risk score, and risk grade according to an embodiment of the present invention. Specific details for implementing the invention

[0037] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings. Prior to this, terms and words used in this specification and claims should not be interpreted as being limited to their ordinary or dictionary meanings, but should be interpreted in a meaning and concept consistent with the technical spirit of the present invention, based on the principle that the inventor can appropriately define the concept of the terms to best describe his invention. Accordingly, the embodiments described in this specification and the configurations illustrated in the drawings are merely the most preferred embodiments of the present invention and do not represent all of the technical spirit of the present invention; therefore, it should be understood that various equivalents and modifications capable of replacing them may exist at the time of filing this application.

[0038] As mentioned above, conventional image analysis technologies have evolved by focusing on single elements or fragmentary situational awareness, such as object detection, behavior recognition, or event detection. While these conventional technologies are effective for detecting the presence of specific objects or single abnormal behaviors within an image, they have structural limitations in comprehensively interpreting composite situations involving interactions between multiple objects, environmental factors, and state changes over time.

[0039] Accordingly, the present invention relates to the field of image data-based situation recognition and risk assessment technology, and is conceived by focusing on a method of extracting multiple visual features equipped with object information, behavior information, background information, and environment information from an input image, inferring a complex situation by analyzing the interaction between the visual features using various artificial intelligence models, and calculating a risk score and risk grade based on the inference result.

[0040] More specifically, the present invention inputs video of a multi-CCTV system (200) in real-time or non-real-time into a visual feature extraction module (110) of a processor (100), and the visual feature extraction module (110) extracts visual features having object information, behavior information, background information, and environment information within the input video. Based on such visual features, a situation inference module (120) generates situation inference information that describes a complex situation including interactions between objects and spatiotemporal context in natural language. A risk assessment module (130) identifies risk factors from the situation inference information and calculates a risk score and a risk grade. A text generation module (140) automatically generates judgment result information based on the situation inference information, risk score, and risk grade as structured text. The structured text information is transmitted to an external analysis device / external terminal (300) including a control system and an alarm system to perform post-processing such as alarm generation and automated report generation.

[0041] First, in order to explain the technical concept of the present invention more clearly, the terms applied to the present invention shall be defined as follows.

[0042] In the present invention, “object information” means information regarding a person, vehicle, object, or individual entity functionally equivalent thereto identified or recognized within an image or image data, and includes at least one of the object’s type, location coordinates, size, direction, speed of movement, trajectory of movement, or a combination thereof.

[0043] In the present invention, “action information” refers to information indicating a state change or motion pattern of an object over time, and includes at least one of a single action or a complex action such as movement, stopping, approach, departure, contact, collision, change of posture, formation or dissolution of a cluster.

[0044] In the present invention, “environmental information” refers to information regarding external conditions in which an object or action occurs, and includes at least one of illuminance, weather conditions, the presence or absence of obstacles, spatial congestion, visibility obstruction, or similar physical or environmental factors; environmental information is a concept included in spatiotemporal information. For reference, the background information is information regarding the type of space in which an object exists or an action occurs, and in the present invention, the background information may be included within the environmental information as needed by the invention.

[0045] In the present invention, “spatiotemporal information” refers to information that indicates how object information, behavior information, and environment information change or are interrelated on the time axis and the space axis, and includes information such as changes in distance between objects, relative positional relationships between objects, movement paths of objects, dwell times, duration of behaviors, or combinations thereof.

[0046] In the present invention, the term “composite situation” refers to a state formed by combining two or more object information, behavior information, environment information, and spatiotemporal information simultaneously or sequentially in time, and means a semantic state or event interpreted based on the interaction and change patterns of said object information, behavior information, and spatiotemporal information, and includes a situation that cannot be sufficiently explained by a single object or a single behavior alone.

[0047] In the present invention, “situation-aware inference information” refers to a technical analysis result inferred by considering together at least two of the following: changes in relative distance between objects, movement paths, dwell times, sequence of actions, and spatiotemporal conditions (including environmental conditions), based on a complex situation which is a state formed by combining two or more object information, action information, interactions between objects, environmental information, and spatiotemporal information simultaneously or sequentially on a time axis or a space axis.

[0048] In the present invention, the term “semantic context” refers to the result of converting the relationships, order, or causality between elements constituting a complex situation into a structured expression that can be interpreted by an artificial intelligence model.

[0049] In the present invention, the term “risk factor” refers to a factor that influences the possibility of risk occurrence or the level of risk within a complex situation, and includes at least one of the following: proximity between objects, sudden change in behavior, environmental constraints, spatiotemporal constraints, possibility of human casualties, possibility of situation spread, or a combination thereof.

[0050] In the present invention, the term “risk score” refers to a quantitative numerical value calculated based on object information, behavior information, environmental information, and spatiotemporal information extracted from an image, and means a value calculated by weighted summing of normalized risk indicators for each of a plurality of risk factors. The risk score can be calculated as a continuous value in the range of 0 to 100 or a normalized value in the range of 0 to 1, and is a technical indicator for numerically expressing the relative magnitude of the risk level.

[0051] In the present invention, the term “risk rating” refers to the result of dividing the risk score into multiple threshold intervals that are predefined or dynamically set to classify the risk level into multiple stages.

[0052] In the present invention, “active threat” means a state in which one or more risk factors within a complex situation are continuously maintained at the present time or are strengthened over time and have the potential to transition into an additional risk situation.

[0053] In the present invention, the term “environmental / social impact” refers to an element indicating the degree of potential impact that a complex situation may have on the surrounding environment, public space, or a large number of people.

[0054] In the present invention, the term “causality of action” refers to a causal relationship in which a subsequent action or event occurs after a certain period of time has elapsed or specific conditions have been met, following a preceding action or a change in the state of an object, and may include, for example, a relationship in which a collision action occurs after a decrease in the distance between objects.

[0055] In the present invention, “time sensitivity” refers to an indicator indicating the possibility that a risk factor in a complex situation may increase rapidly over time or transition to a deterioration of the situation within a short period of time.

[0056] FIG. 1 is a flowchart of a method for calculating a risk score and risk grade based on complex situations according to an embodiment of the present invention. FIG. 1 illustrates a series of processing steps leading from an image input step, a visual feature extraction step, a situation inference step, a risk assessment step, a text generation step, and a transmission to an external system and post-processing step.

[0057] The risk calculation method according to the present invention may comprise: a step of analyzing an input image to extract visual features including object information, behavior information, and environmental information; a step of inferring a complex situation by comprehensively considering the interaction between objects, the temporal flow of behavior, and the spatial arrangement based on the visual features; a risk evaluation step of identifying risk factors using semantic and contextual expressions included in the description and inference basis of the complex situation as input, and calculating a risk score and a risk grade based thereon; and a step of outputting the calculated result in the form of natural language or structured text.

[0058] The above visual feature extraction, situation inference, and risk assessment can be performed using artificial intelligence models such as Large Language Models (LLM), Visual Language Models (VLM), and Multimodal Inference Models (MRM), along with specific instruction prompts, and can integrate object recognition, behavior recognition, and scene understanding in a single inference process.

[0059] FIG. 2 is a configuration diagram of a complex situation-based risk score and risk grade calculation system according to an embodiment of the present invention.

[0060] The system of the present invention includes a processor (100) configured to analyze objects, behaviors, interactions between objects, environmental information, and spatiotemporal information using an artificial intelligence model from images collected from multiple CCTV systems (200).

[0061] The processor (100) comprises: a visual feature extraction module (110) that extracts visual features including object, action, and environment information from video input from a multi-CCTV system (200); a situation inference module (120) that derives interaction between objects and spatiotemporal information using the visual features as input, semantically and contextually infers a complex situation formed by combining objects, actions, interaction between objects, environment information, and spatiotemporal information, and derives situation inference information including a natural language-based explanation of the complex situation; a risk evaluation module (130) that receives the situation inference information, identifies risk factors based on semantic and contextual content included in the situation inference information, and calculates a risk score and a risk grade based on the risk factors; and a text generation module (140) that generates judgment result information derived based on the situation inference information, risk score, and risk grade in a structured text format.

[0062] A method for calculating a risk score and risk grade based on a complex situation is performed using the system of the present invention as described above. This method includes a series of processing steps, such as extracting multiple visual features including object information, behavior information, and environmental information (including background information) from an input image, inferring a complex situation by synthesizing the interaction between multiple objects and spatiotemporal information based on the visual features, calculating a risk score and risk grade using the situation inference information resulting from the complex situation inference, and outputting the calculated risk score, risk grade, and situation inference information in the form of natural language or structured text. The method for calculating a risk score and risk grade based on a complex situation will be described in detail below for each step.

[0063] Referring to FIG. 1, first, the present invention undergoes a video input step (S10) in which images are input to a processor (100) in real-time or non-real-time from a multi-CCTV system (200).

[0064] The above multi-CCTV system (200) may be composed of a fixed CCTV, PTZ camera, vehicle camera, wearable camera, drone camera, or industrial camera, and does not necessarily mean only a system including multiple CCTVs.

[0065] Communication between the above-mentioned multi-CCTV system (200) and the processor (100) can transmit video using known wired or wireless communication methods.

[0066] Next, the present invention undergoes a visual feature extraction step (S20).

[0067] The visual feature extraction module (110) of the present invention derives a plurality of visual features including object information, behavior information, and environment information from an image input from a multiple CCTV system (200) (S20).

[0068] The above visual feature extraction step can be performed on an input image using one or more models among an image language model, a multimodal-based feature extraction model, an object recognition model, an action recognition model, or a scene understanding model included in the visual feature module (110).

[0069] The visual features derived by the above visual feature module (110) may include object information, behavior information, background information, and environment information. That is, the visual feature module (110) extracts object information regarding the location, type, and movement path of an object, information on the operation or behavior of an object, and environment information including illumination, congestion, and the presence of obstacles, as well as background information, from an input image.

[0070] As described above, the object information herein is information regarding a person, vehicle, animal, or object, and the environment information comprises physical and environmental factors such as illumination, weather conditions, the presence of obstacles, spatial congestion, and visibility obstruction; and the background information may be information regarding the type of space where an object exists or an action occurs. Although the background information is described separately from the environment information, as described above, the background information can be considered as information included within the environment information.

[0071] When the above visual feature module (110) includes a Vision-Language Model (VLM), object recognition, action recognition, and scene understanding can be performed integrally in a single inference process by a single dynamic instruction prompt, and a high-dimensional semantic representation combining natural language embeddings and image embeddings can be generated.

[0072] Next, the present invention undergoes a situation inference information generation step (S30).

[0073] The above-mentioned situation inference information generation step is a step in which the situation inference module (120) uses the above-mentioned visual features as input to comprehensively consider changes in relationships or interactions between objects, the temporal flow of actions, the spatial arrangement around objects, and environmental conditions around objects to semantically and contextually infer a complex situation, and then generates it as situation inference information.

[0074] That is, the above-mentioned situation inference module (120) takes as input spatiotemporal information consisting of interactions such as approach / exit / crossing based on changes in relative positions between objects and changes in movement paths based on the above-mentioned visual features, surrounding environmental factors such as illumination / congestion / spatial structure, and changes in the state of objects and duration of actions observed over time, and infers a complex situation based on the input information semantically and contextually through an artificial intelligence model such as a large-scale language model (LLM), a visual language model (VLM), or a multimodal inference model (MRM), and generates situation-aware inference information including a natural language-based explanation of the complex situation.

[0075] The above situation inference module (120) can be performed using one or more of a large-scale language model (LLM), a visual language model (VLM), or a multimodal inference model (MRM), and can adaptively respond to new situations through zero-shot or few-shot-based generalization capabilities. In particular, through zero-shot or few-shot-based generalization capabilities, it can perform the function of adaptively analyzing without adding separate rules even when new objects, behaviors, or situations occur.

[0076] The generation of situation inference information by the above situation inference module (120) can be subdivided into stepwise functions of interaction between objects, interpretation of object behavior, understanding of spatiotemporal context, analysis of environmental information, and generation of inference results and supporting grounds for inference.

[0077] First, the interaction analysis function between objects of the situation inference module (120) can analyze the proximity between objects, relative positional relationships, distance changes, and movement directions to identify the spatial relationships and interaction patterns between multiple objects. At this time, the reduction in distance between objects, repeated approach and departure, or the possibility of collision can be considered as factors for determining the characteristics of the interaction.

[0078] And, through the behavior interpretation function of the situation inference module (120), single or complex behavior patterns such as movement, stopping, approach, departure, contact, or collision of an object can be identified, and a specific behavior flow or event can be inferred based on the temporal order and combination relationship of said behaviors.

[0079] In addition, the spatiotemporal context or spatiotemporal information understanding function of the situation inference module (120) can interpret the temporal flow and spatial structure of a complex situation by synthesizing spatiotemporal information including spatial and temporal elements such as changes in relative positions between objects, movement paths, movement speeds, dwell times, and duration of actions.

[0080] In addition, the environmental information analysis function of the situation inference module (120) can evaluate the influence of environmental factors, such as lighting conditions, the presence of obstacles, spatial congestion, or visibility obstruction, on the behavior of objects or interactions between objects, thereby supplementing the contextual information necessary for complex situation analysis.

[0081] Finally, the inference result and supporting basis for the inference of the situation inference module (120) generates a complex situation by synthesizing the various analysis results above, and generates supporting information in the form of a natural language-based explanation indicating that the inference result for the complex situation was derived based on elements of visual features and spatiotemporal information and interactions between objects derived from visual features.

[0082] In this way, the above-mentioned situation inference module (120) infers a complex situation by taking information on objects, actions, backgrounds, and environments, which are visual features of the image, as input and comprehensively considering the interaction between objects, the temporal flow of actions, spatial arrangements, and environmental elements. At this time, the complex situation is a result of technical analysis that cannot be explained by a single object or a single action alone, and is expressed as a semantic context structured by an artificial intelligence model to be derived as situation inference information.

[0083] Next, the present invention undergoes a risk assessment step (S40).

[0084] The risk assessment step (S40) of the present invention is a step in which the risk assessment module (130) identifies risk factors based on the situation inference information and calculates a risk score and a risk grade.

[0085] Risk assessment can be performed by interpreting a predetermined instruction prompt using a large-scale language model (LLM) or a visual language model (VLM) included in the risk assessment module (130), and is not limited to a specific statistical method or a rule-based method.

[0086] The above risk assessment step may be performed in the order of identifying risk factors, calculating a risk score, and mapping risk grades according to the risk score.

[0087] First, the risk assessment module (130) identifies risk factors by analyzing object information, behavior information, environmental information, and spatiotemporal information included in the complex situation, and can identify factors related to proximity between objects, rapid change in behavior, continuation or repetition of dangerous behavior, environmental constraints, spatial constraints, or the possibility of situation spread as risk factors.

[0088] Next, the risk assessment module (130) calculates a risk score, which can be calculated using the risk factor as input. The risk score calculation can be performed by considering one or more of the following: risk to human life, time sensitivity, threat of progression, proximity, potential for spread, and environmental and social impact. The degree of reflection of each factor can be adjusted according to the characteristics of the situation or setting information.

[0089] Finally, the risk level determination by the risk assessment module (130) divides the calculated risk score into multiple intervals according to a predefined standard or threshold, thereby determining the risk level of the complex situation into multiple risk levels.

[0090] I would like to describe in detail the risk score calculation step and the risk grade determination step mentioned above.

[0091] The risk assessment module (130) of the present invention can calculate a risk score by applying a weight-based quantitative assessment method to a plurality of risk factors included in a complex situation. This can be performed by deriving an individual normalized value (F) for each risk factor derived from situation inference information, and combining the normalized values ​​for the risk factors with a predetermined weight (W) using a mathematical operation method. At this time, the weight may be set as a fixed value or may be dynamically adjusted by an application environment, time zone, or instruction prompt, and the resulting risk score is derived by a reproducible calculation procedure.

[0092] Specifically, the risk factor (F) can be determined based on one or more of the following: i) proximity between objects based on minimum distance or approach speed between objects, ii) rate of change in behavior based on frequency of behavior, duration of behavior, or rapid change in behavior, iii) environmental risk based on illumination, congestion, or presence of obstacles, and iv) time sensitivity based on the rate of change in risk level within a unit of time.

[0093] And, the above risk score can be calculated as shown in the following mathematical formula 1 by a weight-based quantitative evaluation method.

[0094] [Mathematical Formula 1]

[0095] R = 100 × (W1·F1+ W2·F2+ W3·F3+ W4·F4)

[0096] Here, R is the risk score, F1 through F4 are normalized values ​​corresponding to individual risk factors, and W1 through W4 represent weights that are dynamically adjusted according to the characteristics of the risk factors, application fields, or configuration information.

[0097] The above risk score can be calculated as a continuous value in the range of 0 to 100, and can be calculated by combining normalized values ​​and weights for each risk factor.

[0098] The above weights can be adjusted in real time by a large-scale language model (LLM) or a visual language model (VLM), and thus can be configured so that different risk scores are calculated depending on the application environment even for the same situation.

[0099] First, an example of calculating risk indicators, which are normalized values ​​for each risk factor, is shown in Table 1 below, where each risk factor (F) can be calculated as a normalized value (F) in the range of 0 to 1.

[0100] Risk factors Calculation criteria Normalized value (F) Proximity between objects F1 Minimum distance ≤ 0.5 m 0.7~1.0 Behavioral abruptness F2 Rate of change of speed ≥ 200% 0.6 ~ 1.0 Environmental Risk F3 Illuminance <50 lux 0.5 ~ 1.0 Time sensitivity F4 Duration of risk ≥ 10 seconds 0.4 ~ 1.0

[0101] The above criteria for calculating risk factors and the range of normalized values ​​are examples and may be adjusted depending on the application environment or field of application. Additionally, the risk score can be obtained by applying weights to the normalized values ​​of the calculated risk factors using the following mathematical formula 1.

[0102] [Mathematical Formula 1]

[0103] R = 100 × (W1·F1+ W2·F2+ W3·F3+ W4·F4)

[0104] Here, W1 to W4 can be individually set with weights according to the characteristics of the risk factors, and in this embodiment, each has a weight in the range of 0.1 to 0.5, and the sum of all weights is set to be 1.

[0105] For example, if W1= 0.35, W2= 0.25, W3= 0.20, W4= 0.20 and F1= 0.9, F2= 0.8, F3= 0.6, F4= 0.7, the risk score (R) can be calculated as 78 points by substituting these values ​​into the above mathematical formula 2.

[0106] Meanwhile, once the above risk score is calculated, the step of determining the risk level proceeds.

[0107] The risk score calculated in the present invention is mapped to a risk grade according to a plurality of threshold intervals that are predefined or dynamically adjusted by system configuration information, wherein the threshold values ​​determining the threshold intervals can be set based on historical data, system operating environment characteristics, or management policies. For example, the threshold values ​​can be pre-set or periodically updated based on historical risk occurrence data, statistical distributions, or operating environment characteristics.

[0108] As such, the risk score (R) calculated in the present invention can be divided into a plurality of critical intervals and mapped to a risk rating, and the risk rating intervals can be exemplified as shown in Table 2 below.

[0109] Risk score Risk level Report derivation phrases 0 ≤ R < 30 Level 1 (Low Risk) Observation needed 30 ≤ R < 55 Step 2 (Caution) Warning possible 55 ≤ R < 75 Level 3 (Danger) Immediate response 75 ≤ R ≤ 100 Level 4 (High Risk) Emergency measures

[0110] Each item value for the risk level ranges described above is an example and may be changed depending on the system's management policy or application field. Meanwhile, the risk score and risk level of the present invention may be operated such that the grade threshold for the same risk score is lowered by 10 to 20 percent during nighttime hours after sunset, or that it is automatically promoted to a higher risk level in situations of high density of multiple objects. For example, if a score of 55 is classified as 'risk' during the day, a score of 45 or higher at night may be judged as 'risk'.

[0111] In addition, in the present invention, the risk level can be updated by reflecting not only the analysis results at a single point in time but also the trend of risk change accumulated along the time axis.

[0112] Specifically, the risk assessment module (130) analyzes the change pattern of risk scores calculated for a series of multiple frames or time intervals and determines an active threat state when i) the risk score is maintained above a certain threshold, ii) the risk score increases rapidly over time, or iii) the same type of risk factor is repeatedly observed, and may go through the additional step of adjusting to a risk level higher than the existing risk level.

[0113] Meanwhile, in another embodiment of the present invention, a risk score may be determined as an active threat when it satisfies a time accumulation condition rather than a single point in time. For example, if any one or more of the following conditions are satisfied: i) a risk score ≥ 70 is maintained for 5 minutes or more continuously, ii) the risk score increases by 100% or more from 35 to 70 within 30 seconds, or iii) the same risk factor is detected repeatedly 5 times or more within 10 minutes, the risk assessment module (130) may determine the state as an active threat and raise the risk level by one or more steps.

[0114] In addition, according to another embodiment of the present invention, a region-based risk assessment including a spatial diffusion threshold can be performed, and whether risk is diffusion in an environment including multiple CCTV cameras or multiple regions can be determined by the following conditions.

[0115] For example, if conditions are satisfied such as i) when moving to an adjacent area, the same object enters an adjacent CCTV camera area within 10 seconds, ii) when the same type of risk factor is detected in three or more areas in multiple areas, or iii) when the number of areas where the risk factor occurs increases by two or more per minute in terms of the spread rate, an integrated risk score that is 10 to 30 points higher than the single area risk score may be calculated.

[0116] The above-mentioned risk assessment module (130) can reflect this spatial diffusion information to calculate a risk score higher than the risk score within a single image or determine an area-based risk rating, and accordingly, the present invention enables risk assessment of the entire facility or space unit, going beyond local risk assessment from the perspective of a single CCTV camera.

[0117] Meanwhile, if we present an example of calculating the risk score and risk grade by combining all the above embodiments in relation to the contents of Table 2, in the case where 30 people are densely packed in the CCTV input video, the minimum distance between people is 0.4m, the illuminance is 30 lux, and the duration of the dangerous behavior is 4 seconds, then F₁= 0.95, F₂= 0.7, F₃= 0.8, F₄= 0.75, and W1= 0.35, W2= 0.25, W3= 0.20, W4= 0.20, then the risk score R= 82 points and the risk grade can be derived as 'high risk stage'. In addition, the numerical-based judgment result of the above risk score and risk grade can be output together in the form of structured text including a natural language-based explanation by the text generation module (140).

[0118] Meanwhile, in the present invention, the same risk assessment structure of the risk assessment module (130) may be maintained, and the evaluation criteria may be configured to automatically adapt according to the field of application.

[0119] For example, i) in the field of industrial safety, risk factors can be adjusted so that risk scores are calculated based on human life risk and accessibility, ii) in the field of traffic control, the possibility of collision due to proximity between vehicles and the speed of vehicle movement are reflected as risk factors, and iii) in the field of public safety, risk factors can be adjusted so that the density of multiple objects and the possibility of diffusion regarding people are considered as a priority.

[0120] At this time, setting information for each application field can be provided through an instruction prompt, and the risk assessment module (130) can interpret the provided instruction prompt and automatically adjust the risk factor, weight and normalization value, and rating criteria.

[0121] Through the various embodiments described above, the present invention enhances the quantitative and reproducible nature of risk score and risk grade calculations and enables risk assessment considering spatiotemporal diffusion. Furthermore, these embodiments may be applied independently of one another, or two or more may be combined.

[0122] For reference, the above risk score, risk grade, and the basis for their calculation can also be generated in the form of natural language or structured text such as JSON, XML, CSV, etc. by the text generation module (140).

[0123] Next, the present invention undergoes a structured text generation step (S50).

[0124] The structured text generation step of the present invention is a step in which a text generation module (140) receives situation inference information, a risk score, and a risk grade, and outputs judgment result information based thereon in the form of natural language or structured text such as JSON, XML, or CSV.

[0125] Meanwhile, in this embodiment, in order to improve user confidence in situation inference information and risk assessment results, supporting grounds for inference and grounds for risk assessment, as well as visualization information thereof, may be provided together. Specifically, objects or areas determined to be risk factors are highlighted in the image, and key judgment grounds such as distance between objects, movement paths, and dwell times are output in the form of a visual overlay and provided together with a risk score or risk grade, thereby supporting an intuitive understanding of the cause of the risk judgment.

[0126] The structured text information generated by the text generation module (140) can be output in the form of a situation report containing situation inference information, a risk score, and a risk grade.

[0127] Referring to FIGS. 5 and 6, examples of situation reports are shown in which situation inference information including a natural language-based description of a complex situation, a risk score, and a risk rating are output as text by a text generation module (140).

[0128] As shown in Figures 5 and 6, in addition to the risk score and risk grade (risk item), natural language text is described in detail regarding objects included in the situation inference information (detected objects item), environmental information (detected objects item), spatiotemporal information (detected objects item), behavior (description item), interaction between objects (description item), reasoning about complex situations (reasoning item), and supporting evidence for the reasoning (supporting evidence item). Of course, one image of the corresponding input video may also be attached to the above report.

[0129] Next, the present invention undergoes the steps of transmitting the structured text to an external system and post-processing (S60).

[0130] As described above, the structured text generated by the text generation module (140) or the situation report containing the structured text is transmitted to an external analysis device / external terminal (300). The external analysis device / external terminal (300) includes various control systems, databases, alarm systems, and control systems, and the report can be used as an alarm system, a control system control signal, or a control system linkage input.

[0131] Ultimately, the above external analysis device / external terminal (300) can determine the text of situation inference information, such as a situation report, and the risk score and risk level, and if the risk level is high, it can be operated so that the alarm system and control system can be operated automatically.

[0132] Furthermore, the above post-processing step can be utilized for subsequent business processing, such as monitoring log recording, dashboard visualization, external system interface integration, and alarm system integration, by using the structured text generated in the above text generation step as input.

[0133] FIG. 3 illustrates a system configuration and data flow in which a visual feature extraction module (110), a context inference module (120), and a risk assessment module (130) are interconnected through a Model Context Protocol (MCP) based interface, and images, instruction prompts, and intermediate inference results are managed as a single analysis pipeline by an inference orchestration engine (150).

[0134] Figure 4 is a diagram illustrating the internal configuration of an inference orchestration engine, showing a structure in which prompt configuration, context and state management, model call control, MCP-based interface, input-output mapping, and error handling functions are managed integrally.

[0135] Referring to FIGS. 3 and 4, the risk score and risk grade calculation system according to an embodiment of the present invention displays a structure in which a visual feature extraction module (110), a situation inference module (120), and a risk evaluation module (130) are configured independently of each other and are integrally controlled and linked by an inference orchestration engine (150).

[0136] Referring to FIG. 3, the input video of a multi-CCTV system is transmitted to a visual feature extraction module (110) along with an instruction prompt for visual feature extraction by the inference orchestration engine (150) of the processor, and is controlled to call a visual feature extraction module based on a visual language model (VLM) (step ① of FIG. 3).

[0137] Under the control of the inference orchestration engine (150), the image language model (VLM)-based visual feature extraction module (110) processes the input image and instruction prompt to extract visual features including object information, behavior information, background information and environment information, and returns the result to the inference orchestration engine (150) (step ② of FIG. 3).

[0138] The derived visual features are controlled by the inference orchestration engine (150) to call the situation inference module (120) or the risk assessment module (130) with an instruction prompt suitable for situation inference or risk assessment in order to perform situation inference information generation and risk assessment (step ③ of FIG. 3).

[0139] According to the above control, the situation inference module (120) processes the visual features and instruction prompts to infer a complex situation, and the risk assessment module (130) identifies risk factors based on the situation inference information, which is the result of inferring the complex situation, calculates a risk score and a risk grade, and then returns the result to the inference orchestration engine (150) (step ④ of FIG. 3).

[0140] Then, the inference orchestration engine (150) calls the text generation module (140) to generate text or a situation report based on situation inference information, a risk score, and a risk grade, and the text or situation report generated by the text generation module (140) is returned to the inference orchestration engine (150), and the text or situation report is transmitted to an external analysis device / external terminal (300) under the control of the inference orchestration engine (150).

[0141] Referring to FIG. 4, the inference orchestration engine (150) of the present invention includes an MCP (Model Context Protocol) based interface layer, thereby performing the function of exchanging input data, instruction prompts, intermediate inference results, and output results in a standardized manner. In addition, the inference orchestration engine (150) performs functions of artificial intelligence model call control, prompt configuration, context and state management, input-output mapping, and error handling.

[0142] In inference processes based on large-scale language models or image language models, a structure that generates multiple intermediate inference paths, as in the present invention, can lead to error propagation due to the accumulation of intermediate results, difficulty in tracing the inference process, and reduced reproducibility of results as the inference stage becomes longer. Furthermore, if an excessive number of intermediate inference expressions are generated, there is also a possibility that hallucinations or biases may accumulate.

[0143] Accordingly, in the present invention, a Hierarchical Reasoning Model (HRM) included in the inference orchestration engine (150) can be applied, and the Hierarchical Reasoning Model (HRM) does not subordinate the entire inference process to a single process, but rather performs each step of visual feature extraction, risk factor identification, complex situation inference, and risk assessment by logically separating them.

[0144] That is, each step process according to the present invention can be interconnected through an application server based on Model Context Protocol (MCP), and by applying a hierarchical inference model (HRM) included in the inference orchestration engine (150), the processes of visual feature extraction, complex situation inference, risk factor identification, and risk assessment can be logically separated, and the intermediate results derived from each process can be mutually cross-verified to suppress the propagation of errors from higher stages to lower stages. Accordingly, the stability, reproducibility, reliability, and explainability of the entire analysis pipeline can be improved.

[0145] Meanwhile, FIG. 4 illustrates a LangChain-based workflow as an example and establishes a workflow management framework control structure capable of configuring an analysis pipeline consisting of a visual feature extraction step, a situation inference information generation step, and a risk assessment step by automating artificial intelligence model call sequence control, input-output mapping, state management, error handling, and prompt configuration.

[0146] As such, the inference orchestration engine (150) of the present invention can be configured to integrally control the execution conditions of each step, artificial intelligence model calling rules, control logic, intermediate result delivery method and prompt configuration, and thereby can consistently manage all processing flows from visual feature extraction and complex situation inference to the calculation of risk scores and risk grades.

[0147] Although the present invention has been described above in relation to specific embodiments, this is merely illustrative and the present invention is not limited thereto. A person skilled in the art to which the present invention pertains may change or modify the described embodiments without departing from the scope of the present invention, and various modifications and variations are possible within the scope of the technical spirit of the present invention and the equivalent scope of the claims set forth below. Explanation of the symbols

[0149] 100: Processor 110: Visual Feature Extraction Module 120: Context Inference Module 130: Risk Assessment Module 140: Text generation module 150: Inference Orchestration Engine 200: Multi-CCTV System 300: External analysis device / external terminal

Claims

Claim 1 A video input step in which video from a multi-CCTV system is input into a visual feature extraction module; a step in which the visual feature extraction module analyzes the input video and extracts visual features including object, behavior, and environmental information within the input video; a step in which a situation inference module derives inter-object interaction and spatiotemporal information based on the visual features, semantically and contextually infers a complex situation formed by combining objects, behavior, inter-object interaction, environmental information, and spatiotemporal information, and generates situation-aware inference information including a natural language-based description of the complex situation; a step in which a risk assessment module identifies risk factors based on the semantic and contextual content included in the situation inference information and calculates a risk score and a risk grade based on the risk factors. The method comprises the step of a text generation module generating judgment result information including the situation inference information, risk score, and risk grade in a structured text format; wherein the situation inference information generation step generates situation inference information for a complex situation reflecting the interaction between objects, the meaning of the action, and the spatiotemporal context, by taking as input a Chain-of-Thought prompt that instructs to sequentially perform steps of object identification, distance analysis between objects, and behavioral intent inference, visual features extracted from an input image; and the risk score and risk grade calculation step calculates a normalized value for each of a plurality of risk factors included in the complex situation, and calculates a risk score by combining the normalized values ​​for each risk factor with a predetermined weight using a mathematical operation method; and determines a risk grade by mapping the risk score to one of a plurality of threshold intervals that are predefined or determined according to setting information.A method for calculating a risk score and risk grade based on a complex situation within video data, comprising: a step of adjusting the risk grade when the same type of risk factor occurs repeatedly, when the security grade of the area where the video is captured is relatively high, or when the difference in illumination changes rapidly; wherein the plurality of risk factors consist of proximity between objects within the complex situation, duration of behavior, rate of change of behavior, environmental risk, and time sensitivity, wherein the risk score is calculated as a continuous value in the range of 0 to 100, the weights are set so that their sum is 1, and the risk score is calculated by a weighted sum operation. Claim 2 A method for calculating a risk score and risk grade based on complex situations within image data, wherein, in claim 1, the visual feature extraction step, the situation inference information generation step, or the risk score and risk grade calculation step are performed using one or more models selected from a large-scale language model (LLM), a visual language model (VLM), or a multimodal inference model (MRM), which take an input image or visual features and an instruction prompt as input. Claim 3 A method for calculating a risk score and risk grade based on complex situations within image data, wherein the visual feature extraction step is characterized in that an image language model performs object recognition, behavior recognition, and surrounding environment recognition through a single inference process using features related to objects, behaviors, and environment information based on an input image and a command prompt. Claim 4 A method for calculating a risk score and risk grade based on complex situations within image data, wherein the visual feature extraction step is characterized by using a zero-shot or few-shot-based instruction prompt to extract information on undefined objects, behaviors, or environments and adaptively interpreting new undefined situations. Claim 5 delete Claim 6 delete Claim 7 delete Claim 8 delete Claim 9 delete Claim 10 A method for calculating a risk score and risk grade based on complex situations within image data, wherein, in claim 1, the structured text generation step generates judgment result information including inference about the complex situation, supporting grounds for the inference about the complex situation, a risk score, and a risk grade as structured text, and outputs it in a structured text format of JSON, XML, CSV, or YAML. Claim 11 A processor configured to analyze objects, behaviors, interactions between objects, environmental information, and spatiotemporal information using an artificial intelligence model from video collected from multiple CCTV systems, wherein the processor comprises: a visual feature extraction module that extracts visual features including objects, behaviors, and environmental information from video input from multiple CCTV systems; a situation inference module that derives interactions between objects and spatiotemporal information using the visual features as input, semantically and contextually infers a complex situation formed by the combination of objects, behaviors, interactions between objects, environmental information, and spatiotemporal information, and derives situation inference information including a natural language-based explanation of the complex situation; a risk assessment module that receives the situation inference information, identifies risk factors based on semantic and contextual content included in the situation inference information, and calculates a risk score and a risk grade based on the risk factors; and a text generation module that generates judgment result information derived based on the situation inference information, risk score, and risk grade in a structured text format.The above situation inference module takes as input visual features extracted from an input image and a Chain-of-Thought prompt that instructs to sequentially perform steps of object identification, distance analysis between objects, and behavioral intent inference, and generates situation inference information for a complex situation reflecting the interaction between objects, the meaning of the behavior, and the spatiotemporal context; the above risk assessment module calculates a normalized value for each of the multiple risk factors included in the complex situation, calculates a risk score by combining the normalized values ​​for each risk factor with a predetermined weight using a mathematical operation method, determines a risk level by mapping the risk score to one of a plurality of threshold intervals that are predefined or determined according to setting information, adjusts the risk level in cases where the same type of risk factor occurs repeatedly, where the security level of the area where the image was captured is relatively high, or where the difference in illumination changes rapidly, wherein the multiple risk factors consist of proximity between objects within the complex situation, duration of behavior, rate of change of behavior, environmental risk, and time sensitivity, the risk score is calculated as a continuous value in the range of 0 to 100, and the sum of the weights A system for calculating risk scores and risk grades based on complex situations within image data, characterized in that the risk score is set to 1 and is calculated by a weighted summation operation. Claim 12 A system for calculating risk scores and risk grades based on complex situations within image data, characterized in that, in claim 11, the visual feature extraction module, the situation inference module, and the risk assessment module include one or more models selected from a large-scale language model (LLM), a visual language model (VLM), or a multimodal inference model (MRM) that process input images or visual features and instruction prompts as input. Claim 13 A system for calculating risk scores and risk grades based on complex situations within image data, characterized in that, in claim 11, the processor further includes an inference orchestration engine that performs call order control, input-output mapping, state management, error handling, and prompt configuration for a large-scale language model (LLM), a visual language model (VLM), and a multimodal inference model (MRM), and constructs a single analysis pipeline using the inference orchestration engine, and cross-validates step-by-step inference results by applying a hierarchical inference model.

Citation Information

Patent Citations

  • Collision risk warning system

    KR1020260006341A

  • Generative AI-powered construction site safety management Method and System for determining hazard detection and guiding regulatory actions through image analysis

    KR102875205B1

  • Artificial Intelligence-based hierarchical risk situation assessment and natural language voice guidance method, and system using same

    KR102879213B1