A method for constructing an inference pipeline for multi-modal analysis, and an apparatus and a system for performing the same

Abstract

A method for constructing an inference pipeline for multimodal analysis by industrial domain according to one embodiment of the present application comprises: a step of obtaining an image processing pipeline package and an inference engine for multimodal analysis—the inference engine includes an industrial domain agnostic general-purpose inference model and industrial domain adaptive data processing logic—; a step of obtaining, through a client terminal, a setting command for a target rule set related to a specific industrial domain among a plurality of rule sets compatible with the inference engine—each rule set defines inference requirements for a structured vision task for inference of an unstructured vision task using the general-purpose inference model for the industrial domain—; a step of obtaining at least one image processing pipeline that satisfies the inference requirements of the target rule set from the image processing pipeline package based on the setting command for the target rule set; and a step of constructing the inference pipeline based on the at least one image processing pipeline and the inference engine.

Description

Technology Field

[0001] This application relates to the field of artificial intelligence analysis technology, and more specifically, to image signal processing, computer vision systems, and real-time situation awareness systems using multi-sensor analysis and multimodal inference. Specifically, the present invention relates to an integrated management platform that deploys AI software applicable to various industrial domains into a computing environment and automates and simplifies its configuration and operation. Furthermore, the present invention relates to a real-time image processing and analysis system that extracts metadata and processed industrial data from input images and detects hazardous situations based thereon. Background Technology

[0003] With the recent advancement of Artificial Intelligence (AI) technology, the adoption of AI-based analysis applications is increasing in fields such as transportation, industrial safety, public control, and security. To ensure the detection performance and processing speed of AI systems, configurations optimized for the analysis targets and field environments are essential. Consequently, there is a growing need for edge computing technology, which directly processes and analyzes information on field servers where data is generated.

[0004] Real-time multi-source video analysis systems must support various input methods, such as RTSP and video files, and operate in diverse hardware environments ranging from high-end servers to low-end edge devices. In particular, for low-end hardware, optimized design is required to ensure efficient performance with limited computational resources.

[0005] However, conventional AI system deployment and operation technologies had the following limitations.

[0006] First, there was fragmentation and complexity in the deployment process. To run the AI ​​system, processes such as adding servers, uploading software packages and AI models, registering data input sources, and creating service stack components had to be performed sequentially. Since these tasks were carried out in a fragmented manner through individual tools or command-line interfaces (CLI), it was difficult to intuitively grasp the overall deployment process, and the work was excessively time-consuming.

[0007] Second, a high level of technical expertise was required. Each deployment phase demanded specialized knowledge in server management, container technology, network configuration, and AI model management. Consequently, only skilled developers or system engineers could perform the tasks, making it difficult for field engineers or customers to configure the system directly. This increased reliance on human resources and led to higher maintenance costs.

[0008] Third, the pipelines were distributed across hardware. Conventional technologies utilized hardware-specific SDKs (e.g., NVIDIA Deepstream, Qualcomm IM SDK) or general-purpose image processing technologies (e.g., gstreamer) to optimize the computational performance of each piece of hardware. In this process, the pipelines were distributed across hardware, making design and maintenance difficult, and limiting scalability and interoperability in distributed edge or server systems.

[0009] Fourth, it relied excessively on predefined rules. Conventional technology detected specific situations based on predefined rules. Specifically, according to conventional technology, it was necessary to pre-train an AI model on the detection target or define rules such as parameter adjustment and the setting of regions of interest. Furthermore, when combining the inference results of multiple AI models or combining sensors and AI models, the combination method had to be defined in advance. Consequently, it lacked the ability to handle open-ended situations that were not pre-defined.

[0010] Fifth, the interoperability between rule sets and hardware environments was unclear. Conventional technology lacked a mechanism to effectively connect analysis requirements for specific industrial domains with pipelines optimized for each hardware. As a result, pipelines had to be reconfigured when deploying the same rules to different hardware environments, and the reusability and portability of rule sets were limited.

[0011] Sixth, analysis was performed without considering the context of the situation. Conventional technology failed to account for the context of specific situations occurring in the video, frequently resulting in the detection of multiple duplicate specific situations for a single actual event. This degraded the accuracy of the system and generated unnecessary alerts.

[0012] Seventh, it failed to distinguish the progression stages of an event. Conventional technology either detected only the occurrence of an event or processed it using a single rule without distinguishing between the progression stages (occurrence, action, termination), thus failing to accurately reflect actual situational changes. Furthermore, it lacked a mechanism to prevent misjudgment caused by temporary situational changes.

[0013] Therefore, there is a need for the development and research of new technologies that can integrate fragmented AI system deployment processes, enable efficient operation in diverse hardware environments, systematically link hardware and analysis requirements based on rule sets, recognize even undefined situations, provide integrated information by understanding the context of the situation, and systematically track the progression stages of events. The problem to be solved

[0015] The present invention aims to solve the problems of the aforementioned prior art and to solve the following problems.

[0016] The objective of the present invention is to provide a GUI-based integrated management platform that enables even non-experts to deploy systems quickly and accurately by integrating the complex and fragmented AI system deployment process into a single intuitive workflow and abstracting technical complexity.

[0017] The objective of the present invention is to provide an image processing pipeline package capable of integrated operation across various hardware, ranging from high-spec servers to low-spec edge devices, thereby resolving the issue of pipeline distribution across hardware and facilitating design and maintenance.

[0018] The objective of the present invention is to define the inference requirements of an image processing pipeline through industry domain-specific rule sets and to automatically map the image processing pipeline to a general-purpose inference model based on rule set identifiers, thereby systematically linking the hardware environment with analysis requirements and improving the reusability and portability of the rule sets.

[0019] The problem that the present invention aims to solve is to provide an inference engine that processes metadata and industrial data extracted from an image processing pipeline through a Vision-Language Model (VLM) to understand the context of the data and recognize irregular situations, thereby enabling the detection of even undefined open-ended situations.

[0020] The problem that the present invention aims to solve is to improve context understanding performance by storing detection results based on predefined rules and detection results of irregular situations in short-term memory (STM) to preserve and utilize the context between each event, thereby providing multiple situations that have dependent relationships but are understood independently as a single integrated situation.

[0021] The problem that the present invention aims to solve is to provide an event management system capable of performing analysis optimized for each stage by adaptively applying different rules and queries according to the event's progression stage (occurrence, action, termination) and systematically tracking the entire lifecycle of an event.

[0022] The objective of the present invention is to provide a method that can improve the reliability and accuracy of event analysis by verifying whether an event recurs after a delay period to prevent misjudgment caused by temporary changes in circumstances when determining the end of an event, and by storing and utilizing the context of past events in short-term memory.

[0023] The problems that the present invention aims to solve are not limited to those described above, and problems not mentioned will be clearly understood by those skilled in the art from this specification and the attached drawings. means of solving the problem

[0025] A method for constructing an inference pipeline for multimodal analysis by industrial domain according to one embodiment of the present application may further include: a step of obtaining an image processing pipeline package and an inference engine for multimodal analysis—the inference engine includes an industrial domain agnostic general inference model and industrial domain adaptive data processing logic—; a step of obtaining, through a client terminal, a setting command for a target rule set related to a specific industrial domain among a plurality of rule sets compatible with the inference engine—each rule set defines inference requirements for a structured vision task for inference of an unstructured vision task using the general inference model for the industrial domain—; a step of obtaining at least one image processing pipeline that satisfies the inference requirements of the target rule set from the image processing pipeline package based on the setting command for the target rule set; and a step of constructing the inference pipeline based on the at least one image processing pipeline and the inference engine.

[0026] The means for solving the problem of the present invention are not limited to the means for solving the problem described above, and unmentioned means for solving the problem will be clearly understood by those skilled in the art from this specification and the attached drawings. Effects of the invention

[0028] According to a method for constructing an inference pipeline, a method for analyzing events using an inference pipeline, a method for providing event information, an apparatus for performing the same, and / or a system according to one embodiment of the present application, by systematically managing the entire process of distributing an AI analysis application through a GUI-based integrated platform, complex and fragmented distribution processes can be integrated into a single intuitive workflow. This enables even non-experts to distribute the system quickly and accurately, resulting in reduced distribution time, lower dependence on human resources, and reduced maintenance costs.

[0029] According to a method for constructing an inference pipeline, a method for analyzing events using an inference pipeline, a method for providing event information, and an apparatus and / or system for performing the same, in accordance with one embodiment of the present application, by providing an image processing pipeline package that can be operated integrally on various hardware, it is possible to solve the problem of pipeline design and maintenance distributed by hardware and improve scalability and interoperability in distributed edge or server systems.

[0030] According to a method for constructing an inference pipeline, a method for analyzing events using an inference pipeline, a method for providing event information, an apparatus for performing the same, and / or a system according to one embodiment of the present application, inference requirements for a structured vision task are defined through a rule set, and by automatically mapping an image processing pipeline and a general-purpose inference model based on a rule set identifier, the same rule set can be easily deployed to different hardware environments. This improves the reusability and portability of the rule set and has the effect of enabling rapid system construction without the need to redefine the rule set even when hardware changes.

[0031] According to a method for constructing an inference pipeline, a method for analyzing events using an inference pipeline, a method for providing event information, an apparatus for performing the same, and / or a system according to one embodiment of the present application, by providing an architecture that is stably interconnected in a distributed environment, it is possible to configure an efficient system in which an image processing pipeline is run on low-spec hardware and an inference engine is run on high-spec hardware. This has the effect of optimally utilizing hardware resources and ensuring system scalability.

[0032] According to a method for constructing an inference pipeline, a method for analyzing events using an inference pipeline, a method for providing event information, an apparatus for performing the same, and / or a system according to one embodiment of the present application, by selecting and distributing a software package and an AI model optimized for the hardware specifications of a server to be analyzed, it is possible to secure maximum performance in each hardware environment and increase the stability and efficiency of the system.

[0033] According to a method for constructing an inference pipeline, a method for analyzing events using an inference pipeline, a method for providing event information, an apparatus for performing the same, and / or a system according to one embodiment of the present application, by allowing a user to intuitively set rule sets and parameters through a GUI, the operation of a complex AI system can be simplified and the system can be easily managed by field engineers or non-experts.

[0034] According to a method for constructing an inference pipeline, a method for analyzing events using an inference pipeline, a method for providing event information, an apparatus for performing the same, and / or a system according to one embodiment of the present application, by providing an inference engine utilizing a Vision-Language Model (VLM), it is possible to understand the context of data and recognize unstructured situations, thereby enabling the detection of even open-ended situations that are not predefined. This resolves the problem of excessive reliance on predefined rules and has the effect of enabling more flexible and intelligent situation recognition.

[0035] According to a method for constructing an inference pipeline, a method for analyzing events using an inference pipeline, a method for providing event information, an apparatus for performing the same, and / or a system according to one embodiment of the present application, by utilizing short-term memory to preserve and utilize the context between each event, multiple situations that have dependent relationships but are understood independently can be provided as a single integrated situation. Through this, redundant notifications are reduced and the accuracy of situation awareness is improved, thereby enhancing the user experience.

[0036] According to a method for constructing an inference pipeline, a method for analyzing events using an inference pipeline, a method for providing event information, an apparatus for performing the same, and / or a system according to one embodiment of the present application, by adaptively switching analysis rules according to the progression stage of an event and generating a query optimized for each stage, it is possible to systematically track an event from its occurrence to its conclusion and provide clear status information to the user for each stage. This has the effect of preventing unnecessary duplicate detection and enabling a rapid and appropriate response.

[0037] According to a method for constructing an inference pipeline, a method for analyzing events using an inference pipeline, a method for providing event information, an apparatus for performing the same, and / or a system according to one embodiment of the present application, by monitoring for recurrence for a predetermined delay period after determining the end of an event and performing an analysis that considers the context of past events using short-term memory, it is possible to prevent temporary misjudgments and improve the reliability of event analysis. In addition, by understanding multiple events with dependent relationships as a single integrated situation, it is possible to prevent duplicate notifications and improve the user experience.

[0038] The effects of the present invention are not limited to the effects described above, and unmentioned effects will be clearly understood by those skilled in the art from this specification and the accompanying drawings. Brief explanation of the drawing

[0040] FIG. 1 is a schematic diagram showing an event analysis system according to one embodiment of the present application. FIG. 2 is a diagram showing the detailed configuration of an analysis server and the structure of an inference pipeline according to an embodiment of the present invention. FIG. 3 is a diagram illustrating an event analysis aspect performed by an event analysis system according to one embodiment of the present application. FIG. 4 is a flowchart illustrating a method for constructing an inference pipeline (analysis pipeline) according to one embodiment of the present application. FIG. 5 is a drawing illustrating an exemplary graphical user interface of a client terminal to explain an aspect of receiving a setting command for a target rule set from a client terminal according to an embodiment of the present application. FIG. 6 is a drawing illustrating an exemplary graphical user interface of a client terminal to explain an aspect of receiving a setting command for a target rule set from a client terminal according to an embodiment of the present application. FIG. 7 is a detailed flowchart of the step (S1400) of constructing an inference pipeline according to one embodiment of the present application. FIG. 8 is a drawing illustrating an exemplary graphical user interface of a client terminal to explain an aspect of receiving a setting command for allocating resources to an image processing pipeline from a client terminal according to one embodiment of the present application. FIG. 9 is a drawing illustrating an exemplary graphical user interface of a client terminal to explain an aspect of receiving a setting command for selecting a channel to be linked with an artificial intelligence model to be executed on an image processing pipeline from a client terminal according to one embodiment of the present application. FIG. 10 is a drawing for illustrating one aspect of constructing an inference pipeline according to one embodiment of the present application. FIG. 11 is a diagram illustrating an aspect of constructing an inference pipeline according to an embodiment of the present application and an aspect of performing inference using the inference pipeline. FIG. 12 is a diagram illustrating an aspect of analyzing events using a constructed inference pipeline according to one embodiment of the present application. FIG. 13 is a flowchart illustrating a method for constructing an inference pipeline according to another embodiment of the present application. FIG. 14 is a flowchart illustrating a method for analyzing events using a constructed inference pipeline according to one embodiment of the present application. FIG. 15 is a diagram illustrating an exemplary aspect of detecting an event using a constructed inference pipeline according to one embodiment of the present application. FIG. 16 is a diagram illustrating an exemplary event detection aspect using a constructed inference pipeline according to one embodiment of the present application. FIG. 17 is a diagram illustrating an exemplary event detection aspect using a constructed inference pipeline according to one embodiment of the present application. FIG. 18 is a diagram illustrating an exemplary aspect of detecting an event using a constructed inference pipeline according to one embodiment of the present application. FIG. 19 is a flowchart illustrating a method for providing information about an event according to one embodiment of the present application. FIG. 20 is a specific flowchart of a method for providing information about an event according to one embodiment of the present application. FIG. 21 is a drawing for explaining one aspect of event analysis according to the progress stage of an event according to one embodiment of the present application. FIG. 22 is a drawing for explaining one aspect of event analysis according to the progress stage of an event according to one embodiment of the present application. FIG. 23 is a drawing for explaining one aspect of event analysis according to the progress stage of an event according to one embodiment of the present application. FIG. 24 is a drawing for explaining one aspect of event analysis according to the progress stage of an event according to one embodiment of the present application. FIG. 25 is a drawing for illustrating one aspect of event information provided according to the progress stage of an event according to one embodiment of the present application. Specific details for implementing the invention

[0041] The aforementioned objectives, features, and advantages of the present application will become more apparent from the following detailed description in conjunction with the accompanying drawings. However, as the present application is subject to various modifications and may have various embodiments, specific embodiments are illustrated in the drawings and described in detail below.

[0042] Throughout the specification, identical reference numbers generally represent identical components. Additionally, components with identical functions within the same scope of concept appearing in the drawings of each embodiment are described using the same reference numeral, and redundant descriptions thereof are omitted.

[0043] If it is determined that a detailed description of known functions or configurations related to this application could unnecessarily obscure the essence of this application, such detailed description is omitted. Furthermore, numbers used in the description of this specification (e.g., First, Second, etc.) are merely identifiers to distinguish one component from another.

[0044] Furthermore, the suffixes "module" and "part" for components used in the following embodiments are assigned or used interchangeably solely for the ease of drafting the specification, and do not inherently possess distinct meanings or roles.

[0045] In the following examples, singular expressions include plural expressions unless the context clearly indicates otherwise.

[0046] In the following embodiments, terms such as "include" or "have" mean that the features or components described in the specification are present, and do not preclude the possibility that one or more other features or components may be added.

[0047] In the drawings, the size of components may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are arbitrarily depicted for convenience of explanation, and the present invention is not necessarily limited to what is illustrated.

[0048] Where an embodiment can be implemented differently, the order of a particular process may be performed differently from the order described. For example, two processes described consecutively may be performed substantially simultaneously or proceed in the reverse order of the description.

[0049] In the following embodiments, when components are described as being connected, it includes not only cases where the components are directly connected but also cases where components are interposed between other components and are indirectly connected. For example, when components are described as being electrically connected in this specification, it includes not only cases where the components are directly electrically connected but also cases where components are interposed between other components and are indirectly electrically connected.

[0050] A method for constructing an inference pipeline for multimodal analysis by industrial domain according to one embodiment of the present application may further include: a step of obtaining an image processing pipeline package and an inference engine for multimodal analysis—the inference engine includes an industrial domain agnostic general inference model and industrial domain adaptive data processing logic—; a step of obtaining, through a client terminal, a setting command for a target rule set related to a specific industrial domain among a plurality of rule sets compatible with the inference engine—each rule set defines inference requirements for a structured vision task for inference of an unstructured vision task using the general inference model for the industrial domain—; a step of obtaining at least one image processing pipeline that satisfies the inference requirements of the target rule set from the image processing pipeline package based on the setting command for the target rule set; and a step of constructing the inference pipeline based on the at least one image processing pipeline and the inference engine.

[0051] According to one embodiment of the present application, the rule set is generated for each industrial domain, and the inference requirements of each rule set include types of structured vision inference tasks corresponding to any one of a plurality of predefined unstructured vision inference tasks within any one industrial domain, and the types of structured vision inference tasks may include at least one of object detection, segmentation, classification, object tracking, and pose estimation.

[0052] According to one embodiment of the present application, each set of rules may include rule items for generating input data for the general-purpose inference model from the output of an image processing pipeline included in the image processing pipeline package for each industry domain, and parameters that can be changed by a user.

[0053] According to one embodiment of the present application, the step of obtaining a setting command for the target rule set may further include: providing the compatible rule sets through the client terminal; receiving a first input through the client terminal for selecting the target rule set among the compatible rules; receiving a second input through the client terminal for selecting a specific channel to which the target rule set is to be applied; and receiving a third input through the client terminal for a setting value of the changeable parameter.

[0054] According to one embodiment of the present application, the step of obtaining a setting command for the target rule set may further include the step of obtaining a target rule for generating input data of the inference engine generated by applying a setting value for the changeable parameter to a rule item belonging to the target rule set for the specific channel based on the first to third inputs.

[0055] According to one embodiment of the present application, the parameters changeable by the user include site adaptive parameters, wherein the site adaptive parameters may include a first parameter related to at least one of a crop, padding, confidence threshold, region of interest, framedrop rate, frame interval and object type to be detected for analysis through the image processing pipeline, and a second parameter related to a query to be input to the general inference model.

[0056] According to one embodiment of the present application, the target rule may include at least one of a rule set identifier corresponding to the target rule set, a rule identifier corresponding to the target rule, first information regarding a parameter for configuring the output form of the image processing pipeline or a task of the image processing pipeline, second information regarding a query template associated with the rule identifier, third information for generating a query as input data to be input to the general inference model based on the query template and the output of the image processing pipeline, and fourth information regarding logic associated with the rule identifier for processing the output of the general inference model.

[0057] According to one embodiment of the present application, the step of obtaining a setting command for the target rule set may further include: obtaining first rule set information and first rule information that are pre-assigned to the specific channel; receiving a setting command for a candidate rule set and a candidate rule based on the first to third inputs; and restricting the candidate rule from being received as the target rule suitable for analysis regarding the specific channel when the first rule set information and the candidate rule set are the same set, and receiving the candidate rule as the target rule suitable for analysis regarding the specific channel when the first rule set information and the candidate rule set are different sets.

[0058] According to one embodiment of the present application, the step of obtaining a setting command for the target rule set may further include: obtaining second rule set information and second rule information that are pre-assigned to a specific channel and a different channel; receiving a setting command for a candidate rule set and a candidate rule based on the first to third inputs; and receiving the candidate rule that is different from the second rule information as the target rule suitable for analysis regarding the specific channel, even if the second rule set information and the candidate rule set are the same rule set.

[0059] According to one embodiment of the present application, the step of constructing the inference pipeline may further include the step of constructing the inference pipeline by configuring the image processing pipeline to perform an analysis corresponding to an inference requirement corresponding to the target rule for a structured vision task for vision task inference using the general-purpose inference model, based on the at least one image processing pipeline and the target rule, by applying the target rule to the specific channel.

[0060] According to one embodiment of the present application, the step of constructing the inference pipeline may further include: generating structured data that defines input data of the inference engine in response to a setting value for the changeable parameter being input through the third input; and constructing the inference pipeline by configuring the image processing pipeline to generate input data of the inference engine for the specific channel using the structured data.

[0061] According to one embodiment of the present application, the method for constructing the inference pipeline further comprises the step of storing rule management information including at least one of an identifier of the image processing pipeline included in the inference pipeline through the industry domain adaptive data processing logic (RM), a parameter for executing the image processing pipeline, an identifier for a region of interest set to be detected through the image processing pipeline, a rule set identifier corresponding to the target rule set applied per region of interest, and a rule identifier corresponding to a target rule belonging to the target rule set, wherein the rule management information may be characterized by being stored on a channel basis.

[0062] According to one embodiment of the present application, the industrial domain adaptive data processing logic of the inference pipeline may be configured to identify the rule set identifier included in the output of the image processing pipeline by referring to the rule set identifier of the rule management information, map the general inference model of the inference pipeline to perform the unstructured vision task inference based on the identified rule set identifier, and input the input data generated by applying the target rule into the mapped general inference model.

[0063] According to one embodiment of the present application, the industry domain adaptive data processing logic of the inference pipeline may be configured to generate the input data by referencing the rule identifier of the rule management information to obtain a query template associated with the rule identifier, and by performing calculations on variables included in the query template using the output of the image processing pipeline.

[0064] According to one embodiment of the present application, the step of constructing the inference pipeline may further include: a step of allocating resources for the image processing pipeline; a step of setting an artificial intelligence model to be executed on the image processing pipeline; a step of setting a channel to receive images to be analyzed on the inference pipeline; and a step of constructing the inference pipeline by creating a container of the inference pipeline based on the allocated resources, the set artificial intelligence model, and the set channel.

[0065] According to one embodiment of the present application, the artificial intelligence model may be selected from a model pool comprising a plurality of models optimized for different hardware, as at least one artificial intelligence model whose performance has been verified with respect to the hardware specification information of the server to be analyzed, and the image processing pipeline package may be selected from a pipeline pool comprising a plurality of image processing pipelines optimized for different hardware, as at least one image processing pipeline whose performance has been verified with respect to the hardware specification information of the server to be analyzed.

[0066] According to one embodiment of the present application, the general-purpose inference model of the inference engine may be characterized by being selected from a model pool comprising a plurality of inference models optimized for different hardware, as at least one general-purpose inference model whose performance has been verified by considering compatibility with the image processing pipeline and hardware specification information of the server on which analysis is to be performed.

[0067] According to one embodiment of the present application, the step of obtaining a setting command for the target rule set further includes the step of receiving a rule setting command for at least one of the first target rule set and the second target rule set, wherein when the setting command for the first target rule set is received, the inference pipeline is constructed such that a first pipeline group composed of at least one of the first image processing pipeline and the second image processing pipeline is constructed as the at least one image processing pipeline, and when the setting command for the second target rule set is received, the inference pipeline is constructed such that a second pipeline group composed of at least one of the second image processing pipeline and the third image processing pipeline is constructed as the at least one image processing pipeline.

[0068] According to one embodiment of the present application, the inference pipeline may be constructed such that a computer vision-based model that performs structured vision task inference is executed on at least one image processing pipeline, and a vision-language model that performs unstructured vision task inference is executed as the general-purpose inference model.

[0069] According to one embodiment of the present application, a computer-readable recording medium may be provided that records a program for executing a method for constructing an inference pipeline.

[0070] A method for analyzing an event according to one embodiment of the present application may include: receiving a video stream; receiving constraint information for analysis; performing a first processing on the video stream using at least one first model (CV model)—the first model is configured to perform the first processing by referring to the constraint information and output a first data set—; performing a second processing on the first data set by referring to the constraint information to generate a query to be input to the second model; querying the second model to generate a text-type output based on the generated query; performing a third processing on the output generated through the second model by referring to the constraint information; and generating event information based on the result of the first processing and the result of the third processing.

[0071] According to one embodiment of the present application, the constraint information may include at least one of: a constraint identifier; first information regarding parameters of the first model for constituting the first data set; second information regarding a query template associated with the constraint identifier; third information configured to generate the query based on the query template and the first data set; and fourth information regarding a parser associated with the constraint identifier.

[0072] According to one embodiment of the present application, the step of generating the query may further include: receiving a target query template to be used for analysis by referring to the second information of the constraint information; and generating the query by calculating a value for a variable included in the target query template based on the first data set by referring to the third information of the constraint information.

[0073] According to one embodiment of the present application, the step of processing the output generated through the second model for the third time may further include: receiving a target parser to be used for analysis by referring to the fourth information of the constraint information; and parsing the output generated through the second model using the target parser.

[0074] According to one embodiment of the present application, the event information may include at least one of: a video used for analysis; an embedding or metric related to the result of the first processing; a type of event detected by the third processing; and constraint information.

[0075] According to one embodiment of the present application, the first data set may consist of a constraint identifier, object-based metadata, analysis data through the first model, and a video used for analysis.

[0076] According to one embodiment of the present application, the second model may be characterized as being a multimodal model matched as a model to input the query by referring to the constraint identifier included in the first data set.

[0077] According to one embodiment of the present application, the step of performing the first processing further includes the step of determining a first candidate image for an image corresponding to the start time of an activity of interest and a second candidate image for an image corresponding to the end time of an activity of interest, wherein the step of generating the query may further include the step of generating the query based on the first candidate image and the second candidate image.

[0078] According to one embodiment of the present application, the step of generating the query based on the first candidate image and the second candidate image may further include: generating a first query to query the start of the activity of interest based on the first candidate image; and generating a second query to query the end of the activity of interest based on the second candidate image.

[0079] According to one embodiment of the present application, the second candidate image may be determined as an image at a point in time after a predetermined time has elapsed from a point in time corresponding to the first candidate image, or may be determined based on a change of the second candidate image with respect to the first candidate image.

[0080] According to one embodiment of the present application, the step of querying the second model may further include the step of inputting the first candidate image and the first query as the first input value of the second model; and the step of inputting the second candidate image and the second query as the second input value of the second model.

[0081] According to one embodiment of the present application, the second model may be configured to output that an event of interest has occurred when the first response to the first input value and the second response to the second input value correspond to a predetermined response combination.

[0082] According to one embodiment of the present application, the step of performing the first processing further includes the step of obtaining field of view information of a camera that captured the video stream, wherein the step of generating the query may further include the step of receiving a plurality of candidate queries—each of which is a query pre-matched according to the camera field of view—; and the step of determining, based on the field of view information, a query matched with the field of view information among the plurality of candidate queries as a target query.

[0083] According to one embodiment of the present application, the step of performing the first processing further comprises: a step of obtaining a reference image for a normal situation; a step of comparing the reference image and the current image of the video stream through the first model; and a step of determining, based on the comparison result, that the current image is a candidate image for an abnormal situation that is not the normal situation, wherein the step of generating the query may further include a step of generating the query based on the candidate image.

[0084] According to one embodiment of the present application, the step of generating the query may further include: a step of determining a region of interest in which a difference exists by comparing pixels included in the reference image with pixels included in a candidate image for an abnormal situation; and a step of generating the query based on a cropped image including the region of interest.

[0085] According to one embodiment of the present application, the step of performing the first processing comprises: identifying a situation in which at least one object is located in a region of interest based on the video stream; and generating a cropped image including the region of interest and the at least one object associated with the identified situation, and the step of generating the query may further include generating the query based on the cropped image.

[0086] According to one embodiment of the present application, the step of performing the first processing includes the step of obtaining direction information of at least one object based on the video stream; wherein the step of generating the query may further include the step of generating the query based on a third query corresponding to the first direction information when the direction information of the object includes the first direction information, and generating the query based on a fourth query corresponding to the second direction information when the direction information of the object includes second direction information different from the first direction information.

[0087] According to one embodiment of the present application, the step of performing the first processing includes: identifying a situation in which at least one object moves and then stops based on the video stream; and generating a cropped image including the at least one object associated with the identified situation, and the step of generating the query may further include generating a target query corresponding to the stopped situation based on the cropped image.

[0088] According to one embodiment of the present application, the step of performing the first processing includes: identifying a situation in which at least two objects exist based on the video stream; and generating a cropped image including the at least two objects, wherein the step of generating the query may further include generating the query based on at least one of an identifier for each of the at least two objects, object information, the number of objects, and the cropped image.

[0089] According to one embodiment of the present application, the step of performing the first processing further includes the step of calculating the confidence level of the area of ​​interest of an object present in the video stream using a pose estimation technique, wherein the step of generating the query may further include the step of receiving a cropped image of the area of ​​interest and generating a query related to the wearing of protective gear on the area of ​​interest when the calculated confidence level is greater than a predetermined threshold.

[0090] A method for providing information about an event using a vision-language model according to one embodiment of the present application may include: a step of obtaining first image-based information based on a first rule and generating a first query regarding whether an event has occurred based on the first image-based information; a step of transmitting the first query to the vision-language model so that the vision-language model determines whether the event has occurred; a step of providing information that the event has occurred after determining that the event has occurred based on the output of the vision-language model; a step of obtaining second image-based information based on a second rule and generating a second query regarding whether the event has ended based on the second image-based information after determining that the event has occurred; a step of transmitting the second query to the vision-language model so that the vision-language model determines whether the event has ended; and a step of providing information that the event has ended after determining that the event has ended based on the output of the vision-language model.

[0091] According to one embodiment of the present application, the step of generating a second query regarding whether the event has ended may further include: generating a third query to determine whether an action is being performed in accordance with the occurrence of the event based on a third rule after determining that the event has occurred; transmitting the third query to the vision-language model so that the vision-language model determines whether an action is being performed in accordance with the occurrence of the event; and providing information that an action is being performed in accordance with the occurrence of the event after determining from the output of the vision-language model that an action is being performed in accordance with the occurrence of the event.

[0092] According to one embodiment of the present application, the second query may be generated after obtaining information that an action is being performed in response to the third query in accordance with the occurrence of the event.

[0093] According to one embodiment of the present application, the second query is generated at least once for a first time interval included between the time when the event is determined to have occurred and the time when the event is determined to have ended, and is transmitted to the vision-language model, and the third query is generated at least once for a second time interval included between the time when the event is determined to have occurred and the time when the event is determined to have ended, and is transmitted to the vision-language model, wherein at least a portion of the time intervals of the first time interval and the second time interval may overlap.

[0094] According to one embodiment of the present application, the step of generating the second query may further include the step of generating the second query by obtaining third image-based information based on the second rule in response to a determination that an action is being performed in accordance with the occurrence of the event, and using the third image-based information as a first input and the information that an action is being performed in accordance with the occurrence of the event based on the second image as a second input.

[0095] According to one embodiment of the present application, the method may further include the step of, in response to determining that the event has ended: stopping the query of the second query and the third query to the vision-language model, regenerating the first query regarding whether the event occurred, and querying the vision-language model with the regenerated first query.

[0096] According to one embodiment of the present application, the method may further include the step of, in response to determining that an action is being performed in accordance with the occurrence of the event, suspending the query of the third query against the vision-language model and maintaining the query of the second query against the vision-language model.

[0097] According to one embodiment of the present application, the first query is transmitted to the vision-language model at a predetermined first time period, and the second query or the third query may be transmitted to the vision-language model at a predetermined second time period different from the first time period.

[0098] According to one embodiment of the present application, the first time period may be characterized as being shorter than the second time period.

[0099] According to one embodiment of the present application, the step of providing information that the event has ended may further include: a step of obtaining an event end time determined to have ended the event based on the output of the vision-language model; and a step of outputting information that the event has ended at a time when a predetermined time has elapsed from the event end time.

[0100] According to one embodiment of the present application, the method may further include the step of maintaining information that the event has occurred, without outputting information that the event has ended, when it is determined that the event has occurred again between the time when the event ends and the time when the predetermined time has elapsed.

[0101] According to one embodiment of the present application, the method may further include the step of, when it is determined that an event has occurred again between the event termination time and the time after a predetermined time has elapsed, maintaining a query to the vision-language model of the second query for the predetermined time, or generating a fourth query regarding whether the event has started again and querying the vision-language model.

[0102] According to one embodiment of the present application, based on the second query and the output of the vision-language model, the event may be configured to be determined to have ended after it is detected that the event has ended for a predetermined number of times.

[0103] According to one embodiment of the present application, the first query may be configured to include a plurality of subqueries, wherein if it is detected that the event has occurred for at least one of the plurality of subqueries, the event may be determined to have occurred.

[0104] According to one embodiment of the present application, the method may further include: a step of providing information indicating a normal state during a first progress interval comprising at least a portion between the time when the first query is generated and the time when it is determined that the event has occurred; a step of providing information indicating that the action for the event has not yet started during a second progress interval comprising at least a portion between the time when the third query is generated and the time when it is determined that the action following the occurrence of the event is being performed; and a step of providing information indicating that the action for the event is being performed during a third progress interval comprising at least a portion between the time when the second query is generated and the time when it is determined that the event has ended.

[0105] According to one embodiment of the present application, the step of generating a first query regarding whether the event occurs further includes the step of obtaining information based on the first image from a first media processing pipeline corresponding to the first rule, and the step of generating a second query regarding whether the event ends includes the step of obtaining information based on the second image from a second media processing pipeline corresponding to the second rule, wherein at least one model different from the model executed in the first media processing pipeline may be executed in the second media processing pipeline.

[0106] According to one embodiment of the present application, the method further comprises the step of storing event information after determining that the event has occurred, wherein the event information may include information regarding at least one of an identifier for a channel that captured the video used to analyze the event, a unique identifier (UUID), an event type, an identifier for a query input to the vision-language model, a time to live (TTL) for the event information, a risk level of the event, and a 'progress status' of the event.

[0107] According to one embodiment of the present application, the step of generating the second query may further include: determining a change of the first rule to the second rule by referring to the progress of the event of the event information; and generating the second query, which has been verified for performance in advance for the second rule, according to the change.

[0108] According to one embodiment of the present application, the second query may be characterized by being generated to have a chain prompt form with the first query.

[0109] According to one embodiment of the present application, the method may further include the step of updating at least one of the identifier for the query included in the event information, the risk level, the retention time, and the progress status after determining that the event has ended.

[0111] Hereinafter, with reference to FIGS. 1 to 25, a method for constructing a pipeline for event analysis according to one embodiment of the present application, a method for analyzing an event using an inference pipeline (analysis pipeline), a method for providing information about an event, and an analysis server and an event analysis system for performing the same will be described in more detail.

[0112] An event analysis system according to one embodiment of the present application may be configured to construct an inference pipeline for image-based event detection, analyze events using the constructed inference pipeline, and perform an operation of providing information about the events.

[0113] FIG. 1 is a schematic diagram showing an event analysis system (10) according to one embodiment of the present application.

[0114] An event analysis system (10) according to one embodiment of the present application may include a data source server (100), a client terminal (200), and / or an analysis server (1000).

[0115] The data source server (100) can perform the role of supplying data sources to be analyzed by the analysis server (1000) of the event analysis system (10). The data source server (100) can be configured to transmit video data (video sequences) collected from cameras or sensors installed at a specific site to the analysis server (1000). In this case, 'site' refers to a physical location or a management unit, and may be, for example, a construction site, a manufacturing plant, a specific point for traffic control, etc. The data source may include real-time video streams (e.g., streaming via RTSP protocol), stored video files, image sequences, and / or other forms of video data.

[0116] A client terminal (200) is a device used by a user (administrator) to build and manage an inference pipeline. The client terminal (200) provides a graphical user interface (GUI), through which the user can input various configuration commands related to building an inference pipeline and transmit them to an analysis server (1000). The client terminal (200) can be implemented as various types of computing devices, such as desktop computers, laptops, and tablets. Specifically, the client terminal (200) can be implemented through a GUI-based installable operation tool. This operation tool is used by the user to install and manage an artificial intelligence-based event analysis system (10) and can be configured to perform various tasks such as server and channel connection, image processing pipeline package and configuration of artificial intelligence models and inference models, and / or rule configuration.

[0117] The analysis server (1000) can perform the operation of constructing and executing an inference pipeline for multimodal analysis. Specifically, the analysis server (1000) communicates with a client terminal (200) and can construct an inference pipeline based on a configuration command received from the client terminal (200). According to one embodiment, at least one agent module is installed in the analysis server (1000). The agent module can perform the function of receiving commands, configurations, and / or data transmitted from the client terminal (200) and distributing them to appropriate components (resource monitoring module, container management module, log collection module, rule management module, etc.) within the analysis server (1000). Additionally, the agent module can perform the role of synchronizing by serializing processing results or status information on the analysis server (1000) side in real time and transmitting them to the client terminal (200).

[0118] Through this structure, the user can easily perform complex inference pipeline construction tasks through the intuitive GUI of the client terminal (200), and the agent module of the analysis server (1000) can process these tasks by reliably linking with various components inside the analysis server (1000).

[0119] The analysis server (1000) may be any type of server, edge device, PC, tablet PC, smartphone, smart watch, PDA, and / or a combination thereof. Furthermore, the analysis server (1000) may encompass a combination of at least one server. In the analysis server (1000), at least one agent and an inference pipeline may be executed to build an inference pipeline.

[0120] The analysis server (1000) may include a communication module (1100), memory (1200), and / or a processor (1300).

[0121] The communication module (1100) of the analysis server (1000) can communicate with any external device or external server. For example, the analysis server (1000) can receive a video sequence to be analyzed from the data source server (100) through the communication module (1100). For example, the analysis server (1000) can communicate with the client terminal (200) through the communication module (1100) and receive any setting command related to pipeline construction from the client terminal (200) or transmit any data and / or commands related to pipeline construction to the client terminal (200). For example, the analysis server (1000) can obtain any data (e.g., structure information of each model, computation library, parameter information, prompt information, etc.) for executing a first model and / or an inference model (second model) to be executed on the constructed inference pipeline through the communication module (1100). For example, the analysis server (1000) may transmit any event information, which is an analysis result using an inference pipeline, to any external device including a client terminal (200). However, this is merely an example, and the analysis server (1000) may transmit and receive any appropriate data and / or commands to any component through a communication module (1100).

[0122] The analysis server (1000) can connect to a network and transmit and receive various data through a communication module (1100). The communication module (1100) may include a wired type and a wireless type. Since the wired type and the wireless type each have their own advantages and disadvantages, the analysis server (1000) may be equipped with both wired and wireless types depending on the case. Here, in the case of the wireless type, communication methods of the WLAN (Wireless Local Area Network) series, such as Wi-Fi, can be mainly used. Alternatively, in the case of the wireless type, cellular communication, such as LTE or 5G series communication methods, can be used. However, wireless communication protocols are not limited to the examples described above, and it is possible to use any appropriate wireless type of communication method. In the case of the wired type, LAN (Local Area Network) or USB (Universal Serial Bus) communication are representative examples, and other methods are also possible.

[0123] The memory (1200) of the analysis server (1000) can store various information. Various data can be stored in the memory (1200) temporarily or semi-permanently. Examples of memory (1200) may include a hard disk drive (HDD), a solid state drive (SSD), flash memory, ROM (Read-Only Memory), and RAM (Random Access Memory). The memory (1200) may be provided in a form that is built into the analysis server (1000) or a detachable form. The memory (1200) may store various data required for the operation of the analysis server (1000), including an operating system (OS) for running the analysis server (1000) and a program for operating each component of the analysis server (1000).

[0124] The processor (1300) can control the overall operation of the analysis server (1000). For example, the processor (1300) can control the overall operation of the analysis server (1000), including the operation of constructing an inference pipeline to be described later, the operation of performing analysis on events using the constructed inference pipeline, and / or the operation of displaying or providing event information. Specifically, the processor (1300) can load and execute a program for the overall operation of the analysis server (1000) from memory (1200). The processor (1300) can be implemented as an Application Processor (AP), a Central Processing Unit (CPU), a Microcontroller Unit (MCU), or a similar device depending on hardware, software, or a combination thereof. In this case, hardware-wise, it can be provided in the form of an electronic circuit that processes electrical signals to perform control functions, and software-wise, it can be provided in the form of a program or code that drives the hardware circuit. For example, the processor (1300) can be provided in the form of a processing circuitry.

[0126] FIG. 2 is a diagram showing the detailed configuration of an analysis server (1000) and the structure of an inference pipeline according to one embodiment of the present invention.

[0127] Referring to FIG. 2, the analysis server (1000) may include a connection agent (1400) and an inference pipeline (IP), and the client terminal (200) may include a client GUI (210).

[0128] The operation flow of the event analysis system (10) of the present invention is briefly explained as follows.

[0129] First, the user can perform various settings necessary for building an inference pipeline through the client GUI (210) of the client terminal (200). These settings include server connection, channel registration, image processing pipeline package settings, artificial intelligence model / general inference model settings, and / or rule settings (constraint settings). The client GUI (210) provides an intuitive graphical interface to support the user in easily performing complex pipeline building tasks.

[0130] A configuration command entered by a user can be transmitted from a client terminal (200) to a connection agent (1400) of an analysis server (1000). The connection agent (1400) performs the function of transmitting the configuration command received from the client terminal (200) to an appropriate component within the analysis server (1000).

[0131] The analysis server (1000) can perform the operation of building an inference pipeline (IP) based on the received configuration command. In the process of building the inference pipeline (IP), an image processing pipeline (1500) can be configured based on a rule set selected by the user and a target rule (which may also be referred to as constraint information) based on the value setting for the rule item belonging to the rule set.

[0132] When an inference pipeline (IP) is established, the analysis server (1000) can receive a video stream (video sequence) from the data source server (100) and perform analysis. Specifically, the established inference pipeline (IP) can be implemented to perform analysis that detects events present in the video sequence received from the data source server (100) using a determined target rule (constraint information).

[0134] An inference pipeline (IP) according to one embodiment of the present application may include an inference engine (1600) comprising an image processing pipeline (1500) and / or data processing logic (1610) and an inference model (1620).

[0135] The image processing pipeline (1500) may be an intelligence media pipeline that receives various types of input data sources (video streams, images, video clips, etc.) and processes them into a form suitable for artificial intelligence analysis. The image processing pipeline (1500) may be implemented to perform functions such as image decoding, analysis using a computer vision-based artificial intelligence model (e.g., structured vision tasks such as object detection, segmentation, and classification), metadata extraction, and / or image encoding.

[0136] The inference engine (1600) may include data processing logic (1610) and an inference model (1620).

[0137] The data processing logic (1610) is a logic that performs industry domain adaptive data processing and can be configured to operate together with an inference model (1620), which is an industry domain agnostic general-purpose inference model. The data processing logic (1610) can generate a query to be input to the inference model (1620) based on data received from the image processing pipeline (1500), and perform an operation to generate event information by processing the output of the inference model (1620).

[0138] The inference model (1620) is a general-purpose inference model that performs unstructured vision task inference and can be implemented as a multi-modal model including a Vision-Language Model (VLM). The inference model (1620) can be configured to receive queries in the form of images and text as input and generate responses in the form of natural language. Through the inference model (1620), the image processing pipeline can make judgments on various unstructured events that have not been learned in advance.

[0139] Meanwhile, the constraint information (target rule) determined during the process of constructing the inference pipeline (IP) plays an important role throughout the event analysis. The image processing pipeline (1500) can be constructed to perform a first processing by referring to the constraint information, and the data processing logic (1610) can be constructed to perform a second processing (query generation) and a third processing (output parsing of the inference model (1620)) by referring to the constraint information.

[0140] Through this structure, even when using an image processing pipeline (1500) that performs structured tasks, it becomes possible to adapt to analysis tasks of various industrial domains by changing only the constraint information (target rule). For example, by applying constraint information related to industrial safety, it is possible to detect whether PPE is being worn, and by applying constraint information related to traffic, it is possible to detect traffic accident events, thereby enabling various event analyses without retraining the model.

[0141] FIG. 3 is a diagram illustrating an event analysis aspect performed by an event analysis system (10) according to one embodiment of the present application. Specifically, FIG. 3 is a diagram showing the construction of an inference pipeline (IP) and the event analysis process using the inference pipeline (IP) according to one embodiment of the present invention.

[0142] The construction of the inference pipeline and event analysis according to the present invention proceed in the following order:

[0143] S1) Receive pharmaceutical information

[0144] The data processing logic (1610) of the analysis server (1000) can obtain constraint information from the client terminal (200). The constraint information may be generated based on a rule set and a rule set by the user through the client GUI, and the constraint information may include a constraint identifier (e.g., rule set identifier, rule identifier), first information for configuring an image processing pipeline, second information regarding a query template, third information regarding query generation logic, and / or fourth information regarding a parser for processing the output of an inference model (1620). The received constraint information may be used to construct an inference pipeline or to analyze events.

[0145] S2) Integration of image processing pipelines based on inference requirements

[0146] The data processing logic (1610) can be configured to determine inference requirements in the image processing pipeline (1500) based on constraint information and to transmit them to the image processing pipeline (1500). The inference requirements are information that defines how the image processing pipeline (1500) should be configured and operated, and what form of data should be output.

[0147] According to one embodiment, the data processing logic (1610) may be implemented to link the image processing pipeline (1500) and the inference engine (1600) by transmitting inference requirements to the image processing pipeline, including the type and number of artificial intelligence models to be executed in the image processing pipeline (1500), and / or requirements for the task to be performed, Region of Interest information, Confidence Threshold, Crop, Padding, Frame Drop Rate, and / or Frame Interval, and / or the type of object to be detected. Furthermore, the inference requirements may further include the type of computer vision task, such as object detection, segmentation, classification, object tracking, and / or pose estimation, corresponding to a specific unstructured task within a specific industrial domain, the type of pre-trained computer vision model, and / or limitations on training data.

[0148] The image processing pipeline (1500) sets the internal configuration of the image processing pipeline (1500) according to the received inference requirements.

[0149] S3) Reception of video sequence and first processing

[0150] After the inference pipeline is established, the image processing pipeline (1500) can receive a video sequence from the data source server (100). The video sequence may be in various forms, such as a real-time RTSP stream, a stored video file, or an image sequence, and may be a video sequence for a specific channel connected during the inference pipeline establishment process.

[0151] The image processing pipeline (1500) can perform a first processing on a received video sequence. The first processing is performed by referring to constraint information, and the first processing can be performed by structured vision task inference using a computer vision-based artificial intelligence model.

[0152] Examples of structured vision tasks that can be performed in the first processing are as follows:

[0153] Object Detection: A task that detects objects such as people, vehicles, and equipment within an image.

[0154] Segmentation: A task that divides the boundaries of objects within an image into pixel units.

[0155] Classification: A task that classifies the detailed categories of detected objects (e.g., wearing / not wearing a safety helmet)

[0156] Object Tracking: A task that tracks the same object across consecutive frames to determine its movement path.

[0157] Pose Estimation: A task that identifies the location of each body part by representing an object's pose using keypoints.

[0158] S4) Output generation of the image processing pipeline (1500)

[0159] As a result of the first processing, the image processing pipeline (1500) can generate an output through an artificial intelligence model (first model) executed in the image processing pipeline (1500), and the output of the first model may include object-based metadata and / or analysis data.

[0160] Object-based metadata is raw data extracted through the first processing and may include bounding box coordinates, a class identifier associated with the category of the detected object, a confidence score, a segmentation mask, keypoint coordinates and confidence, and / or a tracking identifier.

[0161] The analysis data is high-level information extracted for a specific industry domain by processing metadata, and may include metrics, embeddings which are data representing the features of an object in vector form, original or cropped images used for analysis, and / or constraint identifiers (rule set identifiers, and / or rule identifiers) indicating which rules were used for processing.

[0162] Meanwhile, the image processing pipeline (1500) can be configured to output a data set having a specific type of structure by referring to constraint information.

[0163] S5) Query generation

[0164] The data processing logic (1610) may be implemented to generate a query to be input into the inference model (1620) of the inference engine (1600) by referring to constraint information. This process corresponds to a second processing step and proceeds to the following detailed steps:

[0165] S5-1) Query Template Acquisition: Data processing logic (1610) can acquire a query template corresponding to a constraint identifier (rule identifier) ​​from the Rules / Queries DB. The query template is composed in the form of a natural language question including variables.

[0166] S5-2)Variable operation: Data processing logic (1610) can be implemented to perform operations on variables of a query template using metadata and analysis data included in the output of an image processing pipeline (1500) by referring to constraint information.

[0167] S5-3) Query Generation: Data processing logic (1610) can be implemented to generate a query to be input into the inference model (1620) of the inference engine (1600) by replacing variables of the query template with calculated values. In particular, the generated query can be input into the inference model (1620) along with the image (or cropped image) used for analysis.

[0168] S6) Output generation through the inference model

[0169] The inference model (1620) receives queries and images from the data processing logic (1610). It can be configured to receive and perform unstructured vision task inference to generate output in the form of text.

[0170] The inference model (1620) can be implemented as a multi-modal model including a Vision-Language Model (VLM), and can understand and judge the context of images and text even in various situations that the image processing pipeline has not learned in advance.

[0171] The output of the inference model (1620) is returned to the data processing logic (1610) so that subsequent processing can proceed.

[0172] S7) Inference Model Output Processing

[0173] The data processing logic (1610) can be implemented to process the output returned through the inference model (1620). This process corresponds to a third processing step and can proceed to the following detailed steps.

[0174] S7-1) Parser acquisition and parsing: The data processing logic (1610) can be implemented to obtain parser information corresponding to a constraint identifier (rule identifier) ​​in the Rules / Queries DB and to parse the natural language output of the inference model into structured data using the target parser corresponding to the parser information.

[0175] S7-2) Combine results : The data processing logic (1610) can be implemented to combine the output of the inference model (1620) parsed through the target parser with the output of the image processing pipeline (1500) (e.g., metrics, embeddings, metadata, etc.). This allows the results of the structured vision task and the results of the unstructured vision task to be integrated.

[0176] S7-3) Event Judgment : The data processing logic (1610) can determine whether an event has occurred based on the combined information.

[0177] S8) Create event information

[0178] The data processing logic (1610) can be implemented to generate event information based on the processing result of S7. The event information is generated in a structured form and may include at least one of the following information.

[0179] 1) Basic Information:

[0180] - Channel Identifier: ID of the channel (camera) where the event occurred

[0181] - Unique Identifier (UUID): An ID that uniquely identifies an event

[0182] - Event Type: Category of the detected event (e.g., "No PPE," "Collision Risk," "Fire," etc.)

[0183] - Timestamp: Time of event occurrence

[0184] 2) Analysis Data:

[0185] - Videos or images used for analysis

[0186] - Embeddings or metrics related to the result of the first process

[0187] - Object-based metadata (bounding boxes, classes, confidence levels, etc.)

[0188] 3) Pharmaceutical Information:

[0189] - Rule set identifier

[0190] - Rule identifier

[0191] - Identifier for the applied query

[0192] 4) Event properties:

[0193] - Risk Level: A score or level indicating the severity of an event

[0194] - Status: Current stage of the event (e.g., "Event occurred", "Event being acted upon", "Event ended", etc.)

[0195] - Time to Live (TTL): The period during which event information is retained in memory.

[0196] Meanwhile, the generated event information can be transmitted to a client terminal (200) and displayed to the user in real time through a client GUI, thereby enabling the user to monitor the situation at the site and respond quickly.

[0197] Additionally, the generated event information is stored in the short-term memory of the data processing logic (1610) and can be used as context information during subsequent event analysis. The stored event may be retained until the retention time (TTL) has elapsed or it is explicitly deleted.

[0199] Hereinafter, with reference to FIGS. 4 to 18, a method for constructing an inference pipeline according to one embodiment of the present application, and / or a method for analyzing events using an inference pipeline, will be described in more detail.

[0200] Specifically, with reference to FIGS. 4 to 11, a method for constructing an inference pipeline will be described in more detail, and with reference to FIGS. 12 to 18, a method for analyzing an event using an inference pipeline according to one embodiment of the present application will be described in more detail.

[0201] Meanwhile, in describing the method for constructing an inference pipeline and / or the method for analyzing events using the inference pipeline, some embodiments that overlap with the content previously described in relation to FIGS. 1 to 3 may be omitted. However, this is merely for the convenience of explanation and should not be interpreted as being limited thereto.

[0202] FIG. 4 is a flowchart illustrating a method for constructing an inference pipeline (analysis pipeline) according to one embodiment of the present application. The method for constructing an inference pipeline according to FIG. 4 can be performed by an analysis server (1000) communicating with a client terminal (200).

[0203] A method for constructing an inference pipeline according to one embodiment of the present application may include the steps of: obtaining an image processing pipeline package and an inference engine for multimodal analysis (S1100); obtaining, through a client terminal, a description command for a target rule set related to a specific industry domain among a plurality of rule sets compatible with the inference engine (S1200); obtaining at least one image processing pipeline from the image processing pipeline package that satisfies the inference requirements of the target rule set based on a setting command for the target rule set (S1300); and / or constructing an inference pipeline based on at least one image processing pipeline and an inference engine (S1400).

[0204] In the step (S1100) of acquiring an image processing pipeline package and an inference engine for multimodal analysis, the analysis server (1000) can acquire an image processing pipeline package and an inference engine for multimodal analysis.

[0205] An image processing pipeline package is a set of pipelines that receive various types of input data sources (video streams, images, video clips, etc.) and process them into a form suitable for artificial intelligence analysis. Each image processing pipeline included in the package can perform functions such as video decoding, AI model-based analysis, metadata extraction, and / or video encoding.

[0206] The inference engine (1600) may include a domain-agnostic general-purpose inference model (1620) and a domain-adaptive data processing logic (1610). The general-purpose inference model (1620) is a model capable of performing unstructured vision tasks in various industrial domains without being limited to a specific industrial domain, and may be implemented as a Vision-Language Model (VLM). The domain-adaptive data processing logic (1610) may include logic for preprocessing data to be input to the general-purpose inference model (1620) and post-processing the output of the general-purpose inference model (1620) so that the general-purpose inference model (1620) can be utilized to meet the analysis requirements of a specific industrial domain.

[0207] The general-purpose inference model (1620) of the image processing pipeline package and / or inference engine (1600) acquired by the analysis server (1000) in step S1100 is characterized by being selected in a form optimized for the hardware specifications of the server on which the analysis is to be performed.

[0208] Specifically, an image processing pipeline package can be selected from a pipeline pool containing multiple image processing pipelines optimized for different hardware, comprising at least one image processing pipeline whose performance has been verified against the hardware specification information of the server where analysis is to be performed. The present invention is characterized by first selecting the hardware on which analysis is to be performed and then selecting an image processing pipeline by considering the specifications of the said hardware. This is because there are image processing and analysis SDKs capable of optimizing computations for each hardware, and these must be utilized to configure a pipeline optimized for the hardware. Therefore, the image processing pipeline can be managed and constructed in a form optimized for each hardware.

[0209] For example, a pipeline pool may include hardware-specific image processing pipelines such as the following:

[0210] · Pipelines for NVIDIA GPUs: Built on the NVIDIA Deepstream SDK and optimized to fully utilize CUDA Cores and Tensor Cores. Suitable for data center-class GPUs such as NVIDIA Blackwell or edge devices such as the Jetson series.

[0211] · Pipeline for Qualcomm processors: Built using the Qualcomm IM SDK and optimized to utilize Qualcomm's hardware accelerators.

[0212] · Pipeline for general-purpose NPU: Built for NPUs such as IMX 500 and NXP or other low-end processors, based on custom plugins utilizing general-purpose image processing frameworks such as gstreamer and opencv.

[0213] At this time, the client terminal (200) can be implemented to check hardware specification information (e.g., GPU model name, CPU architecture, type of media accelerator, etc.) of the server to be analyzed, and to select and provide to the user an image processing pipeline package whose performance has been verified for hardware corresponding to the hardware specification information from the pipeline pool. The hardware specification information can be calculated by the analysis server (1000) based on server information (e.g., server name, IP address, port number) obtained through the server addition function via the client terminal (200) and transmitted to the client terminal (200). The hardware specification information becomes a key criterion for selecting the type of image processing pipeline (1500) and / or inference engine (1600) to be installed on the analysis server (1000).

[0214] For example, for NVIDIA GPUs, a pipeline based on the Deepstream SDK can be selected, while for Qualcomm processors, a pipeline utilizing the IM SDK can be selected. Additionally, for the IMX 500, NXP, or other NPUs, if an SDK is available for the hardware, it should be utilized; otherwise, a custom plugin-based pipeline using general-purpose image processing frameworks such as gstreamer or opencv can be selected as a performance-verified pipeline. Specifically, if the server where analysis is performed is equipped with an NVIDIA Blackwell GPU, a high-performance pipeline package based on the Deepstream SDK can be selected to build an inference pipeline capable of processing large video streams in real time; conversely, if it is an edge server equipped with the Jetson series, a lightweight Deepstream-based pipeline package optimized for low-power environments can be selected to build the inference pipeline.

[0216] In the step (S1200) of obtaining a description command for a target rule set related to a specific industry domain among a plurality of rule sets compatible with the inference engine through a client terminal, the analysis server (1000) can obtain a setting command for a target rule set related to a specific industry domain among a plurality of rule sets compatible with the inference engine (1600) through a client terminal (200).

[0217] A rule set defines inference requirements for structured vision tasks to be performed in an image processing pipeline (1500) for inference of unstructured vision tasks using a general-purpose inference model (1620) for a specific industry domain. One or more rule sets may be created per industry domain and may be configured to match the characteristics of events to be detected in the corresponding industry domain.

[0218] The inference requirements of each rule set may include types of structured vision inference tasks corresponding to any one of a plurality of predefined unstructured vision inference tasks within any one industrial domain. The types of structured vision inference tasks may include at least one of object detection, segmentation, classification, object tracking, and pose estimation.

[0219] Additionally, each rule set may include rule items for generating input data for a general-purpose inference model (1620) from the output of an image processing pipeline (1500) included in an image processing pipeline package for each industrial domain, and parameters that can be changed by the user. By inputting setting values ​​for these changeable parameters, the user can fine-tune the inference pipeline (IP) to suit the field conditions.

[0221] Hereinafter, with reference to FIGS. 5 and 6, an aspect of receiving a setting command for a target rule set from a client terminal (200) according to an embodiment of the present application will be described in more detail. FIG. 5 is a drawing illustrating an exemplary graphical user interface (210) of a client terminal (200) for explaining an aspect of receiving a setting command for a target rule set from a client terminal (200) according to an embodiment of the present application. FIG. 6 is a drawing illustrating an exemplary graphical user interface (210) of a client terminal (200) for explaining an aspect of receiving a setting command for a target rule set from a client terminal (200) according to an embodiment of the present application.

[0222] Referring to FIGS. 5 and 6, the user can perform a series of processes to generate target rules by selecting a target rule set through the GUI (210) of the client terminal (200), selecting a specific channel to which the rule set will be applied, and entering setting values ​​for changeable parameters belonging to the target rule set.

[0223] The screen according to FIG. 5(a) shows a user selecting a target rule set. The analysis server (1000) can be implemented to provide the user with multiple rule sets compatible with the inference engine (1600) through the client terminal (200). For example, referring to FIG. 5(a), multiple rule sets predefined by industrial domain (e.g., [Industrial Safety] PPE wearing violation detection, [Industrial Safety] worker detection, etc.) may be displayed.

[0224] The user can select a rule set suitable for the situation to be analyzed from among these list of rule sets. For example, if the user wants to detect whether a worker is wearing safety equipment at an industrial site, the user can select the '[Industrial Safety] PPE Wearing Violation Detect' rule set. The analysis server (1000) can receive input to select a target rule set through the user's selection. Each rule set defines inference requirements for a structured vision inference task of an artificial intelligence model to be executed on an image processing pipeline (1500) prior to performing an unstructured vision inference task using a general inference model (1620) for a specific industrial domain (e.g., industrial safety, traffic control, public safety, etc.). For example, the '[Industrial Safety] PPE Wearing Violation Detect' rule set may include structured vision tasks such as object detection (person detection) tasks and classification (classification of whether safety equipment is worn) tasks.

[0225] The screen according to FIG. 5(b) shows a manner of selecting a specific channel to which a selected set of target rules will be applied. The client terminal (200) may be implemented to provide the user with at least one list of channels to which a set of target rules will be applied.

[0226] As can be seen in '[Industrial Safety] PPE Wearing Violation Detect (A) - Rule Creation' in FIG. 5(b), the state in which a user has selected a rule set for 'PPE Wearing Violation Detect' is shown as an example. The user can select a specific channel from the list to which the target rule set is to be applied. For example, the user can select a channel corresponding to "Camera 01" which films a work area (site) at an industrial site. The analysis server (1000) can receive input selecting a specific channel to which the target rule set is to be applied through the user's selection.

[0227] FIG. 6 illustrates a process in which a user sets changeable parameters belonging to a target rule set for a selected target rule set. An analysis server (1000) may be implemented to provide a screen to the user for selecting a setting value for a changeable parameter belonging to a target rule set through a client terminal (200). The changeable parameter includes field-adaptive parameters and may include a first parameter regarding an image processing pipeline (1500) and a second parameter regarding a general-purpose inference model (1620). The first parameter is a parameter for analysis through an image processing pipeline (1500) and may include a Crop regarding the size of the area around the detected object, a Padding regarding the size of the margin to be additionally included from the object boundary when cropping, a Confidence Threshold regarding the minimum confidence score for considering the detection result of the artificial intelligence model as valid, a Region of Interest to perform analysis, a Framedrop rate to adjust the analysis load, a Frame interval regarding the time interval between frames to perform analysis, and / or at least one parameter regarding the type of object to be detected. The second parameter is a parameter related to a query to be input to a general-purpose inference model (1620) and may include a query template to be used among a plurality of predefined query templates, and / or a parameter regarding the response form of the general-purpose inference model. The analysis server (1000) may receive input regarding the setting value of the changeable parameter through user input regarding the changeable parameter.

[0228] For example, in the case of the '[Industrial Safety] PPE Wearing Violation Detect' rule set shown in Fig. 6, the user can set specific parameters such as the following.

[0229] For example, in the default settings, the user can specify the type of protective equipment to be detected by setting the PPE Set to 'safety helmet, safety shoes, fluorescent, dust mask, vest'. The image processing pipeline (1500) can be configured to run an artificial intelligence model that detects or classifies whether such equipment is being worn. The user can set the cropping padding value to '20px' to add a margin of 20 pixels from the bounding box when cropping a detected person object. This is to include sufficient context in case the object boundary is inaccurate. The user can also set the confidence threshold to '0.4' so that only results classified by the artificial intelligence model with a confidence of 40% or higher are considered valid detections, thereby eliminating false positives with low confidence.

[0230] In the Area of ​​Interest (ROI) settings, users can configure Static ROI and / or Dynamic ROI to designate particularly hazardous areas or priority management areas within the work zone. For example, users can set the area where heavy equipment is operated as the ROI to detect whether workers within that area are wearing PPE.

[0231] In the prompt settings item, the user can specify a query template to check whether a worker is wearing a fluorescent vest by selecting 'whether the worker is wearing a vest' from the question selection. The user can set the query (prompt) to "Find someone who is not wearing a high-visibility fluorescent clothing in the image, if there is, answer 'Yes'. Otherwise, answer 'No'." or modify the initial query template, and can instruct the general inference model (1620) to query to find a person who is not wearing fluorescent clothing in the image and to answer 'Yes' if there is one, and 'No' if there is no one.

[0232] By setting these changeable parameters, the analysis behavior can be optimized to suit site characteristics (e.g., camera resolution, lighting conditions, layout of work areas, types of PPE to be managed, etc.) while using the same set of 'PPE wearing violation detection' rules.

[0233] Meanwhile, FIGS. 5 and 6 illustrate the setting of specific values ​​for a specific set of rules and specific parameters. However, this is merely an example for the convenience of explanation and should not be interpreted restrictively.

[0234] The data processing logic (1610) of the analysis server (1000) according to one embodiment of the present application may perform an operation to generate a target rule based on a first input for selecting a target rule set from a client terminal (200), a second input for selecting a specific channel to which the target rule set is to be applied, and / or a third input for a setting value of a changeable parameter. The target rule may include a specific rule for generating input data of an inference engine (1600) generated by applying a setting value for a changeable parameter to a rule item belonging to the target rule set for a specific channel.

[0235] For example, with reference to FIGS. 5 and 6, for a specific channel 'Camera 01', a target rule for generating input data for an inference engine (1600) can be generated by applying setting values ​​for changeable parameters (PPE Set = "safety helmet, safety shoes, fluorescent, dust mask, vest", Cropping padding = 20px, Confidence Threshold = 0.4, ROI = Static ROI 1 and Static ROI 2, Query = "Find someone who is not wearing a high-visibility fluorescent clothing in the image, if there is, answer 'Yes'. Otherwise, answer 'No'.") to rule items (PPE Set, Cropping padding, Confidence Threshold, ROI, Query Template) belonging to the target rule set '[Industrial Safety] Detecting Violation of PPE Wearing'. The target rule generated in this way can be utilized in the process of building an inference pipeline (IP), and for a video sequence received from the 'Camera 01' channel, an image processing pipeline (1500) can be implemented to perform an analysis regarding the detection of PPE wearing violations according to the specified parameters (setting values) of the target rule.

[0236] According to one embodiment of the present invention, the analysis server (1000) can perform an operation of applying specific constraints by considering existing set rules when generating target rules.

[0237] According to one aspect, the data processing logic (1610) may be implemented to check rule information previously assigned to a specific channel when generating a target rule for that channel. Specifically, the data processing logic (1610) may obtain first rule set information and first rule information previously assigned to a specific channel. Additionally, the data processing logic (1610) may receive a setting command for a candidate rule set and a candidate rule based on first to third inputs obtained through a client terminal (200).

[0238] At this time, the data processing logic (1610) can compare the first rule set information and the candidate rule set to determine whether they are the same rule set. If the first rule set information and the candidate rule set are the same set, the data processing logic (1610) can restrict the candidate rule from being received as a target rule suitable for analysis regarding a specific channel. This is to prevent different rules inheriting the same rule set from being assigned identically to a single channel. On the other hand, if the first rule set information and the candidate rule set are different sets, the data processing logic (1610) can receive the candidate rule as a target rule suitable for analysis regarding a specific channel. Through this, different rules inheriting different rule sets can be assigned to the same channel.

[0239] For example, if "Rule A" belonging to the [Industrial Safety] PPE Wearing Violation Detect rule set is already assigned to the 'Camera 01' channel, and the user attempts to assign "Rule B" belonging to the same [Industrial Safety] PPE Wearing Violation Detect rule set to "Camera 01", the data processing logic (1610) may restrict this to prevent duplicate assignment of different rules belonging to the same rule set. However, if the user attempts to assign "Rule C" belonging to a different rule set, "[Industrial Safety] Worker Detect", to "Camera 01", the data processing logic (1610) may allow this so that various types of analysis can be performed simultaneously on a single channel.

[0240] According to one aspect, the data processing logic (1610) may be implemented to refer to rule information previously assigned to another channel when generating a target rule for a specific channel. Specifically, the data processing logic (1610) may obtain second rule set information and second rule information previously assigned to a specific channel and another channel. Additionally, the data processing logic (1610) may receive a setting command for a candidate rule set and a candidate rule based on first to third inputs obtained through a client terminal (200).

[0241] Regarding constraints on other channels, the data processing logic (1610) may receive the candidate rule as a target rule suitable for analysis regarding a specific channel even if the second rule set information and the candidate rule set are the same rule set, provided that the second rule information and the candidate rule are different. This is to allow different rules inheriting the same rule set to be assigned to different channels.

[0242] For example, if "Rule A" belonging to the [Industrial Safety] PPE Wearing Violation Detection rule set is assigned to the 'Camera 02' channel, and the user attempts to assign "Rule B" belonging to the same [Industrial Safety] PPE Wearing Violation Detection rule set to the 'Camera 01' channel, the data processing logic (1610) can be implemented to allow this. Through this, different rules within the same rule set (e.g., rules with different parameter settings) can be distributed and applied to multiple channels, making it possible to perform analysis tailored to the characteristics of each channel.

[0243] Through these constraints, the event analysis system (10) of the present invention supports a flexible configuration while maintaining consistency in rule setting, thereby preventing unintended duplicate settings or conflicts and simultaneously providing the effect of effectively processing various analysis requirements.

[0244] According to one embodiment of the present invention, a target rule generated according to a setting command for a target rule set obtained through step S1300 may include various information required for the construction and execution of an inference pipeline (IP) in a structured form.

[0245] According to one embodiment, the target rule may include at least one of a rule set identifier corresponding to the target rule set (e.g., a unique identifier of the rule set such as '[Industrial Safety] Detecting PPE Wearing Violation'), a rule identifier corresponding to the target rule (e.g., a rule identifier for "PPE_Rule_Rule A", a rule identifier for "PPE_Rule_Rule B"), first information regarding a parameter or task of the image processing pipeline for configuring the output form of the image processing pipeline (1500), second information regarding a query template associated with the rule identifier, third information for generating a query as input data to be input to a general inference model (1620) based on the query template and the output of the image processing pipeline (1500), and / or fourth information regarding logic associated with the rule identifier for processing the output of the general inference model (1620).

[0246] Through the rule set identifier, it can be clearly identified which rule set a target rule belongs to, and through the rule identifier, rules with different parameter settings can be distinguished even within the same rule set.

[0247] The first information may include information about what structured vision tasks the image processing pipeline (1500) should perform (e.g., object detection, classification, object tracking, pose estimation), what parameters should be applied (e.g., object type to be detected, Confidence Threshold, Cropping padding), what region of interest (ROI) should be analyzed, etc.

[0248] The query template included in the second information defines the basic form of the query (or prompt) to be input into the general-purpose inference model (1620) and may include variables.

[0249] The third information may include logic that defines what value to assign to a variable of the query template and how to generate the query by utilizing which output (metadata, analysis data, metric, etc.) of the image processing pipeline (1500) when generating the query.

[0250] The fourth information may include parser information that defines how to parse and interpret the output of the general-purpose inference model (1620). For example, the fourth information may include information about a parser that converts a response in the form of "Yes" / "No" into a structured value, or a parser that extracts specific keywords from a free-form response.

[0251] Referring again to FIG. 4, in the step (S1300) of obtaining at least one image processing pipeline from an image processing pipeline package that satisfies the inference requirements of a target rule set based on a setting command for a target rule set, the analysis server (1000) may be implemented to obtain at least one image processing pipeline (1500) from an image processing pipeline package that satisfies the inference requirements according to the target rule set and / or the target rule based on the setting command for a target rule set obtained through step S1200. Specifically, in step S1300, an image processing pipeline (1500) capable of performing structured vision tasks required by the target rule set is selected from the image processing pipeline package. For example, if the target rule set requires an object detection task, an image processing pipeline (1500) capable of performing such tasks may be selected. For example, if the target rule set requires a specific type of task, an image processing pipeline (1500) capable of optimally executing at least one artificial intelligence model capable of performing a specific type of task may be selected.

[0252] Meanwhile, regarding inference requirements, the description has focused on the types of tasks. However, this is merely an example, and the inference requirements may further include the types of computer vision tasks such as object detection, segmentation, classification, object tracking, and / or pose estimation corresponding to specific unstructured tasks within a specific industrial domain, the types of pre-trained computer vision models, and / or limitations on training data. Furthermore, the inference requirements according to one embodiment may further include requirements for a pre-configured type of image processing pipeline or a combination of multiple pre-configured types of image processing pipelines.

[0253] In the step (S1400) of constructing an inference pipeline based on at least one image processing pipeline and an inference engine, the analysis server (1000) can construct an inference pipeline (IP) based on at least one image processing pipeline (1500) and an inference engine (1600) that satisfy the inference requirements obtained through the step S1300.

[0254] According to one embodiment, the data processing logic (1610) of the analysis server (1000) may be implemented to configure an image processing pipeline (1500) based on target rules belonging to a set of target rules (the target rules include rule items and setting values ​​of changeable parameters for each rule item). Specifically, the data processing logic (1610) may configure the image processing pipeline (1500) to perform structured vision analysis corresponding to inference requirements corresponding to the target rules by applying target rules to specific channels based on at least one image processing pipeline (1500) and target rules.

[0255] Specifically, the data processing logic (1610) can transmit inference requirements defined in the target rule (e.g., parameters such as a region of interest (ROI), confidence threshold, detected object type, crop, padding, frame drop rate, and / or frame interval) to the image processing pipeline (1500) so that at least one artificial intelligence model to be executed in the image processing pipeline (1500) can be configured to operate according to the requirements. For example, if the target rule is defined to detect a person with a confidence of 0.4 or higher and classify whether a safety helmet is worn within the region of interest of Static ROI 1 for the 'Camera 01' channel, the image processing pipeline (1500) can be configured to execute an object detection model and / or classification model on the ROI area and output only results with a confidence of 0.4 or higher.

[0256] According to one embodiment, the data processing logic (1610) of the analysis server (1000) may be implemented to generate structured data that defines the input data of the inference engine (1600) in response to the input of a setting value for a parameter that can be changed through user input. The data processing logic (1610) may configure an image processing pipeline (1500) to generate input data of the inference engine (1600) for a specific channel using the structured data. Specifically, the image processing pipeline (1500) may be configured to output structured data including a rule set identifier, a rule identifier, an image, and / or metadata and analysis data required for generating input data of the inference engine (1600). Through this, when the data processing logic (1610) receives the output of the image processing pipeline (1500), it can generate the output of the image processing pipeline (1500) into an appropriate query having the form of structured data and transmit it to the inference model (1620).

[0257] Additionally, the data processing logic (1610) can link the image processing pipeline (1500) and the inference model (1620) using the identifier of the target rule set. The identifier of the target rule set serves as a criterion for determining which general-purpose inference model to map among a plurality of general-purpose inference models, and the data processing logic (1160) enables the linkage by mapping the correct inference model (1620) to which the query will be input using the rule set identifier included in the output of the image processing pipeline (1500).

[0258] Through this, the inference engine (1600) of the inference pipeline (IP) and the image processing pipeline (1500) are linked together to construct the inference pipeline (IP).

[0259] The constructed inference pipeline receives a video sequence from a data source server (100), performs structured vision tasks through an image processing pipeline (1500), and performs unstructured vision tasks through an inference engine (1600), thereby enabling event detection. Through this construction method, the effect of rapidly constructing an inference pipeline (IP) that can adapt to various industrial domains by simply changing the target rule set can be provided.

[0261] Below, the aspects of constructing an inference pipeline (IP) will be described in more detail with reference to FIGS. 7 to 9.

[0262] FIG. 7 is a detailed flowchart of the step (S1400) of constructing an inference pipeline according to one embodiment of the present application. FIG. 8 is a diagram illustrating an exemplary graphical user interface (210) of a client terminal (200) to explain one aspect of receiving a setting command for allocating resources for an image processing pipeline (1500) from a client terminal (200) according to one embodiment of the present application. FIG. 9 is a diagram illustrating an exemplary graphical user interface (210) of a client terminal (200) to explain one aspect of receiving a setting command for selecting a channel to be linked with an artificial intelligence model to be executed on the image processing pipeline (1500) from a client terminal (200) according to one embodiment of the present application.

[0263] A step of constructing an inference pipeline according to one embodiment of the present application (S1400) may further include a step of allocating resources for an image processing pipeline (S1410), a step of setting an artificial intelligence model to be executed on the image processing pipeline (S1420), a step of setting a channel to receive images to be analyzed on the inference pipeline (S1430), and / or a step of creating a container of the inference pipeline based on the allocated resources, the set artificial intelligence model, and the set channel (S1440).

[0264] Referring to FIG. 8, in relation to step S1410, the user can set hardware resources to be allocated to instances of at least one image processing pipeline (1500) obtained through step S1300 via the GUI (210) of the client terminal (200). The GUI (210) can visually display the status of the server's hardware resources (CPU, GPU, memory), and the user can input the memory capacity to be used by instances of the image processing pipeline (1500) into the reserved memory and restricted memory fields. For example, the screen may display the current memory usage status and recommended values, such as '4.2GB / 175GB in use (3GB recommended)'. Memory allocation is intended to ensure that processes using the image processing pipeline (1500) operate smoothly within the allocated memory resources without interference with other processes. If resources are not allocated in advance, if one image processing pipeline abnormally overuses memory, it may affect other image processing pipelines or processes of the inference engine.

[0265] Additionally, the user can enter the number of CPU cores that an instance of the image processing pipeline (1500) will use in the CPU field.

[0266] Additionally, the user selects a GPU to be assigned to an instance for an image processing pipeline (1500) in the GPU section. For example, the screen of FIG. 8 may display available GPU indices, the number of image processing pipelines currently assigned to each GPU, and the number of runners for the inference model. For instance, GPU 1 may display "Number of assigned image processing pipelines (IMP): 0, Number of assigned inference model (EFM) runners: 1," allowing the user to understand the resource distribution situation and select an appropriate GPU.

[0267] Referring to the screen according to FIG. 9(a), in relation to step S1420, the user can select an artificial intelligence model to be executed on the image processing pipeline (1500). The GUI (210) may be implemented to display a list of previously uploaded artificial intelligence models. For example, the model name and the ID of each model may be displayed on the screen in a table format.

[0268] At this time, the selectable artificial intelligence models may be models selected from a model pool containing multiple models optimized for different hardware, which have been verified for performance with respect to the hardware specifications of the server where the analysis is to be performed. The selectable artificial intelligence models may be models that have been quantized to meet the computational power and inference requirements of specific hardware (GPU). For example, the list may include an artificial intelligence model that meets the inference requirements for detecting an object called 'person' (e.g., computer vision model), a general-purpose inference model for detecting 'human behavior' (e.g., vision-language model), an optimized model to provide fast inference speed and precision with low power consumption on edge devices, and a model that provides high accuracy on high-performance servers, all of which have been verified for performance with respect to the hardware specifications of the server where the analysis is to be performed. The user may select at least one model suitable for the analysis to be executed on the image processing pipeline (1500) from among the selectable artificial intelligence models.

[0269] Referring to the screen according to FIG. 9(b), in relation to step S1430, the user can select an image channel to be processed by the image processing pipeline (1500). The GUI (210) may display a list of pre-registered channels, and the screen may display channel names (e.g., Camera 001, Camera 002), connection sites per channel (e.g., Site 001, Site 002), and connection status (e.g., Normal) in a table format. The user can select one or more channels to be processed by the image processing pipeline (1500) through checkboxes.

[0270] When a user performs settings for resource allocation, model settings, and / or channel settings, the connection agent (1400) of the analysis server (1000) can receive these setting information.

[0271] Additionally, the connection agent (1400) of the analysis server (1000) may be implemented to automatically generate a container execution configuration based on at least one image processing pipeline (11500) obtained in step S1300, allocated resources, an artificial intelligence model, and / or configured channels. For the CPU, it may be implemented to automatically assign sequential indices based on the number of cores entered by the user, and for the GPU and memory, they may be applied as container runtime options according to specified values. Subsequently, the selected artificial intelligence model is loaded into the container, and the connected channel input stream is bound to the image processing pipeline (1500), so that an instance of the image processing pipeline (1500) of the inference pipeline (IP) can be executed.

[0272] According to one embodiment of the present invention, similar to the image processing pipeline (1500), the inference engine (1600) can also be created and built on a container basis based on user settings.

[0273] Specifically, the user can assign a GPU and a general-purpose inference model (1620) to be used by the inference engine (1600) through the GUI (210) of the client terminal (200). Similarly, a list of selectable general-purpose inference models may be displayed through the GUI, and the selectable general-purpose inference models may be models selected from a pool of models containing multiple inference models optimized for different hardware, with performance verified by considering compatibility with the image processing pipeline and hardware specifications of the server where analysis will be performed. The connection agent (1400) may be implemented to automatically generate a container execution configuration for the inference engine (1600) based on the settings entered by the user. The selected general-purpose inference model (1620) is loaded into the container and bound to the inference engine (1600), and when the container is started, the general-purpose inference model (1620) of the inference engine (1600) is executed together with the data processing logic (1610) to build an inference pipeline (IP) to perform unstructured vision task inference.

[0274] Through this process, an image processing pipeline package and an artificial intelligence model and / or a general-purpose inference model (1620) to be executed on the image processing pipeline (1500) can be combined in a form optimized for the hardware specifications of the server where the analysis is to be performed, thereby constructing an inference pipeline (IP), and by creating and managing the image processing pipeline (1500) and the inference engine (1600) as independent containers, the effect of improving the scalability and flexibility of the system can be provided.

[0275] Meanwhile, with respect to FIGS. 7 to 9, it was explained that resources for instances of the image processing pipeline (1500) and the inference engine (1600), respectively, are allocated through user input. However, this is merely an example, and resource allocation may be implemented to be distributed automatically by considering the selected package, characteristics of the model (size, complexity, number of parameters, etc.), characteristics of the data to be processed (number of channels, resolution, etc.), and / or hardware specifications of the server.

[0276] Meanwhile, with respect to FIGS. 7 to 9, it was explained that the artificial intelligence model and / or general-purpose inference model (1620) to be executed in the image processing pipeline (1500) is ultimately determined through user input. However, this is merely an example, and it may be implemented to be automatically determined based on performance considering the hardware specifications of the server where the analysis is to be performed.

[0278] Meanwhile, the independent container structure of the image processing pipeline (1500) and the inference engine (1600) improves the scalability and flexibility of the system, but at the same time, a mechanism is required to map the query based on the output of the image processing pipeline (1500) to an appropriate inference model (1620).

[0279] Hereinafter, with reference to FIGS. 10 and 11, an aspect of mapping an image processing pipeline (1500) and a general-purpose inference model (1620) according to one embodiment of the present application will be described in more detail. FIG. 11 is a diagram illustrating an aspect of constructing an inference pipeline (IP) according to one embodiment of the present application and an aspect of performing inference using the inference pipeline (IP).

[0280] FIG. 10 is a diagram illustrating an aspect of constructing an inference pipeline (IP) according to one embodiment of the present application. Specifically, FIG. 10 is a diagram illustrating an aspect of constructing an inference pipeline (IP) in which data processing logic (1610) maps an image processing pipeline (1500) and a general-purpose inference model (1620) based on a rule set identifier.

[0281] According to one embodiment of the present application, the data processing logic (1610) of the analysis server (1000) can perform an operation of mapping between an image processing pipeline (1500) and a general inference model (1620) based on a rule set identifier. When processing a video sequence, the image processing pipeline (1500) may be defined to include a rule set identifier applied in analyzing the video sequence in the output. The data processing logic (1610) can determine which general inference model (1620) should be used based on the rule set identifier included in the output of the image processing pipeline (1500).

[0282] Meanwhile, upon the construction of the inference pipeline (IP), the data processing logic (1610) may perform the operation of storing rule management information in the memory of the data processing logic (1610), which includes at least one of an identifier of an image processing pipeline (1500) included in the constructed inference pipeline (IP), a parameter for executing an image processing pipeline (1500 and / or an artificial intelligence model to be executed on the image processing pipeline (1500)), an identifier for a region of interest set to be detected through the image processing pipeline (1500), a rule set identifier corresponding to a target rule set applied per region of interest, and a rule identifier corresponding to a target rule belonging to the target rule set. Meanwhile, the rule management information may be stored and managed on a channel basis. This rule management information may be referenced during the construction or execution of the inference pipeline (IP) and utilized to match the output of the image processing pipeline (1500) with an appropriate general-purpose inference model (1620).

[0283] Specifically, the data processing logic (1610) can identify a rule set identifier included in the output of the image processing pipeline (1500). The image processing pipeline (1500) can be configured to include a rule set identifier in the output that indicates which rule set the data was processed by while performing the first processing.

[0284] At this time, the data processing logic (1610) can be implemented to map a general-purpose inference model (1620) of an inference pipeline (IP) to perform unstructured vision task inference based on the identified rule set identifiers by referencing the rule set identifiers applied by channel unit and region of interest of the rule management information.

[0285] Additionally, with reference to FIG. 11, the data processing logic (1610) can be implemented to input the input data (query and image) generated by applying a target rule into a mapped general-purpose inference model (1620). Specifically, the data processing logic (1610) can be implemented to obtain a query template associated with the rule identifier by referencing the rule identifier of the rule management information, generate input data by applying the target rule (e.g., third information included in the constraint information to be described later) to the output of the image processing pipeline (1500) and performing calculations on the variables included in the query template, and transmit the generated input data to the mapped general-purpose inference model (1620).

[0286] For example, when receiving an output containing a rule set identifier for "[Industrial Safety] PPE Wearing Violation Detect," the data processing logic (1610) may be implemented to map a universal inference model (1620) whose performance has been verified for the corresponding rule set corresponding to the rule set identifier, generate a query to check whether PPE is worn, and input it into the universal inference model (1620). Meanwhile, in the case of '[Industrial Safety] PPE Wearing Violation Detection Rule A', the data processing logic (1610) may be implemented to obtain a query template associated with Rule A, apply the query generation logic (third information) included in Rule A to the output of the image processing pipeline (1500) to compute the variables included in the query template to generate input data, and transmit the generated input data to the mapped universal inference model (1620).

[0287] Through this rule set identifier-based mapping mechanism, the system (10) of the present invention can automatically map a general-purpose inference model (1620) optimized for the characteristics of each rule set and use it for inference, while maintaining independent instance structures of the image processing pipeline (1500) and the inference engine (1600). In addition, even in an environment where multiple inference models are executed simultaneously, the effect of maintaining the stability and accuracy of the system can be provided by ensuring that each data is delivered to the correct inference model.

[0288] According to one embodiment of the present invention, an inference pipeline (IP) may be constructed as a pipeline group comprising a plurality of image processing pipelines (1500) according to a selected target rule set. Specifically, when an analysis server (1000) receives a setting command for a first target rule set, the inference pipeline (IP) may be constructed as a first pipeline group consisting of at least one of a first image processing pipeline and a second image processing pipeline. On the other hand, when an analysis server (1000) receives a setting command for a second target rule set, the inference pipeline (IP) may be constructed as a second pipeline group consisting of at least one of a second image processing pipeline and a third image processing pipeline. In other words, an inference pipeline (IP) constructed according to different rule sets may be constructed to include a common image processing pipeline (e.g., a second image processing pipeline), or may be constructed to include a unique image processing pipeline (e.g., a first image processing pipeline of the first pipeline group, a third image processing pipeline of the second pipeline group) according to the characteristics of each rule set.

[0289] Meanwhile, when an inference pipeline (IP) is established through FIGS. 4 to 11, the inference pipeline (IP) can perform analysis on image-based events.

[0290] The constructed inference pipeline (IP) may be composed of an image processing pipeline (1500) and an inference engine (1600). In the image processing pipeline (1500), a computer vision-based model that performs structured vision tasks is executed, and in the inference engine (1600), a general-purpose inference model (1620) that performs unstructured vision task inference is executed. The image processing pipeline (1500) may be configured to process a video stream input from a data source server (100) to generate a dataset containing object-based metadata and analysis data, and to transmit this to an industry domain adaptive data processing logic (1610). At this time, the dataset may be output in a structured form according to target rules generated during the construction of the inference pipeline (IP).

[0291] The data processing logic (1610) may be configured to generate a query to be input into a universal inference model (1620) by referencing a rule identifier included in a dataset received from an image processing pipeline (1500), selecting a query template to be applied to the data, and performing calculations on the variables of the query template using metadata of the dataset. Meanwhile, the universal inference model (1620) into which the query is input may be a model mapped through a rule set identifier included in the dataset.

[0292] The general-purpose inference model (1620) generates output in the form of natural language based on the generated query, and the data processing logic (1610) can be configured to convert the output of the general-purpose inference model (1620) into structured data through a parser associated with the rule identifier of the target rule. Furthermore, the data processing logic (1610) generates event information based on the analysis results through the image processing pipeline (1500) and the parsed output of the general-purpose inference model (1620), which can be provided to the user through the client terminal (200) or stored in the short-term memory of the data processing logic (1610).

[0294] Hereinafter, with reference to FIGS. 12 to 18, an aspect of event analysis according to one embodiment of the present application will be described in more detail. According to one embodiment, FIGS. 12 to 18 will specifically describe an aspect of analyzing an event using an inference pipeline constructed by the aspect of constructing an inference pipeline described in relation to FIGS. 4 to 11. Meanwhile, in describing the method of analyzing an event using an inference pipeline, some embodiments that overlap with the content previously described in relation to FIGS. 1 to 11 may be omitted. However, this is merely for convenience of explanation and should not be interpreted restrictively thereto.

[0295] FIG. 12 is a diagram illustrating an aspect of analyzing events using a constructed inference pipeline according to one embodiment of the present application.

[0296] As illustrated in FIG. 12, the inference pipeline (IP, analysis pipeline) of the present invention may largely consist of an image processing pipeline (1500), data processing logic (1610), and / or a second model (1620) as a general-purpose inference model.

[0297] The image processing pipeline (1500) can perform the operation of receiving a video sequence from the data source server (100) and performing a primary analysis. A first model can be executed on the image processing pipeline (1500), and the first model is a computer vision-based artificial intelligence model that can perform structured vision tasks based on object detection, such as object detection, segmentation, classification, object tracking, and / or pose estimation. The processing result of the first model can be output in the form of a first data set structured by target rules generated in the pipeline construction, and the first data set can be transmitted to the data processing logic (1610) of the inference engine (1600).

[0298] The data processing logic (1610) can generate a query to be input to the second model (1620) based on the first data set, and perform post-processing of the output of the second model (1620) to finally determine the event present in the image. The data processing logic (1610) may include sub-components such as an RPC Server, a Rules / Queries DB, and / or a Short-term memory. The RPC Server provides a communication interface with an external device including a client terminal (200), the Rules / Queries DB stores and manages various rules and query templates, and the Short-term memory is a sub-component that stores event information to preserve the context between events.

[0299] The second model (1620) is a Vision-Language Model (VLM) configured to generate text-based output based on the generated query. In particular, the second model (1620) performs context-sensing-based unstructured vision task inference and is capable of understanding and judging complex situations that are not predefined.

[0300] The data processing logic (1610) can generate event information by post-processing the output of the second model (1620) to determine events present in the image. Specifically, the data processing logic (1610) can generate event information by combining the result of processing through the image processing pipeline (1500) and the result of post-processing of the data processing logic (1610). The event information may include the video (or cropped image) used for analysis, an embedding or metric based on the result of processing through the image processing pipeline (1500), the type of the detected event, and / or constraint information.

[0301] The image processing pipeline (1500), data processing logic (1610), and second model (1620) included in the inference pipeline (IP) can perform analysis by being organically linked around constraint information. The constraint information is information corresponding to a target rule created by a user through the client terminal (200) described in FIGS. 4 to 11, and may include at least one of a rule set identifier, a rule identifier, a parameter of the first model, a query template, query generation information, and / or parser information. The constraint information defines the analysis operation of the entire inference pipeline (IP) and can be referenced at each processing step.

[0302] FIG. 13 is a flowchart illustrating a method for constructing an inference pipeline according to another embodiment of the present application. While the embodiment of FIG. 4 described above was a goal-oriented approach in which the analysis objective (rule set) is determined first and the system maps a suitable pipeline thereto, the embodiment of FIG. 13 represents a resource or task-oriented construction method in which an engineer first determines the structure of the pipeline by considering available resources or the characteristics of the task to be performed, and then elaborates upon it.

[0303] Referring to FIG. 13, a method for constructing an inference pipeline according to one embodiment of the present application may include the steps of: obtaining an image processing pipeline package and an inference engine for multimodal analysis (S1710); obtaining a setting command to select at least one of a plurality of image processing pipelines included in the image processing pipeline package through a client terminal (S1720); obtaining a setting command to select a target rule set that is one of at least one rule set supported by the selected at least one image processing pipeline (S1730); and constructing an inference pipeline based on the target rule set, the selected at least one image processing pipeline, and the inference engine (S1740).

[0304] In the step (S1710) of obtaining an image processing pipeline package and an inference engine for multimodal analysis, the analysis server (1000) may obtain or prepare a list of basic inference pipelines that can be selected by the user. Here, a basic inference pipeline refers to a pipeline having a structure in which an image processing pipeline and an inference engine including a general-purpose inference model and data processing logic are combined, but in which specific analysis parameters or queries (prompts) are not set. The list may include cases in which only an image processing pipeline is included, cases in which a complete basic inference pipeline combining an image processing pipeline and an inference engine is included, or cases in which these are mixed.

[0305] In the step (S1720) of obtaining a configuration command to select at least one of a plurality of image processing pipelines included in an image processing pipeline package through a client terminal, the user may select one of the basic inference pipeline lists provided through the client terminal (200). At this time, the user may select a basic inference pipeline from the basic inference pipeline list that has a structure optimized for the hardware specifications of the server to perform analysis (e.g., NVIDIA GPU, Qualcomm NPU, etc.) or for the specific task that the engineer intends to perform (e.g., highway traffic control, factory safety management, etc.). For example, the engineer may first select a lightweight basic inference pipeline that can run on a specific edge device from the basic inference pipeline list, thereby starting a system design that prioritizes hardware resource constraints.

[0306] In the step (S1730) of obtaining a setting command to select a target rule set among at least one rule set supported by at least one image processing pipeline selected through the above setting command, the analysis server (1000) may filter (disable) rule sets that are difficult to operate normally in the pipeline by considering the technical characteristics or constraints of the basic inference pipeline selected through step S1720, and enable rule sets that can operate normally in the pipeline and provide them to the user. In one embodiment, the rule sets supported by a single basic inference pipeline may include rule sets of different industrial domains. The user may select a target rule set that is suitable for the analysis purpose (e.g., detection of failure to wear a safety helmet) from the list of enabled rule sets. For example, if the user selects a 'vehicle analysis-only pipeline' in the preceding step, the analysis server (1000) can disable (filter) rule sets such as 'detection of helmet non-wearing' or 'detection of worker collapse' and enable only rule sets related to vehicle objects, such as 'detection of illegal parking', 'detection of lane departure', and 'detection of vehicle fire', and provide them through the client terminal (200). Accordingly, the user can select a target rule set that meets the purpose of analysis from the list of rule sets through the client terminal (200). In one embodiment, the basic inference pipeline may be defined by functions included in the basic inference pipeline, not limited to a specific industrial domain. For example, the basic inference pipeline may include a first basic inference pipeline that detects a bounding box (BBOX) of a specific object within a region of interest (ROI), and a second basic inference pipeline that triggers the generation of a specific query when a bounding box of a person is detected within an image. The rule sets supported by this basic inference pipeline may include rule sets from different industrial domains.In the preceding step, if the user selects the first basic inference pipeline among the first basic inference pipeline and the second basic inference pipeline, a list of rule sets of various industry domains supported by the first basic inference pipeline may be provided to the user through the client terminal (200). Accordingly, the user can select a target rule set that matches the purpose of analysis (e.g., a specific industry domain) from the list of rule sets through the client terminal (200).

[0307] In the step (S1740) of constructing an inference pipeline based on a target rule set, at least one selected image processing pipeline, and an inference engine, the analysis server (1000) can construct a final inference pipeline by refining a basic inference pipeline based on the target rule set selected through step S1730. In this process, parameters or queries defined in the target rule set may be applied to the basic inference pipeline.

[0308] Specifically, in step S1740, the analysis server (1000) may automatically apply optimized parameters (e.g., thresholds, area of ​​interest settings, etc.) defined in the target rule set to the basic inference pipeline in the form of a preset, or provide guidelines for setting parameters to induce the user to set the parameters directly by referring to them. For example, if the 'fire detection' rule set is selected as the target rule set, the reliability threshold may be preset to '0.85' to reduce false alarms and automatically applied to the basic inference pipeline. On the other hand, if the 'intrusion detection' rule set, which is sensitive to the camera installation angle, is selected as the target rule set, the analysis server (1000) may induce the user to set the parameters directly by displaying a guideline in the UI such as "If the camera is installed at a height of 3m or more, set the threshold to 0.6 or higher."

[0309] Additionally, in step S1740, when setting the query (prompt) to be input into the general-purpose inference model, the analysis server (1000) may provide the user with multiple prompt options associated with the target rule set. The user may create rules by selecting one of the provided multiple prompt options as is, modifying the selected prompt to suit the field situation, or directly creating a new prompt according to the provided writing guide through a client terminal. By injecting these finalized settings into the basic inference pipeline, an executable inference pipeline specialized for a specific industry domain is finally constructed.

[0310] FIG. 14 is a flowchart illustrating a method for analyzing events using an inference pipeline (IP) constructed according to one embodiment of the present application. Specifically, FIG. 14 is a flowchart illustrating the flow of an event analysis pattern performed by the inference pipeline (IP) of the event analysis system (10) illustrated in FIG. 12.

[0311] A method for analyzing an event according to one embodiment of the present application may include the steps of: receiving a video stream (S2100); receiving constraint information for analysis (S2200); performing a first processing of the video stream using at least one first model (S2300); generating a query to be input to a second model by performing a second processing of a first data set by referring to the constraint information (S2400); querying a second model that generates a text-type output based on the generated query (S2500); performing a third processing of the output generated through the second model by referring to the constraint information (S2600); and / or generating event information based on the result of the first processing and the result of the third processing (S2700). Each step may be performed by referring to the constraint information, thereby enabling analysis specialized for a specific industry domain according to target rules set by the user through the client terminal (200). The first model may be configured to process an image (first processing) based on parameters related to the first model included in the constraint information, and the data processing logic (1610) may be configured to generate a suitable query to be input to the second model (1620) using the query template and query generation information included in the constraint information. Additionally, the data processing logic (1610) may convert the output of the second model (1620) into a structured form using the parser information included in the constraint information.

[0312] In the step of receiving a video stream (S2100), the image processing pipeline (1500) of the inference pipeline (IP) may receive a video stream from the data source server (100). The video stream may be in any suitable form, including a real-time RTSP stream, a video file, and / or an image sequence. For example, a real-time video stream may be received from a CCTV camera when monitoring a factory work site, and video from a camera installed on a road may be received when analyzing traffic conditions.

[0313] In the step of receiving constraint information (S2200), the inference pipeline (IP) may receive constraint information for analysis. The constraint information may be information corresponding to the target rule generated through the construction aspects of the inference pipeline (IP) described above in FIGS. 4 to 11. That is, when a target rule set is selected through the client terminal (200), a channel is assigned, and changeable parameters are set to generate the target rule, the target rule is transmitted to the inference pipeline (IP) in the form of constraint information to control the execution operation of the inference pipeline.

[0314] Constraint information may include at least one of a constraint identifier, first information defining the behavior of a first model executed in an image processing pipeline (1500), second information regarding a query template, third information regarding query generation logic, and / or fourth information regarding a parser for processing the output of a second model (VLM).

[0315] Constraint identifiers may include rule set identifiers and rule identifiers. Rule set identifiers are intended to distinguish rule sets associated with a specific industry domain, and rule identifiers are intended to identify individual rules within that rule set.

[0316] The first information may be information regarding inference requirements, including parameters for configuring the output form of the image processing pipeline (1500) or information regarding the task of the image processing pipeline (1500). As a specific example, the first information may include parameters related to the object type to be detected, a confidence threshold, region of interest (ROI) information, crop parameters, padding parameters, frame drop rate, and / or frame interval.

[0317] The second information is information about a query template associated with a constraint identifier (rule identifier). The query template defines the basic structure of a query to be queried in the second model (1620), which is in the form of a Vision-Language Model (VLM), and may include variables. Here, the query template may be a query template that has been pre-verified for performance regarding a rule set corresponding to the rule set identifier of the constraint identifier, and may be a query template set during the construction process of the inference pipeline (IP). For example, it may be in the form of "In the image, there is a person at position {bbox_coordinates}. Is this person wearing a safety helmet?", where {bbox_coordinates} is a variable. According to one embodiment, the query may be in the form of a prompt.

[0318] The third information may be information that defines logic for generating a query to be input into the second model (1620) based on the first data set, which is the output of the query template and the image processing pipeline (1500). The third information specifies how to calculate and assign variables included in the query template. For example, the third information may include logic for generating a query by extracting the variable {bbox_coordinates} of a query template, such as "In the image, there is a person at position {bbox_coordinates}. Is this person wearing a safety helmet?", based on metadata included in the first data set.

[0319] The fourth information may be information defining the type of parser and / or the logic of the parser for processing the output of the second model (1620), which is in the form of a vision-language model (VLM) and is associated with a constraint identifier (rule identifier). Since the output of the vision-language model is in the form of natural language text, the output of the second model (1620) can be processed into a refined form through parsing logic to convert it into structured data.

[0320] In the step (S2300) of performing a first processing on a video stream using at least one first model, the image processing pipeline (1500) of the inference pipeline (IP) may perform the first processing on the video stream using at least one first model. The first model may be configured to perform the first processing by referring to constraint information received in step S2200 and to output a first data set as a result. Specifically, the first model may be configured to output the result of the first processing as a first data set in a structured form by referring to the first information of the constraint information.

[0321] The first model is a computer vision-based artificial intelligence model capable of performing structured vision tasks. A structured vision task refers to a visual analysis task with a predefined clear objective and may include tasks related to at least one of object detection, segmentation, classification, object tracking, and pose estimation.

[0322] The first model may be configured to operate according to parameters defined in the first information of the constraint information. For example, if the confidence threshold in the first information is set to 0.7, the first model may be configured to output only detection results with a confidence of 0.7 or higher.

[0323] The first data set generated as a result of the first processing may include a channel identifier where the image was captured, a constraint identifier, object-based metadata, analysis data through the first model, and / or an image (or cropped image) used for analysis.

[0324] Constraint identifiers indicate which rule set and by which rules the data was generated, and rule identifiers can be utilized to select appropriate query templates and parsers in subsequent steps.

[0325] Object-based metadata is raw data output by the first model and may include an object ID, object class, bounding box coordinates, confidence score, segmentation mask, keypoint coordinates and confidence, and / or a tracking identifier.

[0326] The analysis data obtained through the first model refers to industrial data for a specific domain extracted by processing object-based metadata. For example, when monitoring road conditions, the analysis data may include data such as traffic volume, average vehicle speed, and / or the average distance between pedestrians and vehicles calculated from object-based metadata. For example, when monitoring the interior of an industrial site, the analysis data may include data such as whether work clothes and safety helmets are worn by each object.

[0327] The image used for analysis may be the original image used in the first processing or an image cropped for the region of interest.

[0328] As a specific example, when a set of "worker safety equipment detection" rules is applied to a channel monitoring the interior of an industrial site, the first model can be implemented to detect people in the video, classify whether safety helmets and work clothes are worn for the detected person areas to generate a first data set, and transmit it to an inference engine (1600).

[0330] In the step (S2400) of generating a query to be input into a second model by processing a first data set by referring to constraint information, the data processing logic (1610) of the inference engine (1600) may be implemented to generate a query to be input into a second model by processing the first data set by referring to constraint information. This is a process of converting the structured analysis results of the first model into a natural language query that the second model can understand.

[0331] Query generation can be performed in two main steps. The first step is to receive a target query template to be used for analysis by referring to the second information of the constraint information. Specifically, the data processing logic (1610) can obtain a target query template associated with a constraint identifier (rule identifier) ​​included in the first data set from the Rules / Queries DB of the data processing logic (1610) by referring to the second information of the constraint information.

[0332] The second step is to generate a query by calculating the values ​​for variables included in the target query template based on the first data set by referring to the third information of the constraint information. For example, the data processing logic (1610) can generate a final query by referring to the third information of the constraint information, extracting the bbox of an object (e.g., person_001) from the first data set, and replacing the {bbox} variable existing in the target query template.

[0334] In the step (S2500) of querying a second model that generates text-form output based on the generated query, the data processing logic (1610) of the inference engine (1600) may be configured to input the generated query into the second model (1620) that generates text-form output and query it. The second model (1620) may be a multi-modal model including a Vision-Language Model (VLM) and may be configured to receive images and text simultaneously to generate a response in natural language form. Accordingly, the query may further include images, and the images may be images used for analysis included in the first dataset.

[0335] Meanwhile, as described above, the data processing logic (1610) can be implemented to transmit a query to a mapped second model (1620) by referencing a constraint identifier, in particular a rule set identifier, included in the first data set.

[0336] The second model (1620) receives a text query and an image as input, combines the visual content of the image with the meaning of the text query to understand the context, and can generate and output an answer to the query in the form of natural language.

[0338] In the step (S2600) of processing the output generated through the second model by referring to constraint information, the data processing logic (1610) of the inference engine (1600) may be implemented to process the output generated through the second model (1620) by referring to constraint information. The output of the second model (1620) may be in the form of natural language text, and a parsing process is required to convert it into structured data.

[0339] The third processing of the output of the second model (1620) is largely carried out in two steps. The first step is to receive a target parser to be used for analysis by referring to the fourth information of the constraint information. Specifically, the data processing logic (1610) can obtain a target parser associated with a constraint identifier (rule identifier) ​​included in the first data set by referring to the fourth information of the constraint information.

[0340] The second step is to parse the output generated through the second model (1620) using the received target parser. Parsing is the process of extracting key information from natural language text and converting it into a structured form. For example, if the output of the second model is "No, this person is not wearing a safety helmet.", the parser can be configured to extract the "No" keyword and generate structured data such as {"helmet_worn": False}.

[0342] In the step (S2700) of generating event information based on the results of the first processing and the third processing, the data processing logic (1610) of the inference engine (1600) may be implemented to generate event information based on the results of the first processing and the results of the third processing. As the final result of the analysis, the event information may be stored in a short-term memory within the data processing logic (1610), provided to a user through a client terminal (200), or transmitted to an external system (e.g., a command center).

[0343] Event information may include the video used for analysis, the embedding or metric associated with the result of the first processing, the type of event detected by the third processing, and / or constraint information.

[0344] The video used for analysis is the original image or cropped image used in the first and / or second processing, allowing the user to visually confirm the detected event.

[0345] The embedding or metric associated with the result of the first processing is metadata and analysis data generated by the first model, and may include, for example, object bounding box coordinates, confidence score, object tracking ID, and metrics calculated in relation to a specific domain (e.g., calculated traffic volume, average vehicle speed, and / or average distance between pedestrians and vehicles, whether work clothes and safety helmets are worn per object, etc.).

[0346] The type of event detected by the third processing is a classification of the event determined based on the parsed VLM output, and may take the form of, for example, "PPE_Violation_Failure_to_Wear_Helmet", "Traffic_Traffic_Accident", "Worker_Entering_Danger_Zone", etc.

[0347] Constraint information may include constraint identifiers (rule set ID, rule ID) indicating which rule detected the event, thereby enabling the tracking and management of events.

[0348] As a specific example, in the case of a helmet non-wearing event, event information may be generated including the video used for analysis (image / cropped image related to the helmet-non-wearing object), embeddings or metrics (person_bbox, detection_confidence, etc.), event type ("PPE_VIOLATION_NO_HELMET"), and / or constraint information (ruleset_id, rule_id, applied_parameters, etc.).

[0349] In addition, event information may be generated or stored to include a channel identifier, a unique identifier, a timestamp, an identifier for the applied query (prompt), the risk level of the event, the progress of the event, and / or the time to live (TTL).

[0350] Meanwhile, the data processing logic (1610) may be implemented to generate event information by matching the result of the first processing and the result of the third processing using a rule identifier or rule set identifier included in the first data set of the image processing pipeline (1500).

[0352] Hereinafter, with reference to FIGS. 15 to 18, various aspects of detecting exemplary events using an inference pipeline (IP) according to one embodiment of the present application will be described.

[0353] FIGS. 15 to 18 are drawings for illustrating aspects of detecting exemplary events using a constructed inference pipeline (IP) according to one embodiment of the present application.

[0354] Specifically, FIG. 15 illustrates a method for analyzing an event by capturing the start and end times of an activity of interest according to an embodiment of the present invention.

[0355] As illustrated in FIG. 15, the event analysis method of the present invention may operate by selecting two candidate images to determine whether a specific action has occurred, and by independently generating a query for each image to query an inference model (1620).

[0356] Specifically, the image processing pipeline (1500) may be configured to select a first candidate image corresponding to the start time of an action of interest and a second candidate image corresponding to the end time from a video sequence received from a data source server (100). The example in FIG. 15 shows a case where a dangerous action is detected in which a worker removes a safety hook and does not re-hook it. The first candidate image may represent an image of the state in which the act of removing the hook has started (e.g., hook removed), and the second candidate image may represent an image of the state after a certain amount of time has elapsed from the point in time corresponding to the first candidate image (e.g., hook attached or hook still removed).

[0357] The two selected candidate images can each be used to generate independent queries (Q1, Q2). A query (Q1) to inquire about the start of an activity of interest is generated based on the first candidate image, and a query (Q2) to inquire about the termination of an activity of interest or the continuation of a dangerous state is generated based on the second candidate image.

[0358] The generated query (Q1) and the first candidate image are input as the first input values ​​of the second model (1620), so that the second model (1620) generates a first response (O1) based on the first input values, and the generated query (Q2) and the second candidate image are input as the second input values ​​of the second model (1620), so that the second model (1620) generates a second response (O2) based on the second input values. In this way, by inputting the two candidate images independently rather than simultaneously into the multimodal model, the state at each point in time can be clearly determined.

[0359] The inference engine (1600) may be configured to determine that an event of interest has occurred when the first response (O1) and the second response (O2) correspond to a predetermined combination of responses. For example, in the example of FIG. 15, if the first response (O1) is about "the hook is unfastened" and the second response is about "the hook is still unfastened," the combination of these two responses may be determined as an event of interest meaning "a dangerous act of unfastening the safety hook and not fastening it again has occurred." Conversely, if the first response is "the hook is unfastened" but the second response is about "the hook is fastened," it may be determined that no dangerous event has occurred because the worker fastened the hook again.

[0360] The selection of the second candidate image can be performed in various ways. In one embodiment, the second candidate image may be heuristically determined as an image at a point in time after a predetermined amount of time has elapsed from the point in time corresponding to the first candidate image. For example, based on safety regulations or rules of thumb that the act of re-hooking the safety hook must typically be completed within a specific time (e.g., 3 seconds), an image at a specific time (3 seconds) after the point in time corresponding to the first candidate image may be selected as the second candidate image. In another embodiment, the second candidate image may be determined by analyzing the amount of change relative to the first candidate image. For example, the first model may track changes in the worker's hand position or the state of the safety hook while continuously analyzing frames, and an image at the point in time when the amount of change exceeds a threshold value may be selected as the second candidate image.

[0361] These two image-based behavior judgment methods have the advantage of effectively detecting behaviors of interest that involve temporal changes and are difficult to judge using a single image alone, while clearly determining whether dangerous behavior has occurred by independently evaluating each point in time. In particular, in situations where a state temporarily deviates from normal and must be restored, such as the removal of safety equipment, only the actual dangerous situation can be accurately detected by verifying whether the state has been restored.

[0362] Meanwhile, Fig. 15 describes the detection of a specific event. However, this is merely an example for the convenience of explanation and could be applied to detect an event of interest related to any appropriate action involving temporal change.

[0364] FIG. 16 illustrates a method for selecting an optimal query by considering the field of view information of a camera according to one embodiment of the present invention.

[0365] As illustrated in FIG. 16, the event analysis method of the present invention can operate by generating a query by selecting a query (prompt) with previously verified performance based on the field of view information of the camera that captured the video stream.

[0366] Specifically, the image processing pipeline (1500) can obtain field of view information of the camera that captured the video stream during the process of performing the first processing. The field of view information may represent the field of view of the camera. The example in FIG. 16 shows a situation in which an image captured at the first field of view is input into the image processing pipeline (1500).

[0367] In the Rules / Queries DB within the data processing logic (1610), multiple candidate queries that are pre-matched according to the camera angle of view for the same analysis purpose (target rule set) may be stored. For example, for a rule set that detects whether a worker is wearing safety equipment, a query optimized for the first angle of view (Query 1) and a query optimized for the second angle of view (Query 2) may be stored, respectively.

[0368] In the query generation step, the data processing logic (1610) may be implemented to determine, among a plurality of candidate queries based on the acquired angle of view information, the query that matches the angle of view information as the target query. In the example of FIG. 16, when angle of view information corresponding to the first angle of view is acquired, the data processing logic (1610) may be configured to select a query (Query 1) that matches the first angle of view from the Rules / Queries DB and to transmit input data based on the query (Query 1) to the second model (1620). If angle of view information corresponding to the second angle of view is acquired, the data processing logic (1610) may be configured to select a query (Query 2) that matches the second angle of view from the Rules / Queries DB and to transmit input data based on the query (Query 2) to the second model (1620).

[0369] This angle-of-view-based query selection method offers the advantage of maintaining consistently high analytical performance in response to the diversity of camera installation environments. By preparing performance-verified queries for each angle, optimized analysis can be performed regardless of on-site camera installation conditions. Furthermore, since the appropriate query is automatically selected simply by detecting angle-of-view information when applying the same rule set to various sites, user convenience is enhanced by eliminating the need to manually modify or adjust queries.

[0371] FIG. 17 illustrates a method for detecting an abnormal situation by comparing a reference image of a normal situation with a current image according to an embodiment of the present invention.

[0372] As illustrated in FIG. 17, the event analysis method of the present invention can operate by comparing the current image with a reference image for a normal situation and, if a difference occurs, selectively generating a query for the corresponding area.

[0373] Specifically, the image processing pipeline (1500) can acquire a reference image of a normal state during the process of performing the first processing. The reference image is a reference image representing the normal state of the space to be monitored, and can be captured during the initial setup of the system or generated by learning the normal state over a certain period. For example, in the case of internal monitoring of an industrial site, an image of a normal state without smoke or fire can be used as a reference image.

[0374] The first model of the image processing pipeline (1500) can be configured to compare a reference image with the current image of the video stream. This comparison can be performed through various computer vision techniques, such as pixel-level difference analysis, feature matching, or image similarity measurement.

[0375] Based on the comparison results, if the difference between the reference image and the current image is greater than a threshold, the current image may be determined to be a candidate image for an abnormal situation rather than a normal situation. For example, in a smoke detection scenario, if opaque gray or white areas that were not present in the reference image appear in the current image, it can be determined as a candidate image for an abnormal situation.

[0376] Furthermore, in the query generation step, a region of interest where a difference exists can be determined by comparing the pixels included in the reference image with the pixels included in the candidate image for the abnormal situation. The data processing logic (1610) can be implemented to receive a cropped image containing the region of interest and to generate a query based on the received cropped image.

[0377] For example, if a significant difference in pixel values ​​is detected in a specific area near the ceiling as a result of comparing a reference image and a current image in a smoke detection situation, a cropped image of that area may be generated. The data processing logic (1610) may be configured to generate the cropped image and a query based on target rules related to smoke detection (e.g., "In the highlighted region of the image, is there any smoke or abnormal substance? Answer Yes or No and describe what you see.") and transmit them to a second model (VLM). At this time, the second model (1620) may focus on the cropped area of ​​interest to determine the presence of smoke and output a response regarding the occurrence of an event.

[0378] This reference image-based anomaly detection method can offer the advantage of increasing computational efficiency by selectively performing inference only on areas where changes have occurred, and improving analysis accuracy by allowing the inference model to focus on the changed parts through cropping the region of interest.

[0380] FIG. 18 illustrates a method for identifying a situation in which a plurality of objects exist and analyzing an event based thereon according to an embodiment of the present invention.

[0381] As illustrated in FIG. 18, the event analysis method of the present invention can operate by identifying a situation in which at least two objects (preferably three or more objects) exist in a video stream, generating a cropped image containing the objects, and optionally configuring a query.

[0382] Specifically, the image processing pipeline (1610) can be configured to identify a situation in which at least two or more objects exist based on a video stream during the process of performing the first processing. The example in FIG. 18 shows a situation in which multiple workers are gathered in a specific area inside an industrial site. The first model can detect each worker individually and extract the bounding box of the individual object and the coordinate information of the bounding box.

[0383] When multiple objects are identified, the image processing pipeline (1500) may be configured in the first processing step to generate a cropped image that includes all of these objects. For example, the area indicated by the dotted line in FIG. 18 represents a cropped area that includes all of the multiple workers. This cropped image may be set as a minimum bounding region that encompasses the bounding boxes of individual objects, or may be generated including appropriate margins.

[0384] In the query generation step, the data processing logic (1610) can be implemented to generate a query based on at least one of an identifier for each object, object information, the number of objects, and a cropped image. For example, from the first data set, information that 5 workers have been detected (number of objects), an identifier of the object, location information of each worker (object information), and / or a cropped image may be extracted, and the data processing logic (1610) may be implemented to generate a query based on the target rule for detecting multiple objects (e.g., In the image, 5 people are detected at the following locations: (x1, y1, x1', y1', id1), (x2,y2, x2', y2', id2), (x3,y3, x3', y3', id3), (x4,y4, x4', y4', id4), (x5,y5, x5', y5', id5). Are these people gathered in a group? If yes, describe their interaction).

[0385] The generated query and cropped image can be input into the second model (1620), and the second model (1620) can be implemented to detect various events, such as work interruption and / or abnormal clustering behavior, based on the query and cropped image.

[0386] This multi-object-based event analysis method can effectively detect collective behaviors or interactions that are difficult to judge based on a single object alone, and offers the advantage of detecting atypical events that cannot be detected by computer vision models through contextual understanding of vision-language models.

[0388] Furthermore, the event analysis method of the present invention may include various embodiments that selectively generate queries in response to various situations.

[0389] In one embodiment, the image processing pipeline (1500) may be configured to identify, through a first processing, a situation in which at least one object is located in a region of interest based on a video stream, and to generate a cropped image including the region of interest and the object. In the query generation step, the data processing logic (1610) may be configured to generate a query based on the cropped image. For example, if a situation in which a worker has entered a region of interest designated as a danger zone is identified, a cropped image including the region and the worker is generated, and the data processing logic (1610) may be configured to generate a query that queries for the occurrence of an event of interest based on the cropped image.

[0390] In one embodiment, the image processing pipeline (1500) may be configured to obtain direction information (heading) of at least one object based on a video stream through a first processing step. In the query generation step, the data processing logic (1610) may be configured to generate different queries depending on the direction information of the obtained object. Specifically, if the direction information of the object includes first direction information, a third query corresponding to the first direction information (e.g., based on a query template with performance verified in advance for the first direction information) may be generated, and if it includes second direction information, a fourth query corresponding to the second direction information (e.g., based on a query template with performance verified in advance for the second direction information) may be generated and implemented to query the second model. For example, analysis of events of interest may be performed using different queries when a worker moves toward a danger zone and when moving in the opposite direction of a danger zone.

[0391] In one embodiment, the image processing pipeline (1500) may be configured to identify, through a first processing, a situation in which at least one object moves and then stops based on a video stream, and to generate a cropped image containing the object. In the query generation step, the data processing logic (1610) may be configured to generate a query corresponding to the stopping situation based on the cropped image. For example, if a case in which a vehicle stops while moving on a highway is identified, the data processing logic (1610) may be configured to transmit a cropped image containing the object related to the identified situation and a query to determine whether an event has occurred (e.g., Is this vehicle stopped due to an accident or mechanical failure? Are there any visible hazards?) to a second model so that the second model can be queried to determine whether an event of interest has occurred.

[0392] In one embodiment, the image processing pipeline (1500) may be configured to calculate the confidence level of a part of interest of an object present in a video stream using a pose estimation technique through a first processing. Furthermore, the image processing pipeline (1500) may be configured to generate a cropped image of the part of interest of the object when the confidence level is greater than a predetermined threshold. In the query generation step, the data processing logic (1610) may be implemented to receive a cropped image of the part of interest and generate a query related to the wearing of protective gear on the part when the calculated confidence level is greater than a predetermined threshold. For example, if the keypoint confidence level for the worker's head is detected to be higher than a predetermined confidence threshold, the data processing logic (1610) may generate a query related to whether a safety helmet is worn (e.g., 'Is this person wearing a safety helmet on their head?') along with a cropped image of the head, so that the second model can accurately determine whether a safety helmet is worn. On the other hand, if the calculated confidence is smaller than a predetermined threshold (i.e., the region of interest is not clearly visible), query generation may be omitted to prevent inaccurate judgments.

[0394] Hereinafter, a method for providing event information according to an embodiment of the present application will be described in more detail with reference to FIGS. 19 to 25. Specifically, with reference to FIGS. 19 to 25, an event progression stage (event occurrence, event action in progress, event termination in progress) according to an embodiment of the present application will be analyzed, and an aspect of providing information regarding the event progression stage will be described in detail. Meanwhile, in describing the method for providing event information, some embodiments that overlap with the content previously described in relation to FIGS. 1 to 18 may be omitted. However, this is merely for convenience of explanation and should not be interpreted restrictively thereto, and the method for constructing an inference pipeline and the method for analyzing events using the inference pipeline described in relation to FIGS. 1 to 18 may be applied in the same way to this embodiment.

[0395] FIG. 19 is a flowchart illustrating a method for providing information about an event according to one embodiment of the present application. Specifically, as illustrated in FIG. 19, the method for providing event information of the present invention operates by adaptively switching rules and queries according to the progress stage of the event, systematically tracking the occurrence of the event, actions regarding the event, and the termination of the event, and providing information to the user.

[0396] A method for providing event information according to one embodiment of the present application may include: a step of obtaining first image-based information based on a first rule and generating a first query regarding whether an event has occurred based on the first image-based information (S3100); a step of transmitting the first query to a vision-language model so that the vision-language model determines whether an event has occurred (S3200); a step of providing information that an event has occurred after determining that an event has occurred based on the output of the vision-language model (S3300); a step of obtaining second image-based information based on a second rule and generating a second query regarding whether an event has ended based on the second image-based information after determining that an event has occurred (S3400); a step of transmitting the second query to a vision-language model so that the vision-language model determines whether an event has ended (S3500); and / or a step of providing information that an event has ended after determining that an event has ended based on the output of the vision-language model (S3600).

[0397] The key feature of the present embodiment is that the first rule and first query for determining whether an event occurs and the second rule and second query for determining whether an event ends (or the third rule and third query for determining whether an action is taken on the event) are different from each other. This is because the situations and conditions to be determined at the start and end points of the event are different, and it is to perform an analysis optimized for each step.

[0398] In step S3100, the data processing logic (1610) of the inference pipeline (IP) may be configured to acquire first image-based information based on a first rule and to generate a first query regarding whether an event has occurred based on the first image-based information. The first rule defines rules and parameters for detecting whether a specific event has occurred in a normal state. Specifically, the step of generating the first query (S3100) may further include the step of acquiring first image-based information from a first media processing pipeline (which may also be referred to as a first image processing pipeline) corresponding to the first rule. For example, in a traffic accident detection scenario, the first rule defines a region of interest, a confidence threshold, etc., to identify a vehicle collision or an abnormal vehicle stop situation in the image, and in the first media processing pipeline, an object detection model (e.g., YOLO) may be executed to compute vehicle objects and analysis data (e.g., vehicle speed). The first query can be generated in a form that allows the vision-language model (1620) to determine whether an event has occurred, such as "Is there a vehicle collision or accident visible in the image?"

[0399] In step S3200, the data processing logic (1610) may transmit the first query (including an image) generated through step S3100 to the vision-language model so that the vision-language model determines whether an event has occurred. The vision-language model may be configured to output a response regarding whether an event has occurred based on the first query. The data processing logic (1610) may be configured to finally determine whether an event has occurred based on the response of the vision-language model.

[0400] In step S3300, the data processing logic (1610) can be implemented to provide information that an event has occurred after determining that an event has occurred based on the output of the vision-language model. This information (information on the event progression stage described later) can be displayed on a user interface (UI) via the client terminal (200) or transmitted to an external server (e.g., Command Center) to enable an immediate response.

[0401] In one embodiment, the first query may include multiple subqueries rather than a single query. In this case, the data processing logic (1610) may be implemented to determine that an event has occurred when it is detected that an event has occurred for at least one of the multiple subqueries. For example, in the case of traffic accident detection, the first query may include subqueries such as (1) "Are there any vehicles in an abnormal position or orientation?", (2) "Is there visible vehicle damage or collision?", and (3) "Are there any stopped vehicles blocking traffic lanes?". In this case, if a response indicating that a traffic accident has been detected is output through a vision-language model for any of these, the data processing logic (1610) may be implemented to determine that a traffic accident event has occurred.

[0402] After it is determined that an event has occurred, the inference pipeline (IP) may automatically switch to a mode for detecting the end of the event. This switch may be automatically performed by a related sub-rule included in the target rule created when the aforementioned inference pipeline (IP) is built. Through this switch, the first rule for detecting the occurrence of an event may be changed to a second rule for detecting the end of an event (or a third rule for detecting whether an action regarding the event described below is being performed).

[0403] In step S3400, the data processing logic (1610) of the inference pipeline (IP) may be configured to acquire second image-based information based on a second rule and to generate a second query regarding whether an event has ended based on the second image-based information. The second rule defines rules and parameters for determining whether an event that has occurred has ended. Specifically, the step of generating the second query (S3300) may further include the step of acquiring second image-based information from a second media processing pipeline (which may be referred to as a second image processing pipeline) corresponding to the second rule. Meanwhile, in the second media processing pipeline, at least one model different from the model executed in the first media processing pipeline may be executed. In this case, the meaning of the model being different may encompass the meaning that the setting value for at least one parameter included in the model is different. For example, the second query may be generated in a form that allows the vision-language model to determine whether an event has ended, such as "Have the accident vehicles been cleared from the road? Is traffic flowing normally?"

[0404] In step S3500, the data processing logic (1610) may transmit the second query (including an image) generated through step S3400 to the vision-language model so that the vision-language model determines whether the event has ended. The vision-language model may be configured to output a response regarding whether the event has ended based on the second query. The data processing logic (1610) may be configured to finally determine whether the event has ended based on the response of the vision-language model.

[0405] In step S3600, the data processing logic (1610) of the inference pipeline (IP) can be implemented to provide information that the event has ended after determining that the event has ended based on the output of the vision-language model.

[0406] In one embodiment, information that an event has ended may not be provided immediately but after a certain period of time. Specifically, the data processing logic (1610) may be implemented to obtain an event end time determined to have ended based on the output of the vision-language model, and to output information that the event has ended at a time when a predetermined period of time has elapsed from the event end time. This is to prevent a situation where the event appears to have ended temporarily but then recurs. Meanwhile, if it is determined that the event has occurred again for a predetermined period, the data processing logic (1610) may be implemented to maintain information that the event has occurred without outputting information that the event has ended.

[0407] In another embodiment, whether an event has ended may be determined through multiple verifications rather than a single determination. Specifically, the data processing logic (1610) may be implemented to determine that the event has ended after it has been detected that the event has ended for a predetermined number of times (e.g., n=3), based on the output of the second query and the vision-language model. This prevents temporary misjudgment and increases the reliability of the event end determination.

[0409] FIG. 20 is a specific flowchart of a method for providing information about an event according to one embodiment of the present application. Specifically, FIG. 20 is a flowchart illustrating an analysis aspect of an event progression stage including a step of determining whether to take action after an event occurs according to one embodiment of the present invention.

[0410] After determining that an event has occurred according to one embodiment of the present application, the step of generating a second query regarding whether the event has ended (S3400) may further include: a step of generating a third query to determine whether an action is being performed in accordance with the event occurrence based on a third rule after determining that the event has occurred (S3410); a step of transmitting the third query to a vision-language model so that it is determined whether an action is being performed in accordance with the event occurrence in the vision-language model (S3420); and / or a step of providing information that an action is being performed in accordance with the event occurrence after determining from the output of the vision-language model that an action is being performed in accordance with the event occurrence (S3430).

[0411] In step S3410, the data processing logic (1610) of the inference pipeline (IP) may be implemented to generate a third query to determine whether action is being taken in response to the event based on a third rule after determining that an event has occurred. The third rule defines rules and parameters for determining whether appropriate response measures are being taken for the event that occurred. For example, in a traffic accident detection scenario, the third rule defines conditions for checking whether an ambulance, police car, or tow truck has arrived, or the activities of response personnel at the accident scene, and based on the third rule, the media processing pipeline may be configured to detect objects related to an ambulance, police car, or tow truck in the video. The third query may be generated in a form that allows the vision-language model to determine whether action is being taken for the event, such as "Are there any emergency responders or vehicles present at the accident scene? Is assistance being provided?"

[0412] In step S3420, the data processing logic (S1610) may transmit the third query generated in step S3410 to the vision-language model so that it is determined whether an action is being performed in response to an event occurrence in the vision-language model, thereby enabling the vision-language model to determine whether an action is being performed for the event. The vision-language model may be configured to output a response regarding whether an action is being performed for the event based on the third query. The data processing logic (1610) may be configured to finally determine whether an action is being performed for the event based on the response of the vision-language model.

[0413] According to one embodiment, a first query to confirm the occurrence of the aforementioned event may be transmitted to a vision-language model at a predetermined first time period. On the other hand, a second query to confirm the termination of the event and / or a third query to confirm whether an action regarding the event is being performed may be transmitted to the vision-language model at a second time period different from the first time period. In this case, the first time period may be characterized as being shorter than the second time period. This is because confirming the occurrence of an event is relatively more urgent than confirming an action or termination. By setting the first time period to be short, the occurrence of a new event can be detected quickly, thereby providing the effect of shortening the initial response time and enabling rapid action.

[0414] In step S3430, the data processing logic (1610) can be implemented to provide information that an action is being performed in response to an event after determining from the output of the vision-language model that an action is being performed in response to an event. This can provide the effect of enabling the user to clearly understand the progress stage of the event.

[0415] FIG. 21 is a diagram illustrating an aspect of event analysis according to the progress stage of an event according to an embodiment of the present application. Specifically, FIG. 21(a) is a diagram illustrating an aspect in which a process for determining whether an action for an event is being performed and a process for determining whether the event has ended are executed sequentially, and FIG. 21(b) is a diagram illustrating an aspect in which a process for determining whether an action for an event is being performed and a process for determining whether the event has ended are executed in parallel.

[0416] As illustrated in FIG. 21(a), in one embodiment, a second query (Q2) for checking whether an event has ended may be generated after obtaining information that an action is being performed in response to a third query (Q3) for checking whether an action regarding the event is being performed. This is a sequential approach in which the action regarding the event is first confirmed and then the termination is determined.

[0417] Specifically, the inference engine (1600) may be implemented to confirm the occurrence of an event through a query of the first query (Q1), and then to determine whether action regarding the event is being performed through a query of the third query (Q3). Furthermore, the inference engine (1600) may be implemented to generate a second query (Q2) after it is determined that action regarding the event is being performed, and to check whether the event has ended through a vision-language model. For example, in the case of a high-severity traffic accident, since the removal of the accident vehicle and traffic normalization (i.e., the end of the event) are possible only after an ambulance and police arrive and perform action, it may be reasonable to perform the action confirmation first.

[0418] As illustrated in FIG. 21(b), in another embodiment, a second query (Q2) for checking whether an event has ended and a third query (Q3) for checking whether an event has been taken can be generated in parallel and transmitted to a vision-language model. Specifically, the inference engine (1600) can be implemented such that the second query (Q2) is generated at least once for a first time interval included between the time when the event is determined to have occurred and the time when the event is determined to have ended, and transmitted to the vision-language model. On the other hand, the inference engine (1600) can be implemented such that the third query (Q3) is generated at least once for a second time interval included between the time when the event is determined to have occurred and the time when the event is determined to have ended, and transmitted to the vision-language model. In this case, at least a portion of the time intervals of the first time interval and the second time interval may overlap. That is, the process of checking whether an event has ended and the process of checking whether an event has been taken can be performed in parallel. For example, in the case of a minor traffic accident, the accident vehicle can be quickly removed and the process terminated without official emergency measures; therefore, it may be reasonable to perform the event termination confirmation via the second query (Q2) and the event action confirmation via the third query (Q3) in parallel to simultaneously monitor which condition is satisfied first.

[0420] FIG. 22 is a diagram illustrating an aspect of event analysis according to the progress stage of an event according to an embodiment of the present application. Specifically, FIG. 22 illustrates a detailed aspect in which a judgment that an action for an event is being performed in a parallel execution method according to FIG. 21(b) according to an embodiment of the present invention is made before a judgment that the event has ended.

[0421] According to one embodiment, in a situation where the process of confirming the termination of an event and confirming whether an action regarding the event is being performed is performed in parallel, if it is first determined through the third query (Q3) that an action regarding the occurrence of an event is being performed, the inference engine (1600) can adaptively switch the analysis mode.

[0422] Referring to FIG. 22(a), a query transition is shown after it is first determined that an action regarding an event is being performed. Specifically, after the occurrence of an event is confirmed through the first query (Q1), a second query (Q2) to check whether the event has ended and a third query (Q3) to check whether an action regarding the event has been performed may be queried in parallel. In this process, if it is first determined that an action regarding the event is being performed based on the response to the third query (Q3), the inference engine (1600) may be implemented to regenerate the second query (Q2') based on information that an action regarding the event is being performed and a new image.

[0423] Specifically, the inference engine (1600) can generate a second query by obtaining third image-based information based on a second rule in response to a determination that an action is being performed in accordance with the occurrence of an event, and using the third image-based information as one input and the information that an action is being performed in accordance with the occurrence of an event as another input. For example, if information regarding an action that an ambulance and police have arrived at the scene is confirmed in a traffic accident scenario, the inference engine (1600) can be implemented to generate a query (Q2') to check whether the event has ended, reflecting the situation of the action, such as "Given that emergency responders are now present, have the accident vehicles been cleared and is traffic resuming?" based on the action information.

[0424] According to one embodiment, the inference engine (1600) may be implemented to stop the query to the non-language model of the third query (Q3) and maintain the query to the non-language model of the second query (Q2) in response to the determination that an action is being performed in response to the occurrence of an event. Since it is confirmed that an action is being performed for the event, there is no longer a need to repeatedly check whether an action has been performed, and the focus is now focused on confirming the end of the event. Referring to FIG. 22(b), while the query through the second query (Q2) continues to be performed, if the action regarding the event is confirmed by the query through the third query (Q3), the query through the third query (Q3) is stopped, while the query through the second query (Q2 or Q2') is maintained to finally confirm the end of the event.

[0425] This adaptive query switching method can enable more accurate and context-appropriate event tracking by dynamically adjusting analysis strategies based on the actual progress of events.

[0427] FIG. 23 is a diagram illustrating an aspect of event analysis according to the progress stage of an event according to an embodiment of the present application. Specifically, FIG. 23 illustrates a detailed aspect in which the determination of the termination of an event in the parallel execution method according to FIG. 21(b) according to an embodiment of the present invention is made before the confirmation of the action regarding the event.

[0428] According to one embodiment, in a situation where the process of confirming the termination of an event and confirming whether an action regarding the event is being performed is performed in parallel, if the termination of the event is confirmed first through the second query (Q2), the inference engine (1600) can adaptively switch the analysis mode.

[0429] Specifically, the inference engine (1600) may be implemented to stop querying the non-language model of the third query (Q3) and / or the second query (Q2) in response to the determination that the event has ended. This is because, since the event has ended, there is no longer a need to continue verifying whether action has been taken regarding the event or whether the event has ended. At this time, the inference engine (1600) may be implemented to regenerate the first query (Q1) regarding whether the event has occurred and to query the non-language model with the regenerated first query (Q1). This means that the event analysis system returns to a normal state monitoring mode and prepares to detect the occurrence of a new event. FIG. 23 shows that a re-query of the first query (Q1) begins after the 'determination of event end,' thereby enabling the system to cyclically monitor the event. This processing method allows for an effective response to situations where the event can naturally end without verification of action. For example, in the case of a minor traffic accident where the parties quickly move their vehicles and traffic returns to normal, the event may be terminated without any action being taken regarding the traffic accident. In such cases, it may be more efficient to immediately recognize the termination and switch the analysis mode to detect new events that may occur next, rather than continuing to verify actions.

[0431] FIG. 24 is a diagram illustrating an aspect of event analysis according to the progress stage of an event according to an embodiment of the present application. Specifically, FIG. 24 is a diagram illustrating an analysis aspect in which event termination information is provided after a certain period of time has elapsed following an event termination determination according to an embodiment of the present invention, and the event recurs before the certain period of time has elapsed.

[0432] As previously described in FIG. 19, according to one embodiment, information that an event has ended may be implemented so that it is not provided immediately but after a certain period of time. FIG. 24 illustrates specific scenarios of this method.

[0433] FIG. 24(a) illustrates a situation in which the termination of an event is confirmed. As described above, the inference engine (1600) can be implemented to display information about the termination of an event to the user after a predetermined time (t) has elapsed from the point in time when it is determined that the event has ended through a query via the second query (Q2).

[0434] According to one embodiment, during a delay period (t), the inference pipeline (IP) detects whether an event has recurred, and if it is confirmed that there has been no recurrence, it may be implemented to provide information about the termination of the event.

[0435] FIG. 24(b) illustrates the case where an event reoccurs during a delay period (t). While waiting for a predetermined time (t) to elapse after the event has ended through a second query (Q2), the reoccurrence of the event may be detected. The reoccurrence of the event may be detected by configuring the first query (Q1) to be queried during the delay period (t). In this case, the inference pipeline (IP) may be implemented to maintain information that the event has occurred, rather than outputting information that the event has ended.

[0436] FIG. 24(c) illustrates a case where additional verification is performed upon the recurrence of an event. If it is determined that an event has occurred again between the time the event ended and the time after a predetermined period has elapsed, the event analysis system may respond in one of the following two ways.

[0437] First, the inference engine (1600) may be implemented to re-verify whether the event has ended by sending the second query (Q2) back to the non-language model for a predetermined time (t). Second, the inference engine (1600) may be implemented to query the non-language model by generating a new fourth query regarding whether the event has restarted.

[0438] This mechanism for delayed provision of event termination information and recurrence processing can prevent misjudgment caused by temporary changes in circumstances and enhance system reliability by providing event termination information only after it is confirmed that the event has actually been completely terminated.

[0440] FIG. 25 is a drawing for explaining an aspect of event information provided according to the progress stage of an event according to an embodiment of the present application. Specifically, FIG. 25 is a drawing for explaining an aspect of providing different state information to a user according to the progress stage of an event according to an embodiment of the present invention.

[0441] As illustrated in FIG. 25, the event information providing method of the present invention provides status information classified according to the progress stage of the event, thereby enabling the user to clearly understand the current situation.

[0442] Figure 25(a) shows the provision of information at each stage of progress in a sequential execution method.

[0443] Specifically, the inference engine (1600) may be implemented to provide information indicating a 'normal situation' during a first progress period that includes at least a portion of the time between when the first query (Q1) is generated and when it is determined that an event has occurred. This means that the event analysis system is monitoring whether an event has occurred, but no anomalies have been detected yet.

[0444] Furthermore, the inference engine (1600) may be implemented to provide information indicating that 'an event has occurred but no action regarding the event has been taken' during a second progress interval that includes at least a portion between the time when the third query (Q3) is generated (or the time when it is determined that an event has occurred) and the time when it is determined that an action regarding the event is being performed. This means that an event has been detected but no response action has been started yet.

[0445] Furthermore, the inference engine (1600) may be implemented to provide information indicating that 'an action regarding the event is being performed' during a third progress interval that includes at least a portion between the time when the second query (Q2) is generated (or the time when it is determined that an action regarding the event is being performed) and the time when it is determined that the event has ended. This means that a response action regarding the event is in progress and the situation is in the process of being resolved.

[0446] Furthermore, the inference engine (1600) may be implemented to provide information indicating that the event is in a 'event termination state' during a fourth progress period that includes at least a portion of the time after the point in time when the event is determined to be terminated. This means that the event has been completely resolved and has returned to a normal state.

[0447] FIG. 25(b) shows the provision of information at each stage of progress in a parallel execution method. Similar to the sequential method in FIG. 25(a), it provides four stages of progress information, but differs in that the query to check for event termination via the second query (Q2) and the query to check whether action regarding the event is being performed via the third query (Q3) are executed in a temporally overlapping manner. In the case of a parallel execution method, appropriate status information is provided at each stage of progress, so that depending on the actual progress, 'normal situation', 'event has occurred but action regarding the event has not yet been performed', 'action regarding the event is being performed', and 'event termination situation' can be implemented to be displayed.

[0448] A method for providing event information according to one embodiment of the present application may include additional embodiments for systematically managing event information and adaptively generating queries according to progress status.

[0449] According to one embodiment, the data processing logic (1610) of the analysis server (1000) may be implemented to store event information in a short-term memory included in the data processing logic (1610) after determining that an event has occurred. As described above in relation to FIG. 10, the event information may be managed on a channel basis and may include information regarding at least one of an identifier for the channel that captured the video used to analyze the event, a unique identifier (UUID), an event type, an identifier for a query input into a vision-language model, a time to live (TTL) for the event information, the risk level of the event, and the progress status of the event. Here, the progress status of the event may be classified into states such as the aforementioned 'event occurrence', 'before action regarding the event is performed', 'while action regarding the event is being performed', and 'event termination'.

[0450] According to one embodiment, the data processing logic (1610) may be implemented to dynamically change the rules to be applied to the analysis by referring to the progress status of the event of the event information stored in short-term memory. Specifically, the data processing logic (1610) may decide to change a first rule for checking whether an event has occurred to a second rule for checking whether the event has ended (or a third rule for checking whether an action regarding the event is being performed) by referring to the progress status of the event of the event information (e.g., the event has occurred). Additionally, the data processing logic (1610) may be implemented to generate a second query (or a third query) whose performance has been verified in advance for the changed second rule (or third rule). That is, the prompt applied changes depending on the progress status of the event, and the prompt applied according to the progress stage of the detected situation may be adjusted to a previously verified prompt or a prompt directly entered by the user. For example, when the progress status of a traffic accident event changes to 'event occurrence', the data processing logic (1610) can be implemented to detect this, switch the existing first rule (accident occurrence detection rule) to a second rule (accident termination detection rule), and generate a second query with verified performance for the second rule.

[0451] In one embodiment, the second query and / or third query may be generated to have a chain prompt form with the first query. This means that the progress stage of an event is detected and updated in the form of a chain prompt. The chain prompt method allows the results of the previous step's query to be utilized in generating the next step's query, thereby enabling tracking of each progress stage while maintaining the context and continuity of the event.

[0452] In one embodiment, the data processing logic (1610) may be implemented to update at least one of the identifier, risk level, retention time, and progress for the query included in the event information after determining that the event has ended. This allows the entire lifecycle of the event to be recorded and subsequently utilized for analysis or statistical purposes.

[0453] In one embodiment, the event analysis system can utilize short-term memory (STM) to preserve and utilize the context between events. Specifically, the data processing logic (1610) can be implemented to retrieve past events from short-term memory from a channel where a new or updated event has occurred and utilize them for the analysis of the event. Events stored in short-term memory are managed based on the risk level and Time to Live (TTL) of the event information, and can be managed to delete detected events from short-term memory if they are not updated for a certain period. In this way, past events that are determined not to affect the new situation are filtered out, and event information can be managed so that only relevant past events are utilized for the analysis of new events.

[0454] Meanwhile, the data processing logic (1610) may be implemented to generate a query by utilizing information about past events in short-term memory when a new event is detected in a specific channel. Through this, the event analysis system according to one embodiment of the present application can provide the effect of detecting events more accurately and integrally by considering the context of past events, rather than simply judging individual events independently.

[0455] The features, structures, effects, etc. described in the embodiments above are included in at least one embodiment of the present invention and are not necessarily limited to only one embodiment. Furthermore, the features, structures, effects, etc. exemplified in each embodiment may be combined or modified and implemented in other embodiments by a person skilled in the art to which the embodiments belong. Accordingly, details regarding such combinations and modifications should be interpreted as being included within the scope of the present invention.

[0456] Furthermore, although the embodiments have been described above, this is merely illustrative and does not limit the invention. Those skilled in the art will understand that various modifications and applications not exemplified above are possible within the scope of the essential characteristics of the embodiments. In other words, each component specifically shown in the embodiments may be modified and implemented. Differences related to such modifications and applications should be interpreted as being included within the scope of the invention as defined in the appended claims. Explanation of the symbols

[0458] 1000: Analysis Server

Claims

Claim 1 A method for constructing an inference pipeline for multimodal analysis by industrial domain using an electronic device, comprising the step of acquiring an image processing pipeline package and an inference engine for multimodal analysis—wherein the inference engine comprises an industrial domain agnostic general inference model including a Vision-Language Model and industrial domain adaptive data processing logic for generating a query to be input to the general inference model—; A step of obtaining, through a client terminal, a setting command for a target rule set related to a specific industrial domain among a plurality of rule sets compatible with the inference engine—each rule set is for unstructured vision task inference using the general-purpose inference model for the industrial domain and defines inference requirements for a structured vision task including at least one of object detection, segmentation, classification, object tracking, and pose estimation to be performed on an image processing pipeline—; and a step of obtaining at least one image processing pipeline that satisfies the inference requirements of the target rule set from an image processing pipeline package based on the setting command for the target rule set;A method for constructing an inference pipeline, comprising the step of constructing the inference pipeline based on at least one image processing pipeline and the inference engine, wherein the industry domain adaptive data processing logic is configured to map the general inference model to perform the unstructured vision task inference by referencing a rule set identifier to be included in the output of the image processing pipeline, wherein the industry domain adaptive data processing logic of the constructed inference pipeline is configured to generate a query to be input to the general inference model performing the unstructured vision task based on the output of the structured vision task inference of the at least one image processing pipeline that reflects the inference requirements of the target rule set related to the specific industry domain. Claim 2 A method for constructing an inference pipeline according to claim 1, wherein the rule set is generated for each industry domain, and the inference requirements of each rule set include types of structured vision inference tasks corresponding to any one of a plurality of predefined unstructured vision inference tasks within any one industry domain. Claim 3 A method for constructing an inference pipeline according to claim 1, wherein each set of rules comprises rule items for generating input data for the general-purpose inference model from the output of an image processing pipeline included in the image processing pipeline package according to the industry domain, and parameters changeable by the user. Claim 4 A method for constructing an inference pipeline according to claim 3, wherein the step of obtaining a setting command for the target rule set further comprises: providing the compatible rule sets through the client terminal; receiving a first input through the client terminal for selecting the target rule set among the compatible rules; receiving a second input through the client terminal for selecting a specific channel to which the target rule set is to be applied; and receiving a third input through the client terminal for a setting value of the changeable parameter. Claim 5 A method for constructing an inference pipeline according to claim 4, wherein the step of obtaining a setting command for the target rule set further comprises the step of receiving a target rule for generating input data of the inference engine generated by applying a setting value for the changeable parameter to a rule item belonging to the target rule set for the specific channel based on the first to third inputs. Claim 6 A method for constructing an inference pipeline according to claim 3, wherein the parameters changeable by the user include site adaptive parameters, wherein the site adaptive parameters include a first parameter related to at least one of a crop, padding, confidence threshold, region of interest, and framedrop rate, frame interval, and object type to be detected for analysis through the image processing pipeline, and a second parameter related to a query to be input to the general-purpose inference model. Claim 7 A method for constructing an inference pipeline according to claim 5, wherein the target rule comprises at least one of: a rule set identifier corresponding to the target rule set; a rule identifier corresponding to the target rule; first information regarding a parameter for configuring the output form of the image processing pipeline or a task of the image processing pipeline; second information regarding a query template associated with the rule identifier; third information for generating a query as input data to be input to the general inference model based on the query template and the output of the image processing pipeline; and fourth information regarding logic associated with the rule identifier for processing the output of the general inference model. Claim 8 A method for constructing an inference pipeline according to claim 5, wherein the step of obtaining a setting command for the target rule set comprises: obtaining first rule set information and first rule information that are pre-assigned to the specific channel; receiving a setting command for a candidate rule set and a candidate rule based on the first to third inputs; and further comprising the step of restricting the reception of the candidate rule as the target rule suitable for analysis regarding the specific channel when the first rule set information and the candidate rule set are the same set, and receiving the candidate rule as the target rule suitable for analysis regarding the specific channel when the first rule set information and the candidate rule set are different sets. Claim 9 A method for constructing an inference pipeline according to claim 5, wherein the step of obtaining a setting command for the target rule set further comprises: obtaining second rule set information and second rule information that are pre-assigned to a specific channel and a different channel; receiving a setting command for a candidate rule set and a candidate rule based on the first to third inputs; and receiving the candidate rule that is different from the second rule information as the target rule suitable for analysis regarding the specific channel, even if the second rule set information and the candidate rule set are the same rule set. Claim 10 A method for constructing an inference pipeline according to claim 5, wherein the step of constructing the inference pipeline further comprises the step of constructing the inference pipeline by configuring the image processing pipeline to perform an analysis corresponding to an inference requirement corresponding to the target rule for a structured vision task for vision task inference using the general-purpose inference model, based on the at least one image processing pipeline and the target rule, by applying the target rule to the specific channel. Claim 11 A method for constructing an inference pipeline according to claim 5, wherein the step of constructing the inference pipeline further comprises: a step of generating structured data that defines input data of the inference engine in response to inputting a setting value for the changeable parameter through the third input; and a step of constructing the inference pipeline by configuring the image processing pipeline to generate input data of the inference engine for the specific channel using the structured data. Claim 12 In claim 5, the method for constructing the inference pipeline further comprises the step of storing rule management information including, through the industry domain adaptive data processing logic, at least one of an identifier of the image processing pipeline included in the inference pipeline, a parameter for executing the image processing pipeline, an identifier for a region of interest set to be detected through the image processing pipeline, a rule set identifier corresponding to the target rule set applied per region of interest, and a rule identifier corresponding to a target rule belonging to the target rule set, wherein the rule management information is stored on a channel basis. Claim 13 A method for constructing an inference pipeline according to claim 12, wherein the industry domain adaptive data processing logic of the inference pipeline is configured to input the input data generated by applying the target rule to the mapped general-purpose inference model by referring to the rule set identifier of the rule management information. Claim 14 A method for constructing an inference pipeline according to claim 13, wherein the industrial domain adaptive data processing logic of the inference pipeline is configured to obtain a query template associated with the rule identifier by referencing the rule identifier of the rule management information, and to generate the input data by performing calculations on variables included in the query template using the output of the image processing pipeline. Claim 15 A method for constructing an inference pipeline according to claim 1, wherein the step of constructing the inference pipeline further comprises: a step of allocating resources for the image processing pipeline; a step of setting an artificial intelligence model to be executed on the image processing pipeline; a step of setting a channel to receive images to be analyzed on the inference pipeline; and a step of constructing the inference pipeline by creating a container of the inference pipeline based on the allocated resources, the set artificial intelligence model, and the set channel. Claim 16 A method for constructing an inference pipeline according to claim 15, wherein the artificial intelligence model is selected from a model pool comprising multiple models optimized for different hardware, and at least one artificial intelligence model whose performance has been verified with respect to the hardware specification information of the server to be analyzed is selected, and the image processing pipeline package is selected from a pipeline pool comprising multiple image processing pipelines optimized for different hardware, and at least one image processing pipeline whose performance has been verified with respect to the hardware specification information of the server to be analyzed is selected. Claim 17 A method for constructing an inference pipeline according to claim 15, wherein the general-purpose inference model of the inference engine is selected from a model pool comprising a plurality of inference models optimized for different hardware, and is selected as at least one general-purpose inference model with verified performance, taking into account compatibility with the image processing pipeline and hardware specification information of the server on which analysis is to be performed. Claim 18 A method for constructing an inference pipeline according to claim 1, wherein the step of obtaining a setting command for the target rule set further comprises the step of receiving a rule setting command for at least one of the first target rule set and the second target rule set, wherein when the setting command for the first target rule set is received, the inference pipeline is constructed such that a first pipeline group composed of at least one of the first image processing pipeline and the second image processing pipeline is constructed as the at least one image processing pipeline, and when the setting command for the second target rule set is received, the inference pipeline is constructed such that a second pipeline group composed of at least one of the second image processing pipeline and the third image processing pipeline is constructed as the at least one image processing pipeline. Claim 19 A method for constructing an inference pipeline according to claim 1, characterized in that a computer vision-based model that performs structured vision task inference is executed on at least one image processing pipeline. Claim 20 A computer-readable recording medium having a program for executing the method according to claim 1 on a computer.

Images