Method for extracting video context from external descriptions

Abstract

To provide a method and system for scaling an AI solution in a new industrial environment such as new assembly line or new factory.SOLUTION: A method includes: executing a zero shot image labeler on a video to generate a plurality of labels corresponding to images of the video; calculating an image embedding vector for each of the labels of the images; generating context labels to replace each of the labels corresponding to each of the images based on context labels determined for current and previous images in time by referencing a context database with the image embedding vector to determine the context labels from an inspected event graph, nodes of the event graph being indicative of events that can happen during a duration of the video; and replacing each of the plurality of labels with the generated context labels.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] This disclosure is directed in general towards video systems, and more specifically, towards systems and methods for extracting video context from external descriptions. [Background technology]

[0002] Many industrial applications require observing activities that involve the interaction between workers and industrial assets or tools, and deriving inferences from these observations. For example, it may be necessary to observe a worker performing a repetitive task and calculate the time it takes the worker to complete each cycle of the task, as well as the variation of such times across multiple cycles of the same task. This is called cycle time calculation.

[0003] In another example, there is a need to observe whether workers are adhering to or violating standard operating procedures (SOPs) for industrial activities. For instance, an SOP can specify strictly defined procedures for assembling parts, and if a violation occurs, it must be reported to the line supervisor or the appropriate authorities.

[0004] A more common example is observing an industrial environment and drawing conclusions about the occurrence of a certain event from a combination of asset and worker conditions. Traditionally, this observation and reasoning process has been carried out by line supervisors themselves. However, this is inefficient because supervisors cannot be present at all work sites simultaneously. It is also prone to human error and bias. Against this backdrop, with the advent of Industrial IoT (IIoT), there is an increasing trend to install sensors (such as video cameras) that record industrial environments (such as factories), and to use artificial intelligence (AI) to infer and observe what is happening in the factory from the data. Inference can include drawing some conclusions about events occurring in the factory based on a combination of asset and worker conditions. Inference can also infer worker behavior and key performance indicators (KPIs) related to worker tasks, such as compliance or violation of SOPs and cycle time.

[0005] A common AI approach to image data is to use image classification. Different classifications include "cycle start" and "cycle stop" for cycle time calculation, "normal operation," "prohibited operation," and "unknown operation" for SOP compliance, or "event A" start for generalized event-based analysis. To train such classification machine learning (ML) models, training data containing labeled images must be prepared. To label an image, it is necessary to understand the context of that image. Often, this is a specialized task that can only be performed by industry experts.

[0006] Figure 1 shows an example of a related technical system for image labeling to train a machine learning (ML) model. As shown in Figure 1, a data engineer 401 needs to conduct a joint session with an industry domain expert 101, who will look at images from an input video 201 and explain what industry activity is taking place in each image. This will be used as a label for the images in the input video. Based on the contextual information from the expert's explanation, the data engineer 401 labels the input video images with contextual information to create contextualized image labels 301. [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] As expected, the process described above is not scalable due to the manual human interaction between industry domain experts and data engineers. For example, because industrial activities are highly specialized, human interaction between industry domain experts and data engineers is necessary every time a video analytics solution is introduced to a new industrial assembly line or a new factory. This is because, even with the common issue of SOP compliance, what an SOP means differs from assembly line to factory, and the data needs to be trained accordingly. While there are databases of publicly available images, it should be kept in mind that a large-scale labeled image database containing all data variations is not suitable for industrial scenarios. The main reason why a large-scale labeled image database is unsuitable for industrial scenarios is the sensitive information contained in the images. Therefore, training is required for each new industrial scenario.

[0008] The implementation of related technologies is plagued by the increased time required to scale AI solutions in new industrial environments such as new assembly lines and new factories, as well as the associated costs of hiring data engineers. [Means for solving the problem]

[0009] An exemplary implementation utilizes domain context about industrial activities calculated based on natural language input from industry domain experts, and further includes a method for automatically labeling images in videos observing subsequent instances of that industrial activity with the domain context, without requiring data engineers.

[0010] One embodiment of a method for extracting video context from an external description to solve the above problem is a method for extracting video context from an external description, which involves receiving an input video for processing, running a zero-shot image labeler on the input video to generate multiple labels corresponding to multiple images of the input video, calculating an image embedding vector for the multiple labels, and inspecting an event graph. (Hereafter, "examined event graph" will be read as "examined event graph.") To determine context labels, the context database is referenced using image embedding vectors. Based on the context labels determined temporally for the current and previous images, context labels are generated to replace each of the labels corresponding to each image. The nodes in the event graph represent events that may occur during the video's duration. Each of the multiple labels is replaced with the generated context label.

[0011] Aspects of the present disclosure include a method for extracting video context from an external description, comprising: means for receiving an input video for processing; means for running a zero-shot image labeler on the input video that generates a plurality of labels corresponding to a plurality of images of the input video; means for calculating an image embedding vector for the plurality of labels; means for generating context labels to replace each of the labels corresponding to each of the images, based on a temporally determined context label for the current image and a previous image, by referring to a context database with the image embedding vector in order to determine a context label from an inspected event graph; and means for replacing each of the plurality of labels with the generated context label, where the nodes of the event graph represent events that may occur during the duration of the video.

[0012] Aspects of this disclosure include a computer program that causes a processor to execute instructions that include: receiving video for processing, running an image labeler on the video to generate a plurality of labels corresponding to images in the video, and calculating an image embedding vector for each of the image labels. The program causes the processor to generate context labels that replace each of the labels corresponding to each of the images based on context labels determined for the temporally current and previous images, by referring to a context database with the image embedding vector to determine context labels from an inspected event graph, where the nodes in the event graph represent events that may occur during the duration of the video, and each of the plurality of labels is replaced with the generated context label. The computer program and instructions may be stored in a non-transient computer-readable medium and executed by one or more processors.

[0013] Aspects of the present disclosure include a processor configured to receive video for processing, run an image labeler on the video, generate a plurality of labels corresponding to images in the video, and compute an image embedding vector for each of the image labels. The processor generates context labels that replace each of the labels corresponding to each of the images based on context labels determined for the temporally current and previous images, by referring to a context database with the image embedding vector to determine context labels from an inspected event graph, where the nodes in the event graph represent events that may occur during the duration of the video, and each of the plurality of labels is replaced with the generated context label.

[0014] Aspects of the present disclosure include a system having a context database and a processor, the system which may include a processor configured to receive video for processing, run an image labeler on the video to generate a plurality of labels corresponding to images in the video, and compute an image embedding vector for each of the image labels. The system generates context labels that replace each of the labels corresponding to each of the images, based on context labels determined in time for the current and previous images, by referring to the context database with the image embedding vector to determine context labels from an inspected event graph, where the nodes in the event graph represent events that may occur during the duration of the video, and each of the plurality of labels is replaced with the generated context label. [Effects of the Invention]

[0015] The present invention provides a method for extracting video context from external descriptions that can curb the increase in time required to scale AI solutions in new industrial environments such as new assembly lines and new factories, and reduce the associated costs of hiring data engineers. [Brief explanation of the drawing]

[0016] [Figure 1] FIG. 1 is a diagram showing an example of a related art system for image labeling for machine learning (ML) model learning. [Figure 2] FIG. 2 is a diagram showing an example of a learning mode and an operation mode. [Figure 3] FIG. 3 is a diagram showing an example of an inspection event graph according to an embodiment. [Figure 4] FIG. 4 is a diagram showing an example of context information (node table) of an inspection event graph according to an embodiment. [Figure 5] FIG. 5 is a diagram showing an example of context information (edge table) of an inspection event graph according to an embodiment. [Figure 6] FIG. 6 is a diagram showing an example of an inspection task graph according to an embodiment. [Figure 7] FIG. 7 is a diagram showing context information (node table) of an inspection task graph according to an embodiment. [Figure 8] FIG. 8 is a diagram showing context information (edge table) of an inspection task graph according to an embodiment. [Figure 9] FIG. 9 is a diagram showing the processing of a context generation module according to an embodiment. [Figure 10] FIG. 10 is a diagram showing an example of assigning a value to a weighting parameter αK according to an embodiment. [Figure 11] FIG. 11 is an exemplary explanatory diagram of a solution scenario that can be provided according to an exemplary embodiment. [Figure 12] FIG. 12 shows an exemplary computing environment with an exemplary computer device suitable for use in some exemplary implementations.

DETAILED DESCRIPTION OF THE INVENTION

[0017] The following embodiments will be described in detail. The figures and examples illustrate the embodiments in detail. Reference figures between figures and descriptions of redundant elements have been omitted for clarity. Terms used throughout this specification are provided as examples and are not intended to limit them. For example, the use of the term “automatic” may include fully automatic or semi-automatic embodiments with user or administrator control over certain aspects of the embodiment, depending on the desired embodiment for those skilled in the art practicing embodiments of the present invention. Selection may be performed by the user through a user interface or other input means, or through a desired algorithm. The exemplary implementations described herein may be used individually or in combination, and the functions of the exemplary implementations may be implemented in any way according to the desired implementation.

[0018] Figure 2 shows exemplary learning and operation modes executed by processor 1210 (Figure 12).

[0019] In learning mode 100, raw data context 101a is used, explained in natural language by experts in each industry domain (domain experts). The form that raw data context 101a can take varies. For example, raw data context 101a may include text entered by experts, or it may include text recordings of experts speaking. Instead of human experts, it may be a manual from which important information has been extracted via computer algorithms to construct raw form data (raw data context) 101a.

[0020] Input of data context 101a from industry domain experts is output to and supplied to context description module 102. Context description module 102 outputs an event graph 103 that documents the graphical relationships between the various events described in the input context 101a. This can be done manually only for one sample of the input context 101a. For multiple samples (e.g., input from multiple domain experts 101), a generalized approach can be followed, as is known in related technologies. One way to generalize is to consider events as well as tasks. Even in the case of tasks, the time taken to perform each of these steps can also be potentially extracted from the input of the industry domain experts 101.

[0021] Figure 3 shows an example of an inspection event graph of event graph 103 (see Figure 2) executed by processor 1210 (Figure 12). This is an example in which the event sequence describes a typical workflow related to the inspection of a part at an inspection point. In the example in Figure 3, the nodes of the event graph may include, but are not limited to, nodes where neither the worker nor the part is at inspection point 103-a, node 103-b where the part has arrived at the inspection point but the worker is not present, node 103-c where the part has arrived at the inspection point and the worker is present but not performing the inspection, node 103-d where the part has arrived at the inspection point and the worker is present and performing the inspection, and node 103-e where the part has left the inspection point and the worker is present but not performing the inspection.

[0022] The text embedding module 104 takes text from the event graph 103, generates a high-dimensional vector, and embeds it in the event graph. This is done by taking sentences as input strings and outputting a real-number N-dimensional vector such that sentences with similar meanings have similar embedding vectors in a Euclidean sense. Outputting the N-dimensional vector can be done using various embedding solutions available in related technologies. For example, the sentences "I like Lionel Messi" and "My favorite sport is soccer" are (u -v) 2 Generate embedding vectors "u" and "v" such that the value is small. "u" and "v" represent the embedding vectors for each sentence.

[0023] The event graph 105 with embeddings, generated by the text embedding module 104, is stored in the context database 106. The context database 106 contains two tables. The context database 106 manages the association between the embedding vectors and the context information of each node in the inspected event graph, as described later, and can be implemented by any type of hardware database according to the desired implementation, such as (but not limited to) a storage system or a cloud-based system.

[0024] Figure 4 shows an example of context information 107 (node ​​table) of an inspection event graph according to an embodiment. The first table in Figure 4 is the node table for the example inspection event graph in Figure 3. This table can include the following information:

[0025] An ID field that counts the number of entries.

[0026] The NodeID field uniquely identifies a node in a graph containing event information. For example, a node in an event graph might represent an event that could occur during the duration of a video.

[0027] The Description field contains text generated from the domain expert's input that describes the content of the event. It describes the content of a typical workflow related to the inspection of parts at inspection points in the event sequence in Figure 3.

[0028] The Embedding field is an N-dimensional real-valued vector that represents the embedded value corresponding to the text in the Description field. The value of N depends on the specific embedding method. For example, N=512 is the default value for the associated art embedding CLIP. This Embedding field corresponds to the embedding vector generated by, for example, the text embedding module 104.

[0029] The Estimated Duration field indicates the estimated duration of the event. If an estimated duration is specified, it can be calculated from the input provided by the domain expert. If not specified, the field displays N / A or Not Available.

[0030] The Level field indicates the depth of the graph in which the node is located; in other words, it shows the hierarchy of each node.

[0031] Figure 5 shows an example of context information 107 (edge ​​table) of the inspection event graph according to the embodiment. The second table is an edge table as shown in Figure 5 for the example inspection event graph in Figure 3. This table can include the following information:

[0032] An ID field that counts the number of entries.

[0033] The EdgeID field uniquely names the edges of the graph that connect nodes containing event information. The EdgeID field corresponds to, for example, the inspection points in the workflow shown in Figure 3.

[0034] The Starting Node field specifies the starting node for the given edge.

[0035] The End Node field indicates the end node of the specified edge.

[0036] A Conditions field, if any, captures any additional special conditions needed to capture dependencies between events. In the example above, there are no special conditions.

[0037] Figure 6 shows an example of an inspection event graph 103 according to the embodiment.

[0038] To consider how solutions unfold for more generalized events, let's consider a more detailed inspection event graph, as shown in Figure 6. Here, the events capture only the detailed tasks that the worker must perform. This is referred to herein as an inspection task graph, and the term "task" is used interchangeably with "event" only for this type of situation. Thus, Figure 6 can be called an inspection task graph.

[0039] In the first step 103-f, the worker stands in front of the assembly line belt and waits for the part to arrive and stop under the inspection magnifying glass. Next, the worker refers to the work instructions and determines the specifications to be checked on both the right side 103-h and the left side 103-g of the part. After checking the specifications on one side, the worker proceeds to inspect the sides of the part (103-i, 103-j). The worker may begin the inspection on either side first. Only after both sides have been inspected can the worker proceed to the next step. This step involves signing the Manufacturing Execution System (MES) kiosk to indicate that the work is complete (103-k). After the work is completed, the worker takes one of two actions, depending on whether they have determined the entire part to be defective: either discard the part in the defective box 103-l or press button 103-m to advance the part.

[0040] Furthermore, the event list also describes tasks that workers are not allowed to perform, such as picking up parts (step 103-x).

[0041] Note that steps that should not be taken, such as step 103-x, are described in a novel way, in contrast to the related techniques.

[0042] Figure 7 shows the context information 107 (node ​​table) corresponding to the inspection task graph shown in Figure 6. Note that some events (tasks) are at the same level, indicating tasks that can be executed in any order. Also, prohibited events (tasks, procedures) are assigned a special level number (for example, a node indicating the prohibited action of picking up a part has the level "-1").

[0043] Figure 8 shows the context information 107 (edge ​​table) of the inspection task graph, corresponding to the inspection task graph shown in Figure 6. As mentioned above, the Condition field now captures special conditions that are additionally required to capture inter-task dependencies, such as AND and NOT conditions.

[0044] In operating mode 200 (Figure 2) of the computer device 1205 (Figure 12), it automatically labels videos (input videos 201) that observe the same activities for which an industry domain expert provided input in learning mode 100.

[0045] When the zero-shot image labeler 202 receives an input video 201, it generates text-based image labels 203. The zero-shot image labeler 202 performs the generation of multiple labels corresponding to multiple images in the input video 201. These labels are based on a model pre-trained on a large corpus of publicly available datasets (hence the name "zero-shot"). Therefore, while the image labels 203 correlate to some extent with the activity occurring in the input video 201, they cannot fully capture the domain context. This is because, although the training data for the zero-shot image labeler 202 is publicly available, it does not contain domain-specific data.

[0046] The same text embedding module 204 as text embedding module 104 receives multiple image labels 203, calculates an embedding vector for each image label, and outputs an embedded image label 205 with the embedding vector attached. The embedded image label 205 is then output to the context generation module 206.

[0047] Figures 9 and 10 illustrate the process of determining context labels from the inspected event graph. Context label determination involves generating context labels based on temporally determined context labels for the current and previous images by referencing the context database 106 using image embedding vectors computed by the text embedding module 204. Each of the generated context labels replaces the corresponding label for each image. The event graph nodes represent events that may occur during the video duration. Each of the multiple labels is replaced with the generated context label.

[0048] Figure 9 shows the processing of the context generation module 206 executed by the processor 1210 (Figure 12). The context generation module 206 includes the following steps:

[0049] In step 206-1, the context generation module 206 calculates a task embedding list R of prohibited events from the event graph 105. These include examples such as the level = -1 event in Figure 7.

[0050] In step 206-2, the context generation module 206 receives an image embedding IK (embedded image label 205) as input, where 0 ≤ K ≤ M-1, and M is the total number of images in the input video.

[0051] In step 206-3, the context generation module 206 receives an event embedding tn, where 0 ≤ n ≤ T-1, and T is the total number of nodes in the task graph with valid actions such that tn does not belong to the list R from step 206-1.

[0052] In step 206-4, the context generation module 206 checks whether the image embedding IK is reliably close to the entry for which list R is set. To do this, the context generation module 206 calculates the following:

[0053]

number

[0054] Here, D(x,y) calculates the distance between two vectors x and y, similar to the Euclidean distance. If this condition is true, a warning is issued. To understand the meaning of this condition, consider the case where the overall application was for SOP compliance. This condition means that an SOP violation has occurred.

[0055] In step 206-5, the context generation module 206 determines the distance in the embedding space and the previous image embedding I K-1 Considering both the calculated tasks for the image embedding IK, the event embedding closest to the image embedding IK nCalculate this. To execute this, calculate the following.

[0056]

Number

[0057] Here, d(a, b) calculates the distance between nodes a and b of the event graph. Therefore, this term attempts to set n at time K to the node of the task graph that is the next node determined at the previous time instant. That is, the node at the previous time is n K and the next node of the es graph is n K-1 + 1. This is because the task graph is constructed considering the order of tasks. The parameter α (K-1 indicates how much weight should be given to this term. This is determined as follows as a function of the estimated duration value from the node table. K Find n

[0058] Find n K-1 ≠ n K and set α K = α0. Here, α0 is a certain initial number. n K+i = n K is set to α K+i = α0g(i). Here, g(i) is an increasing function of i, and the growth rate is directly proportional to the estimated duration value. This idea is to capture the intuition that when a new task is started and enough time has passed until its estimated duration value, the next task is expected to start.

[0059] In this way, in order to determine the context label from the inspected event graph, by referring to the context database with the image embedding vector, based on the context labels determined temporally for the current image and the previous image, generate a context label for replacing each label corresponding to each of the images, and replace each of the plurality of labels with the generated context label.

[0060] Figure 10 shows the weighting parameter α, executed by processor 1210 (Figure 12) according to an embodiment of what was described in

[0067] . K This figure shows an example of assigning values ​​to a variable. Figure 10 shows an example of event selection based on graphical considerations.

[0061] In step 207-1, event selection determines that a new task was started at time K.

[0062] In step 207-2, the event selection is a K Initialize it to a low value a0.

[0063] In step 207-3, event selection reads the estimated duration, which is the estimated time for the new task T.

[0064] In step 207-4, event selection is performed for each subsequent time instant K+i, a K Let a = a0exp ( iw / T ) (where w is a constant), K It increases. In this case, g(i) = exp (iw / T) In step 206-6, the content generation module 206 replaces the label of image k with the task description of task n, where n is calculated from step 206-5, and generates an image label with context 301.

[0065] As an example, consider the event graph in Figure 3 to see how the solution works in a situation like the one shown in Figure 11. Figure 11 is an illustrative diagram of a possible solution according to the embodiment. As shown in Figure 11, the three image frames I1, I2, and I3, and the zero-shot image classifier, generate a description that is not inaccurate but fails to capture the domain context mentioned in the event graph Figure 3. This can be done as the output of the context generation module by appropriately selecting the following parameters. Figure 11 also shows α within the dashed box. K=0 is indicated by a dashed box. In this case, the judgment does not consider the graphical relationship between events and incorrectly predicts that the worker is still performing the inspection. Appropriate α K By incorporating this, the solution correctly predicts that the next task should have started.

[0066] Through the embodiments described herein, the time required to deploy AI solutions that require domain-context labeled image data can be reduced. Furthermore, the exemplary implementations also reduce costs because they require fewer resources for manual data engineering.

[0067] Figure 12 shows an exemplary computing environment having exemplary computer devices suitable for use in several exemplary implementations. The computer device 1205 in the computing environment 1200 may include one or more processing units, cores, or processors 1210, memory 1215 (e.g., RAM, ROM, and / or similar), internal storage 1220 (e.g., magnetic, optical, solid-state storage, and / or organic), and / or an I / O interface 1225, any of which may be coupled to a communication mechanism or bus 1230 for communicating information, or embedded in the computer device 1205. The I / O interface 1225 may also be configured, depending on the desired implementation, to receive images from a camera or provide images to a projector or display.

[0068] Computer device 1205 may be communicatively coupled to an input / user interface 1235 and an output device / interface 1240. Either or both of the input / user interface 1235 and the output device / interface 1240 may be wired or wireless interfaces and may be detachable. The input / user interface 1235 may include any physical or virtual device, component, sensor, or interface that can be used to provide input (e.g., buttons, touchscreen interfaces, keyboards, pointing / cursor controls, microphones, cameras, Braille, motion sensors, accelerometers, optical readers, and / or similar). The output device / interface 1240 may include displays, televisions, monitors, printers, speakers, Braille, etc. In some exemplary implementations, the input / user interface 1235 and the output device / interface 1240 may be embedded in or physically coupled to the computer device 1205. In other exemplary implementations, other computer devices may function as, or provide, the functions of the input / user interface 1235 and output device / interface 1240 of computer device 1205.

[0069] Examples of computer devices 1205 include, but are not limited to, highly mobile devices (e.g., smartphones, devices mounted in vehicles and other machines, devices carried by humans and animals), mobile devices (e.g., tablets, notebooks, laptops, personal computers, portable televisions, radios, etc.), and devices not designed for mobility (e.g., desktop computers, other computers, information kiosks, televisions, radios, etc., having one or more processors embedded therein and / or coupled thereto).

[0070] Computer device 1205 may be communicatively coupled (for example, via I / O interface 1225) to external storage 1245 and network 1250 for communication with any number of network-connected components, devices, and systems, including one or more computer devices of the same or different configurations. Computer device 1205 or any connected computer device may function, provide services, or be referred to as a server, client, thin server, general machine, special-purpose machine, or other label.

[0071] The IO interface 1225 may include, but is not limited to, wired and / or wireless interfaces using any communication or IO protocol or standard (e.g., Ethernet, 802.11x, Universal System Bus, WiMAX, modem, cellular network protocol, etc.) for communicating information to and from at least all connected components, devices, and networks within the computing environment 1200. The network 1250 may be any network or combination of networks (e.g., the Internet, local area network, wide area network, telephone network, cellular network, satellite network, etc.).

[0072] Computer device 1205 may use and / or communicate using computer-usable media or computer-readable media, including transient media and non-transient media. Transient media include transmission media (e.g., metal cables, optical fibers), signals, carrier waves, etc. Non-transient media include magnetic media (disks, tapes, etc.), optical media (CD-ROMs, digital video discs, Blu-ray discs, etc.), solid-state media (RAM, ROMs, flash memory, solid-state storage, etc.), and other non-volatile storage or memory.

[0073] Computer device 1205 can be used to implement techniques, methods, applications, processes, or computer executable instructions in several exemplary computing environments. Computer executable instructions may be obtained from transient media, stored in non-transient media, and retrieved from non-transient media. Executable instructions may originate from one or more of any programming languages, scripting languages, and machine languages ​​(e.g., C, C++, C#, Java®, Visual Basic, Python, Perl, JavaScript®, etc.).

[0074] The processor(s) 1210 can run under any operating system (OS) (not shown) in a native or virtual environment. One or more applications can be deployed, including a logical unit 1260, an application programming interface (API) unit 1265, an input unit 1270, an output unit 1275, and an inter-unit communication mechanism 1295 for different units to communicate with each other, with the OS, and with other applications (not shown). The units and elements described may vary in design, function, configuration, or implementation, and are not limited to the description provided. The processor(s) 1210 may take the form of a hardware processor such as a central processing unit (CPU), or a combination of hardware and software units.

[0075] In some exemplary implementations, once information or execution instructions are received by the API unit 1265, they may be transmitted to one or more other units (e.g., a logic unit 1260, an input unit 1270, and an output unit 1275). In some embodiments, the logic unit 1260 may be configured to control the flow of information between units and direct the services provided by the API unit 1265, the input unit 1270, and the output unit 1275, in some exemplary implementations described above. For example, one or more processes or implementation flows may be controlled by the logic unit 1260 alone or in conjunction with the API unit 1265. The input unit 1270 may be configured to receive input for the computations described in the exemplary embodiments, and the output unit 1275 may be configured to provide outputs based on the computations described in the exemplary embodiments.

[0076] The processor(s) 1210 is configured to execute a method or instruction that includes receiving a video 201 for processing, running an image labeler 202 on the video to generate a plurality of image labels 203 corresponding to the images in the video, and calculating an image embedding vector for each of the image labels. The processor 1210 also generates context labels 206 to replace each of the labels corresponding to each of the images based on the context labels determined for the temporally current and previous images, by referring to a context database with the image embedding vector to determine a context label from the inspected event graph, where the nodes in the event graph represent events that may occur during the duration of the video, and replace each of the plurality of labels with the generated context label 301, as shown in Figure 2. It also replaces each of the plurality of labels with the generated context label 301, as described herein with respect to Figure 11.

[0077] Depending on the desired implementation, the context database manages the association between event embedding vectors and contextual information with each node of the inspected event graph, via an event graph with embedding 105, as shown in Figure 5.

[0078] The processor(s) 1210 is configured to perform the methods or instructions described herein. Here, generating a context label to replace each of the labels corresponding to each of the images further includes determining a node from a checked event graph associated with an event embedding vector in a context database having the minimum distance from the image embedding vector, and generating a context label from the context information associated with the node, as described with respect to Figure 9.

[0079] A processor(s) 1210 is configured to perform the methods or instructions described herein, where each node is associated with an estimated duration in the context database. Here, the calculated distance between the image embedding vector and the event embedding vector is weighted, compared to the estimated duration, based on the duration of time between the video image and the current and previous images within the time associated with the same context label, as described with respect to Figure 10.

[0080] Depending on the desired implementation, the context database can manage prohibited procedures within the inspected event graph. In such an exemplary implementation, the processor(s) 1210 may be configured to perform the methods or instructions described above, and further include raising an alert for non-compliance with standard operating procedures for generated context labels containing one of the prohibited procedures, as described with respect to Figure 9.

[0081] The processor(s) 1210 is configured to perform the methods or instructions described herein, and further includes identifying a task from the generated context labels and determining the cycle time of the identified task from the length of the video across the generated context labels associated with the identified task, as described with respect to Figure 10. In an exemplary implementation, the estimated durations between each task can be aggregated to determine the cycle time.

[0082] The processor(s) 1210 is configured to perform the methods or instructions described above, and further includes providing an indication that an abnormal event has occurred for one or more of the generated context labels indicating an abnormal event. For example, if it is necessary to consider an abnormal event related to a node in the inspected event graph and issue a warning, the event can be detected through the exemplary implementations described herein and controlled to indicate that an abnormal event has occurred.

[0083] The processor(s) 1210 can be configured to perform the methods or instructions described herein, further including storing the generated context labels along with the video and indexing the video with a search engine. In an exemplary implementation, each of the generated context labels is associated with a time in the video, so the context labels, along with the corresponding timestamp and duration, can be provided to a search engine to index the video based on time and / or label, according to the desired implementation.

[0084] Some parts of the detailed explanation are presented in terms of symbolic representations of algorithms and computer operations. These algorithmic descriptions and symbolic representations are means used by those skilled in the field of data processing technology to convey the essence of the innovation. An algorithm is a set of defined steps that lead to a desired final state or result. In the examples, the steps performed require a visible amount of physical operation to achieve the visible result.

[0085] Unless otherwise stated, as will be evident from the discussions, discussions throughout this specification using terms such as “processing,” “calculation,” “computation,” “determination,” and “display” may include the operations and processes of a computer system or other information processing device that manipulate and convert data represented as physical (electronic) quantities in the registers and memory of a computer system into other data similarly represented as physical quantities in the memory or registers of a computer system or other information storage, transmission, or display devices.

[0086] Exemplary embodiments also relate to apparatus for performing the operations described herein. This apparatus may be specifically configured for a particular purpose and may include one or more general-purpose computers that are selectively started or reconfigured by one or more computer programs. Such computer programs may be stored on computer-readable media such as computer-readable storage media or computer-readable signal media. Computer-readable storage media may include, but are not limited to, tangible media such as optical disks, magnetic disks, read-only memory, random-access memory, solid-state devices, and drives, and may include any other type of tangible or non-transient media suitable for storing electronic information. Computer-readable signal media may include media such as carrier waves. The algorithms and representations presented herein are not inherently related to any particular computer or other apparatus. Computer programs may include purely software implementations containing instructions that perform the operation of a desired implementation.

[0087] Various general-purpose systems may be used with the programs and modules according to the embodiments herein, or it may be convenient to construct more specialized devices for performing the desired method steps. Furthermore, the embodiments are not described with reference to any particular programming language. It will be understood that various programming languages ​​may be used to carry out the teachings of the embodiments described herein. Instructions in a programming language may be executed by one or more processing units, such as a central processing unit (CPU), a processor, or a controller.

[0088] As is known in the art, the operations described above can be performed by hardware, software, or any combination of software and hardware. Various embodiments of the exemplary implementations may be implemented using circuit and logic devices (hardware), while other embodiments, when performed by a processor, may be implemented using instructions stored on a machine-readable medium (software) that causes the processor to perform the method of performing the implementation of the application. Furthermore, some exemplary implementations of the application may be performed by hardware alone, while other exemplary implementations may be performed by software alone. Moreover, the various functions described may be performed by a single unit or may span a number of components in any number of ways. When performed by software, the method may be performed by a processor such as a general-purpose computer based on instructions stored on a computer-readable medium. If desired, the instructions may be stored on the medium in a compressed and / or encrypted format.

[0089] Furthermore, other embodiments of the present application will be apparent to those skilled in the art from the considerations of the present specification and the practice of the teachings of the present application. Various aspects and / or components of the exemplary embodiments described herein can be used individually or in any combination. This specification and the exemplary embodiments are intended to be considered illustrative only, and the true scope and spirit of the present application are shown by the following claims. [Explanation of symbols]

[0090] Experts in 101 Industry Domains 101a Raw data context 102 Context Description Module 103 Event Graph 104 Text Embedding Module Event graph with 105 embeddings 106 Context Database 107 Contextual Information 201 Input Video 202 Zero Shot Image Labeler 203 Image Labels 204 Text Embedding Module 205 Embedded image labels 206 Context Generation Module 301 Contextualized image labels

Claims

1. In a method for generating contextualized image labels that attach descriptions as labels to images in an input video, In learning mode, Enter the data context in raw format, From the input data context, an event graph showing the relationships between events is output. Read the text from the event graph, generate a vector, and embed it in the event graph. An embedded event graph containing embedded vectors is stored in a context database, and the context database associates the embedded vectors with context information for each node in the inspected event graph. In operating mode, Receive input video for processing, A zero-shot image labeler is executed on the input video to generate multiple labels corresponding to multiple images in the input video. For the aforementioned multiple labels, an image embedding vector is calculated, and the embedded image label with the embedded vector is output. In order to determine context labels from the inspection event graph, the image embedding vector is used to generate context labels that are used to replace each of the labels corresponding to each of the multiple images, based on the context labels determined temporally for the current image and previous images, by referring to the stored context database. The nodes in the event graph above represent events that may occur during the duration of the video. The generated context labels are used to replace each of the multiple labels. How to generate contextualized image labels.

2. In a method for generating contextualized image labels according to claim 1, To replace each of the labels corresponding to each of the aforementioned multiple images, generating a context label is: Determine the node associated with the event embedding vector from the inspection event graph in the context database that has the minimum distance from the image embedding vector. Generate context labels from context information associated with a node. How to generate contextualized image labels.

3. In the method for generating contextualized image labels according to claim 2, Each of the aforementioned nodes is associated with the estimated duration of the context database, The calculated distance between the image embedding vector and the event embedding vector in the context database having the minimum distance from the image embedding vector is weighted based on the time duration between the video image and the current and previous images within the time associated with the same context label, compared to the estimated duration. How to generate contextualized image labels.

4. In a method for generating contextualized image labels according to claim 1, The aforementioned context database manages prohibited procedures within the inspection event graph, If the generated context label contains one of the prohibited procedures, a warning regarding non-compliance with standard operating procedures will be issued. How to generate contextualized image labels.

5. In a method for generating contextualized image labels according to claim 1, Furthermore, the task is identified from the generated context label, The cycle time for an identified task is calculated from the length of the video across the generated context labels associated with that task. How to generate contextualized image labels.

6. In a method for generating contextualized image labels according to claim 1, For one or more generated context labels indicating an abnormal event, provide a display indicating that an abnormal event has occurred. How to generate contextualized image labels.

7. In a system that generates contextualized image labels that attach descriptions as labels to images in an input video, It has a context database and a processor, The aforementioned processor, In learning mode, Enter the data context in raw format, From the input data context, an event graph showing the relationships between events is output. Read the text from the event graph, generate a vector, and embed it in the event graph. An embedded event graph containing embedded vectors is stored in a context database, and the context database associates the embedded vectors with context information for each node in the inspected event graph. In operating mode, Receive input video for processing, A zero-shot image labeler is executed on the input video to generate multiple labels corresponding to multiple images in the input video. For the aforementioned multiple labels, an image embedding vector is calculated, and the embedded image label with the embedded vector is output. To determine context labels from the inspection event graph, a context database stored in image embedding vectors is referenced to generate context labels that replace each of the labels corresponding to each of the multiple images, based on the context labels determined temporally for the current image and previous images. The nodes in the event graph above represent events that may occur during the duration of the video. Replace each of the aforementioned labels with the generated context label. A system that generates image labels with context.

8. In a system for generating contextualized image labels according to claim 7, Generating a context label to replace each of the labels corresponding to each of the aforementioned multiple images is, Determine the node associated with the event embedding vector from the inspection event graph in the context database that has the minimum distance from the image embedding vector. Generate context labels from context information associated with a node. A system that generates image labels with context.

9. In a system for generating contextualized image labels according to claim 8, The nodes in the aforementioned event graph are associated with the estimated duration in the context database. The processor weights the calculated distance between the image embedding vector and the event embedding vector based on the time duration between the video image and the current and previous images within the time associated with the same context label, compared to the estimated duration. A system that generates image labels with context.

10. In a system for generating contextualized image labels as described in claim 7, The aforementioned context database manages prohibited procedures within the inspection event graph, The aforementioned processor further, If the generated context label contains one of the prohibited procedures, control the system to issue a warning about non-compliance with standard operating procedures. A system that generates image labels with context.

11. In a system for generating contextualized image labels as described in claim 7, The aforementioned processor further, Identify the task from the generated context label, The cycle time for the identified task is calculated from the total video length of the generated context labels associated with the identified task. A system that generates image labels with context.

12. In a system for generating contextualized image labels as described in claim 7, The aforementioned processor further, Control whether one or more generated context labels indicating an abnormal event are displayed to show that an abnormal event has occurred. A system that generates image labels with context.

Images