Domain knowledge for activity verification via image recognition

Abstract

A method for verifying machine-vision classifications from singular or a plurality of image data includes capturing image-related data, identifying data and ingesting the data into a semantic framework to generate additional context relating to an activity occurring in the sequence of captured image data. Using the generated context, an action may be inferred as occurring in the sequence of captured image data. The activity or action may be related to a safety requirement or that a task has or has not been performed. Each task may be associated with a semantic template by linking people performing the task, actions performed in the task, and objects present during the task. Upon verification of the task status, a notification indicating the verification status may be communicated to an appropriate person. Notifications may be provided by a human machine interface that can be augmented reality (AR) application for display to a user.

Description

DOMAIN KNOWLEDGE FOR ACTIVITY VERIFICATION VIAIMAGE RECOGNITIONTECHNICAL FIELD

[0001] This application relates to machine vision. More particularly, this application relates to identifying semantic information from captured image data or video streams.BACKGROUND

[0002] Image recognition from camera and sensor feeds are used to analyze a scene for specific elements; as of now, focus has been on determining what is in an image and how accurate it is. There are and will be other tasks that require analysis over time and in combination with other data to evaluate a conclusion about what has happened (task complete) or if something could have happened as prescribed (safety equipment used) which is something that happens manually or with excessive computational processing and often with custom algorithms.SUMMARY

[0003] According to embodiments described in this disclosure, a method for verifying activity in a plurality of images includes capturing in an image capture device, a sequence of captured images, identifying data in the sequence of captured images and providing the identified information to a semantic framework of an environment containing the objects in the sequence of captured images, and generating additional context relating to an activity occurring in the sequence of captured images. Using the generated context, an action may be inferred as occurring in the sequence of captured images. The activity 'or action may be related to a safety requirement that may be observed in the capturedimages. In some embodiments, the activity may indicate that a task has or has not been performed as expected. For example, for a given task, it may be required that a particular tool be present, the images may be processed to identify the presence of the required tool. In embodiments, a confirmation that a task is performed, may require that a particular person or a required number of people are present in the scene. Additionally, or alternatively, task completion may be inferred by the time elapsed being sufficient for performing the task even though the actual task is not captured in the image sequence. For example, a person may be observed entering a room or be in the scene behind another object. A known task is expected to be performed in the field of view. Information about the task, such as how long it should take to perform the task, and whether the observed person possesses the skill to perform the task, is utilized to determine a status of the task. For example, if the person is seen entering and leaving the location and remains on site to perform the task, an inference may be made that the task was performed. Each task, or step in a task may be associated with a semantic template by linking at least two of people performing the task, actions performed in the task, and objects present during the task. Upon verification of the task status, a notification indicating the verification status of the task may be communicated to an appropriate person, robot, manufacturing process, or other entity. Notifications may be provided by way of a human machine interface that also utilizes augmented reality (AR) applications for display to a user. In some embodiments, the notification may be used for further learning by passing the notification back to a learning model.

[0004] According to another embodiment of the disclosure, a system for verifying activity in a plurality of images includes a computer processor and a non-transitorycomputer memory in communication with the computer processor, the non-transitory computer memory storing instructions that when executed by the computer processor, cause the computer processor to perform the steps of capturing in an image capture device, a sequence of captured images, identifying data in the sequence of captured images and providing the identified information to a semantic framework of an environment containing the objects in the sequence of captured images, and generating additional context relating to an activity occurring in the sequence of captured images. The non-transitory computer memory may further store instructions causing the computer processor to perform the steps of inferring an action occurring in the sequence of captured images based on the semantic framework.

[0005] The verified action may involve any number of contextual information such as a safety requirement or confirmation that a task has been performed. Confirmation that the task has been performed comprises in the computer processor, verifying that a required tool for performing the task is present in the sequence of captured images, verifying that an expected entity is performing the task in the sequence of captured images, or that a minimum amount of time to perform the task has elapsed even is the task itself is not captured in the image sequence. The computer processor may further create a semantic template for the task by linking at least two of people performing the task, actions performed in the task, and objects present during the task. The system may generate and communicate a notification regarding the status of the task. The notification may be communicated via a human machine interface which can include an AR application to a mobile device, such as a headset, a tablet or a mobile phone or a computer display.BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The foregoing and other aspects of the present invention are best understood from the following detailed description when read in connection with the accompanying drawings. For the purpose of illustrating the invention, there is shown in the drawings embodiments that are presently preferred, it being understood, however, that the invention is not limited to the specific instrumentalities disclosed. Included in the drawings are the following figures:

[0007] FIG. 1 is an illustration of a task captured in images and identifying task actions in the images according to aspects of embodiments of this disclosure.

[0008] FIG. 2 is an illustration of a sample image for identifying a necessary component of a task in an image according to aspects of embodiments of this disclosure.

[0009] FIG. 3 is an illustration of a sample image for identifying the performance requirement of a task in an image according to aspects of embodiments of this disclosure.

[0010] FIG. 4 is a process flow diagram for identifying activity in a sequence of images according to aspects of embodiments of this disclosure.

[0011] FIG. 5 is a block diagram of a computer system, which may be used to make and use aspects of embodiments of this disclosure.DETAILED DESCRIPTION

[0012] In the embodiments described hereinafter, sequences of common images are used to demonstrate a process of transitioning from such native data sources (machine vision produced data formats) to ingestion to a structured semantic framework (sampleinference, rules, query) and returning additional context allowing for conclusion data that may be later extended to industry repetitive tasking and quality assurance.

[0013] An object of the process is to use objects and actions observed in multiple images over time to understand more information about what has happened, what could have happened, and what should have happened for cyclical industry processes where additional context is necessary for certain actions. For example, validating if a task was executed in a given space may be determined based on context derived from the sequence of images. It is also possible to monitor, for example, that no change in state occurred to detect that an object was not placed or moved as indicated or expected.

[0014] Embodiments of this disclosure provide a semantic orchestration template for entities, objects, and other elements detected in image sequences in order to align those tags for use in semantic inferencing technology to gain contextual conclusions with respect to industry-style settings. This template is based not just on what is observed, but what should be observed given a set of domain parameters such as expert inspection knowledge, worker tasking, or manufacturing processes.

[0015] Industry activities are often repetitious in known spaces with known people and known equipment. Even so, there are still questions regarding workers, continuous risks, tasking that involves validating steps, and quality checking, among others. (Especially nowadays with a fast attrition and new hiring). These factors require data, experience, knowledge, and other formats of information in order to answer these questions. To monitor or predict all possible events in real-time requires immense amounts of equipment and processing, particularly custom image processing. There exist numerous image processing algorithms that attempt to automatically tag images. Once tagged,existing semantics are used to help refine the tags generated using similarities or to use similar objects to help improve the tagging algorithms. Embodiments of the methods and systems described herein introduce templates for repeatable processes that link object, people, actions, and features, etc. together for use in validation sequences. While conventional image processing may determine what is in an image, that information is not used to validate procedures, states, actions, and other industry spatial situations.

[0016] FIG. 1 is an illustration for monitoring a simple example task according to an embodiment of this disclosure. The process will be described with reference to a simple task of Caring for a particular dog (dog A 109). The task of caring for dog A 109 includes 3 procedural steps. First dog A 109 must be collected 103. Second, dog A 109 should be brushed 105. Finally, dog A 109 should be walked 107. Each step may be associated with certain verifying information. For instance, brushing dog A 109 may require that a dog brush is detected. Likewise, walking dog A 109, may require that a leash is observed, and that the surrounding location changes. Further, predefined timespans may be required for a task. For example, brushing dog A 109 may be deemed as requiring a minimum of five minutes to complete.

[0017] Each procedural step is associated with a template for the process that defines the elements that make up the step. In step 103, for collecting dog A 109, the template includes verifying that dog A 109 is present in the image, and that the location of the image is conducive to walking dog A 109. The image associated with step 103 shows two dogs, dog A 109 and dog B 1 11 . Image processing may be used to identify dog A 109 as the subject of interest and that the dogs appear to be located at a park, providing a potential location for walking. Over time it may be noticed that a human 113 enters thescene, followed by the absence of dog A 109 in the images. From this information it may be inferred that dog A 109 was picked up and step 1 of the task is complete.

[0018] In the image depicting step 2 105, the image includes dog A 109, a human actor 113, and a tool (dog brush) 1 15. The template for brushing dog A 109 may include verifying that proper tool 115 is present, that the human 1 13 is holding the correct tool 1 15, that a particular motion is occurring (brushing motion as opposed to petting), and that a sufficient time has elapsed for example, a minimum of five minutes to complete step 2 105. In the final step 107, the image depicts a human 113 with dog A 109 connected by a leash. The template for this step may also define the location (a park) and require that the background changes to indicate movement. A minimum timeframe may be included (e.g., 30 minutes) to ensure that dog A 109 has been received sufficient exercise from the walk. Once the third step has been satisfied, additional semantic queries may be triggered, such as verifying that the completed steps were completed in the proper order. Based on the successful completion of the task, an update may be returned via a human machine interface that dmay be an AR device such as a headset, a tablet, or a mobile device.

[0019] The cyclical nature of these industry processes allows the aspects and features of these processes to be encoded for semantic inference, which may replace some of the time-intensive real-time camera processing and validation or at least assist with these pipelines. Furthermore, using cameras and sensors, real future training data may be gathered within industry buildings. This data collection and training may be automated. However, several layers are required to connect these data resources, other data resources (text, database, heuristics), and semantic knowledge are required to enabledecision making beyond manual comparisons, time-intensive verifications for quality assurance. Prior to implementing such a pipeline for real industry data, conventional methods required gaining a thorough understanding of all of the data transformations and resource connections through a combination of manual analysis, consulting manuals, database entries, and human experiential knowledge.

[0020] Now it is more streamlined to orchestrate these data components together for complex scenarios which still have knowledge gaps today. In the longer term, efficiency in reducing compute power for certain image processing steps is also potentially possible if applied strategically. For example, if semantics can help narrow down activities and validate steps, then certain image compute resources can be more selectively applied to other camera and sensor feeds (ideally dynamically as conclusions are drawn) were a scenario needs more camera angles, segmentation, etc. Semantics may even be used in combination with image processing to alleviate processing for higher resolutions for certain scenarios.

[0021] Contextualizing scenes based on repeatable industry procedures through time also means that richer information can be passed back to the workers as well as those actors monitoring a situation. This knowledge can become integral for automation, worker assisted technologies like augmented reality, robotics, and other application such as learning tools and language translation depending on user customizations. It also largely paves the way to pass data back to learning models and send data to applications like those used for augmented reality all linked to knowledge graph constructs for real-time updates and dynamic decisioning.

[0022] The technical features of embodiments of this disclosure include components to achieve a new method of semantic verification templates from image tagging and using the tagging data for semantic verifications and validations once it exists.

[0023] 1 . Determine an observed task state: Identify the parts of the observed state that exist or have been identified by the image processing. Examples of this include the following: Objects / object features detected, actors (humans, robots, etc.) / actor features detected, length of time of detection, actions and length of time actions have been occurring, locations, and so on.

[0024] 2. Match to known template set: Using 1 ) a pre-existing task (or step) in industrial processes, these with their constraints and requirements can be stored as a semantic reference and then used to 2) check against the incoming scene / process observation data. Ingest the new information into the semantic models.

[0025] 3. Infer the results of the industrial step: These together (set in #2) can inform if a step is considered validated or not and why (i.e. safety helmet and gloves are observed on worker in image, therefore the safety equipment requirement is achieved OR the metal has been cooling for x amount of time, therefore it has been long enough to move without damage and complete the next step.

[0026] One sample implementation includes using sample images and entity tagging as can be done through an open-source tool for contextual object segmentation and recognition such as Common Objects in Context or COCO (https: / / cocodataset.Org / #home), by non-limiting example. This allows object segmentation and observation to be used in a way that it is linked to entity tagging, entity recognition, and then to semantics and does not require accumulation of learning dataand new industry-specific detection algorithms which are time-intensive for initial implementation. These simple samples are replaced with real industry data as it is accumulated for known tasks and processes that can be encoded in semantic structures following this orchestration approach. The usage of this semantic templating approach will be explained in the three following examples. Other potential uses may be addressed without departing from the scope of this disclosure.

[0027] Example Scenario 1 : In a building complex or on the shop floor, a human was at a certain location and orientation for a task for a known amount of time (say to mount a PLC with power supply). Even though what they were doing cannot be seen precisely, knowing how long a given task needs for proper completion (even at a minimum bound) helps determine if that task is even on deck to be checked off as complete or if a new human needs to be sent to re-check. Completion monitoring can help determine if it is worth expending time for another worker to proceed to the scene to do further quality assessment.

[0028] Example Scenario 2: In a production facility that just received a new shipment of products, there is a box that needs to be moved from one place to another. Cases where the task for the worker is to move this may indicate that worker did go to that location, remained there long enough, but left without taking the box with them. If the system knows that 1 ) these actions happened from image processing, but that also 2) something else should have happened, then there may already exist a semantic hook to alert or observe for gestures that something is missing, is a wrong part, damaged, etc. enabling assistance for what to do next with minimal time and without calling over other works from their locations.

[0029] FIG. 2 is an illustration of scenario 2, where a box 201 is expected to be moved from its current location to a second location 203. The worker 210 is identified as the worker who is supposed to move the box 201 to its new location 203 and worker 210 remains in the scene for a time period of sufficient length to complete the task. However, the worker 210 leaves scene without taking the box 201. The system may provide a notification to worker 210 with next steps to perform to ensure that the task is completed.

[0030] Example Scenario 3: In a warehouse, large items are moved around regularly. A certain number of people are needed to complete the lifting and repacking of large items such as a heavy train door. This knowledge can come from the semantic graph that is informed about processes ahead of time to help conclude how many workers it ‘should’ be detecting in the image frames for this task. This graph data can eventually feed the image learning algorithms such that dynamic image processing can happen to look for specific elements in a scene that should exist for safety, completion, etc.

[0031] FIG. 3 provides an illustration 300 of scenario 3, where two workers 301 , 303 are moving a heavy object 305 in the plant. If the semantic graph indicates that object 305 may only safely be moved by four workers, the system may create an alert and communicate the alert to an appropriate party. For example, workers 301 , 303 may be alerted to the possible hazard of moving the object 305 without enough assistance, which may cause injury or damage to the object 305. Alternatively, a supervisor may be alerted to the safety violation in order to intercede and provide the required number of workers to complete the task.

[0032] FIG. 4 is a process flow diagram for inferring an action in captured images according to aspects of embodiments of this disclosure. A sequence of captured imagesor information is obtained 401 from a scene. Conventional image processing may be used to classify and identify objects captured in the images 402. Information may be obtained from a variety of sources including but not limited to, RGB cameras, RGBD cameras, infrared sensors or a point cloud. Information obtained from the images is ingested into a semantic framework 403, which allows for querying and inference from the knowledge graph of the semantic framework. Using the semantic framework, context relating to the sequence of images is generated 404. Based on the semantic framework and the generated context, an action (or absence of an action) occurring in the images is inferred405. The status of the inferred action may be communicated to an appropriate recipient406. For example, a notification of the failure to complete a required task may be communicated to the worker responsible for performing the task.

[0033] These methods provide an improvement in machine vision by leveraging the power of the semantic framework to infer activities that are occurring in the images, in addition to merely classifying and identifying components of the images. This knowledge may be used to enhance tools such as AR applications, which may be used to guide activities in the observed area. These notifications may be communicated in real time to validate and insure that required tasks, including safety measures are performed. This improves safety in environments like industrial settings and improves efficiency and quality.

[0034] FIG. 5 illustrates an exemplary computing environment 500 within which embodiments of the invention may be implemented. Computers and computing environments, such as computer system 510 and computing environment 500, are known to those of skill in the art and thus are described briefly here.

[0035] As shown in FIG. 5, the computer system 510 may include a communication mechanism such as a system bus 521 or other communication mechanism for communicating information within the computer system 510. The computer system 510 further includes one or more processors 520 coupled with the system bus 521 for processing the information.

[0036] The processors 520 may include one or more central processing units (CPUs), graphical processing units (GPUs), or any other processor known in the art. More generally, a processor as used herein is a device for executing machine-readable instructions stored on a computer readable medium, for performing tasks and may comprise any one or combination of, hardware and firmware. A processor may also comprise memory storing machine-readable instructions executable for performing tasks. A processor acts upon information by manipulating, analyzing, modifying, converting or transmitting information for use by an executable procedure or an information device, and / or by routing the information to an output device. A processor may use or comprise the capabilities of a computer, controller or microprocessor, for example, and be conditioned using executable instructions to perform special purpose functions not performed by a general-purpose computer. A processor may be coupled (electrically and / or as comprising executable components) with any other processor enabling interaction and / or communication there-between. A user interface processor or generator is a known element comprising electronic circuitry or software or a combination of both for generating display images or portions thereof. A user interface comprises one or more display images enabling user interaction with a processor or other device.

[0037] Continuing with reference to FIG. 5, the computer system 510 also includes a system memory 530 coupled to the system bus 521 for storing information and instructions to be executed by processors 520. The system memory 530 may include computer readable storage media in the form of volatile and / or nonvolatile memory, such as read only memory (ROM) 531 and / or random-access memory (RAM) 532. The RAM 532 may include other dynamic storage device(s) (e.g., dynamic RAM, static RAM, and synchronous DRAM). The ROM 531 may include other static storage device(s) (e.g., programmable ROM, erasable PROM, and electrically erasable PROM). In addition, the system memory 530 may be used for storing temporary variables or other intermediate information during the execution of instructions by the processors 520. A basic input / output system 533 (BIOS) containing the basic routines that help to transfer information between elements within computer system 510, such as during start-up, may be stored in the ROM 531 . RAM 532 may contain data and / or program modules that are immediately accessible to and / or presently being operated on by the processors 520. System memory 530 may additionally include, for example, operating system 534, application programs 535, other program modules 536 and program data 537.

[0038] The computer system 510 also includes a disk controller 540 coupled to the system bus 521 to control one or more storage devices for storing information and instructions, such as a magnetic hard disk 541 and a removable media drive 542 (e.g., floppy disk drive, compact disc drive, tape drive, and / or solid-state drive). Storage devices may be added to the computer system 510 using an appropriate device interface (e.g., a small computer system interface (SCSI), integrated device electronics (IDE), Universal Serial Bus (USB), or FireWire).

[0039] The computer system 510 may also include a display controller 565 coupled to the system bus 521 to control a display or monitor 566, such as a cathode ray tube (CRT) or liquid crystal display (LCD), for displaying information to a computer user. The computer system includes an input interface 560 and one or more input devices, such as a keyboard 562 and a pointing device 561 , for interacting with a computer user and providing information to the processors 520. The pointing device 561 , for example, may be a mouse, a light pen, a trackball, or a pointing stick for communicating direction information and command selections to the processors 520 and for controlling cursor movement on the display 566. The display 566 may provide a touch screen interface which allows input to supplement or replace the communication of direction information and command selections by the pointing device 561 . In some embodiments, an augmented reality device 567 that is wearable by a user, may provide input / output functionality allowing a user to interact with both a physical and virtual world. The augmented reality device 567 is in communication with the display controller 565 and the user input interface 560 allowing a user to interact with virtual items generated in the augmented reality device 567 by the display controller 565. The user may also provide gestures that are detected by the augmented reality device 567 and transmitted to the user input interface 560 as input signals. Other inputs, such as inputs provided by sensor 563 may be provided and information communicated by the sensor 563 may be further processed. Sensor 563 may be an image capture device like an RGB camera or RGB camera or other sensor such as an infrared sensor, RADAR or LIDAR device.

[0040] The computer system 510 may perform a portion or all of the processing steps of embodiments of the invention in response to the processors 520 executing one or moresequences of one or more instructions contained in a memory, such as the system memory 530. Such instructions may be read into the system memory 530 from another computer readable medium, such as a magnetic hard disk 541 or a removable media drive 542. The magnetic hard disk 541 may contain one or more datastores and data files used by embodiments of the present invention. Datastore contents and data files may be encrypted to improve security. The processors 520 may also be employed in a multiprocessing arrangement to execute the one or more sequences of instructions contained in system memory 530. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions. Thus, embodiments are not limited to any specific combination of hardware circuitry and software.

[0041] As stated above, the computer system 510 may include at least one computer readable medium or memory for holding instructions programmed according to embodiments of the invention and for containing data structures, tables, records, or other data described herein. The term “computer readable medium” as used herein refers to any medium that participates in providing instructions to the processors 520 for execution. A computer readable medium may take many forms including, but not limited to, non- transitory, non-volatile media, volatile media, and transmission media. Non-limiting examples of non-volatile media include optical disks, solid state drives, magnetic disks, and magneto-optical disks, such as magnetic hard disk 541 or removable media drive 542. Non-limiting examples of volatile media include dynamic memory, such as system memory 530. Non-limiting examples of transmission media include coaxial cables, copper wire, and fiber optics, including the wires that make up the system bus 521 . Transmissionmedia may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0042] The computing environment 500 may further include the computer system 510 operating in a networked environment using logical connections to one or more remote computers, such as remote computing device 580. Remote computing device 580 may be a personal computer (laptop or desktop), a mobile device, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to computer system 510. When used in a networking environment, computer system 510 may include modem 572 for establishing communications over a network 571 , such as the Internet. Modem 572 may be connected to system bus 521 via user network interface 570, or via another appropriate mechanism.

[0043] Network 571 may be any network or system generally known in the art, including the Internet, an intranet, a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a direct connection or series of connections, a cellular telephone network, or any other network or medium capable of facilitating communication between computer system 510 and other computers (e.g., remote computing device 580). The network 571 may be wired, wireless or a combination thereof. Wired connections may be implemented using Ethernet, Universal Serial Bus (USB), RJ- 6, or any other wired connection generally known in the art. Wireless connections may be implemented using Wi-Fi, WiMAX, and Bluetooth, infrared, cellular networks, satellite or any other wireless connection methodology generally known in the art. Additionally, several networks may work alone or in communication with each other to facilitate communication in the network 571 .

[0044] An executable application, as used herein, comprises code or machine- readable instructions for conditioning the processor to implement predetermined functions, such as those of an operating system, a context data acquisition system or other information processing system, for example, in response to user command or input.

[0045] An executable procedure is a segment of code or machine-readable instruction, sub-routine, or other distinct section of code or portion of an executable application for performing one or more particular processes. These processes may include receiving input data and / or parameters, performing operations on received input data and / or performing functions in response to received input parameters, and providing resulting output data and / or parameters.

[0046] A graphical user interface (GUI), as used herein, comprises one or more display images, generated by a display processor and enabling user interaction with a processor or other device and associated data acquisition and processing functions. The GUI also includes an executable procedure or executable application. The executable procedure or executable application conditions the display processor to generate signals representing the GUI display images. These signals are supplied to a display device which displays the image for viewing by the user. The processor, under control of an executable procedure or executable application, manipulates the GUI display images in response to signals received from the input devices. In this way, the user may interact with the display image using the input devices, enabling user interaction with the processor or other device.

[0047] The functions and process steps herein may be performed automatically or wholly or partially in response to user command. An activity (including a step) performedautomatically is performed in response to one or more executable instructions or device operation without user direct initiation of the activity.

[0048] The system and processes of the figures are not exclusive. Other systems, processes and menus may be derived in accordance with the principles of the invention to accomplish the same objectives. Although this invention has been described with reference to particular embodiments, it is to be understood that the embodiments and variations shown and described herein are for illustration purposes only. Modifications to the current design may be implemented by those skilled in the art, without departing from the scope of the invention. As described herein, the various systems, subsystems, agents, managers and processes can be implemented using hardware components, software components, and / or combinations thereof.

Claims

CLAIMSWhat is claimed is:1 . A method for verifying activity in a single or plurality of image data comprising: capturing in an image capture device or sensor, a sequence of captured image data; identifying objects in the sequence of captured image data and providing the identified information to a semantic framework of an environment containing the objects in the sequence of captured image data to verify and validate anything that is expected in a scene or process is in fact being observed as expected; and generating additional context relating to an activity occurring in the sequence of captured image data.

2. The method of Claim 1 , further comprising: inferring an action occurring in the sequence of captured image data based on the semantic framework.

3. The method of Claim 1 , wherein the verified activity comprises a safety requirement.

4. The method of Claim 1 , wherein the verified activity comprises confirmation that a task has been performed.

5. The method of Claim 4, wherein confirmation that the task has been performed comprises:verifying that a required tool for performing the task is present in the sequence of captured image data.

6. The method of Claim 4, wherein confirmation that the task has been completed comprises: verifying that an expected entity is performing the task in the sequence of captured images.

7. The method of Claim 4, wherein confirmation that the task has been completed comprises: verifying that a minimum amount of time to perform the task has elapsed.

8. The method of Claim 1 , further comprising: creating a semantic template for the task by linking at least two of people performing the task, actions performed in the task, and objects present during the task.

9. The method of Claim 1 , further comprising: returning a notification indicating the verification status of the task.

10. The method of Claim 9, further comprising:' providing the notification to human machine interface such as an augmented reality (AR) application for display to a user.1 1 . The method of Claim 9, wherein the notification is passed back to a learning model.

12. A system for verifying activity in a singular or plurality of images or image-related data comprising: a computer processor; anda non-transitory computer memory in communication with the computer processor, the non-transitory computer memory storing instructions that when executed by the computer processor, cause the computer processor to perform the steps of: capturing in an image data capture device or sensor, a sequence of captured image data; identifying data in the sequence of captured image data and providing the identified information to a semantic framework of an environment containing the objects in the sequence of captured image data; and generating additional context relating to an activity occurring in the sequence of captured image data.

13. The system of Claim 12, the non-transitory computer memory further storing instructions that when executed by the computer processor, cause the computer processor to perform the steps of: inferring an action occurring in the sequence of captured image data based on the semantic framework.

14. The system of Claim 12, wherein the verified activity comprises a safety requirement.

15. The system of Claim 12, wherein the verified activity comprises confirmation that a task has been performed.

16. The system of Claim 15, wherein confirmation that the task has been performed comprises:in the computer processor, verifying that a required tool for performing the task is present in the sequence of captured images.

17. The system of Claim 15, wherein confirmation that the task has been completed comprises: in the computer processor, verifying that an expected entity is performing the task in the sequence of captured image data.

18. The system of Claim 15, wherein confirmation that the task has been completed comprises: in the computer processor, verifying that a minimum amount of time to perform the task has elapsed.

19. The system of Claim 12, the non-transitory computer memory further storing instructions that when executed by the computer processor, cause the computer processor to perform the steps of: creating a semantic template for the task by linking at least two of people performing the task, actions performed in the task, and objects present during the task.

20. The system of Claim 12, the non-transitory computer memory further storing instructions that when executed by the computer processor, cause the computer processor to perform the steps of: returning a notification indicating the verification status of the task.

Images