Automatic task sequence extraction from manuals for augmented reality\XR applications

Abstract

A method for producing instructions for user displays such as in an augmented reality (AR) environment includes dividing a page containing a plurality of visual and textual depictions into separate elements and images, each separate image files containing one of the visual depictions, extracting textual information from each of the separate images, linking the extracted textual information with the visual depiction in each separate image file, and ingesting the linked information into a semantic knowledge graph. In addition, the method may include querying the semantic knowledge base to retrieve information about a task relating to one of the separate image files and formatting the retrieved information into a visualized instruction for display as in an AR device. The visualized instruction may contain a graphical depiction and textual information linked from one of the separate image files and may be formatted to guide the user in how to perform the task.

Description

AUTOMATIC TASK SEQUENCE EXTRACTION FROM MANUALS FOR AUGMENTED REALITY / XR APPLICATIONSTECHNICAL FIELD

[0001] This application relates to augmented reality (AR) end user displays. More particularly, this application relates to automatically generating instructions for AR applications.BACKGROUND

[0002] In industry settings, such as manufacturing plants, factories, and warehouses, tasking is often created using a variety of data resources and trainings. These can be databases, human experiential information, graphic or spatial information, heuristics, or artifacts like manuals that come in text and illustrative form. Since decisions need to be made including all of these sources for the best outcomes, it either means human bias in memorizing too much information wherever possible or automating the joining of these resources into knowledge structures for human use that are faster, more reliable, and more complete.

[0003] Sometimes there are information formats that make knowledge generation more streamlined, such as extracting and aligning data from a database that already adheres to a formal schema. Other times, it is in formats such as drawings that must be viewed and interpreted to determine the correct ordering of steps and what to do in each one. Consider the style of IKEA (ikea.com) manuals which are designed for simplicity but often include several instructions per page combined to a single large image page - pages that if scanned would be single full images themselves. Now consider pictorial representations for complex industry setting activities. While it may be possible to stopwork at every step of assembling simple furniture, it is often a huge pain point to stop, pause, or set everything down to determine the next step from graphical groupings. This can lead to safety risk, lost time and money, potential damages depending on the task specifics. Also, the information is presented in 2D and will need to be translated into the 3D real-world space.SUMMARY

[0004] Embodiments described in this disclosure include a method for producing visual instruction in an augmented reality environment. The method includes dividing a page containing a plurality of visual depictions into a plurality of separate images, each separate image files containing one of the visual depictions, extracting textual information from each of the separate images, linking the extracted textual information with the visual depiction in each separate image file, and ingesting the linked information into a semantic knowledge graph (KG). In addition, the method may further include querying the semantic knowledge base to retrieve information about a task relating to one of the separate image files, and formatting the retrieved information into a visualized instruction for display such as in an augmented reality (AR) device.

[0005] The visualized instruction may contain a graphical depiction and textual information linked from one of the separate image files and may be formatted to guide the user in how to perform the task. The textual information may be extracted based on raster or pixels in the image or by extracting the textual information from a structured source in the separate image file. For example, the structured source may be contained in a portable document format (PDF) file. The original information may be contained in aninstruction manual created by an original equipment manufacturer (OEM). A portion of the textual information comprises numerical representations corresponding to an order of operation for a step for completing a task.

[0006] In some embodiments of this disclosure a system for producing visual instruction in an augmented reality environment includes a computer processor and a non-transitory computer memory in communication with the computer processor, the non- transitory computer memory storing instructions that when executed by the computer processor cause the computer processor to perform the steps of dividing a page containing a plurality of visual depictions into a plurality of separate images, each separate image files containing one of the visual depictions, extracting textual information from each of the separate images, linking the extracted textual information with the visual depiction in each separate image file, and ingesting the linked information into a semantic knowledge graph (KG). Additional instructions may cause the computer processor to perform the steps of querying the semantic knowledge base to retrieve information about a task relating to one of the separate image files and formatting the retrieved information into a visualized instruction for display in an augmented reality (AR) device. A graphical depiction and textual information linked from one of the separate image files may be displayed to guide the user in how to perform the task. The textual information may be extracted based on raster or pixels in the image or from a structured source in the separate image file Wherein the structured source is contained in a portable document format (PDF) file. The original source page may be contained in an instruction manual created by an original equipment manufacturer (OEM). The textual information mayinclude numerical representations corresponding to an order of operation for a step for completing a task.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The foregoing and other aspects of the present invention are best understood from the following detailed description when read in connection with the accompanying drawings. For the purpose of illustrating the invention, there is shown in the drawings embodiments that are presently preferred, it being understood, however, that the invention is not limited to the specific instrumentalities disclosed. Included in the drawings are the following Figures:

[0008] FIG. 1 is an illustration of a system and method for extracting instructions from a physical or digital document for use as an input for AR application according to aspects of embodiments of this disclosure.

[0009] FIG. 2 is a process flow diagram for extracting visual instructions from a document according to aspects of embodiments of this disclosure.

[0010] FIG. 3 is an illustration of providing AR based instructions according to aspects of embodiments of this disclosure.

[0011] FIG. 4 is a block diagram of a computer system for implementing systems and methods according to embodiments of this disclosure.DETAILED DESCRIPTION

[0012] This invention is an image processing technique and data processing pipeline; it creates a workflow plus automated data service for automatically extracting IKEA-style images from larger combined images and instruction manuals to be able to ingest theminto a semantic graph connected to an extended reality / augmented reality (XR / AR) content device(s).

[0013] FIG. 1 is an illustration of an example process for extracting visual and textual information from documents of varying formats to provide visual instructions to an XR / AR device. Using a series of existing manuals 100, in this case the proposed type is the well- known IKEA (ikea.com) furniture assembly manuals that embody characteristics of instructional artifacts available for more complex industry processes. While this may be substituted for a similar image set, IKEA instruction manuals themselves are useful in that they exist as PDF (format common to industry). The pages of such instruction manuals 100 may have differing amounts of instructions per page each that exist as a PDF or single image rendering. Each page may comprise multiple images 100a - 10Of that include instructions 101 d, 101c and the layout and scale of images differs for each page or set. These instructions may include numeric indicators 102d, 102c, which convey the order in which the steps should be performed. Conventionally these images 100a - 10Of would need to be separated and tagged prior to graph ingestion. This may possibly be achieved with an image processing algorithm that tags the features in the images 100a - 10Of. This type of tagging is helpful but does not yet lend itself to determining 1) the different portions of an image 100a - 10Of that represents each instruction 101d, 101c, 2) the order in which each instruction should take place 102d, 102c or 3) extraction and understanding of textual information from the images.

[0014] Presently, there are data management systems for these types of instructions; for example, there are well-managed databases that will be used to serve up sets of instructions in PDF or image form providing the order is known already known or hasbeen tagged by a human from the start whereas this invention determines this order and context where it is not yet provided. Some scenarios have instructions which are well- structured text and ordered such that existing programming libraries can be used to extract text and images. A need remains in cases that needed solving but where is it not easy to pull apart pages of multiple instructions in raster / pixel form, deduce the text, and then link everything together for it to be stored as semantic knowledge 103. As far as understood, image tagging and segmentation will only go so far alone, especially as there are not consistent groupings and placements of instructions between manual pages. Traditionally, this would all need to be done manually to be interpretable by XR / AR devices, which is the end goal for interpreting the initial instructions.

[0015] Embodiments of this disclosure extract numeric data 102d, 102c and textual instructions 101 d, 101c and ingest the information into a semantic knowledge graph 103. The stored semantic knowledge provides inputs to an AR application 104, which provides a visualization 106 of the instructions via an AR device 105. The AR device 105 may be a wearable headset, a tablet, mobile phone, or any other device capable of displaying contextual information to a user in a real-world scenario. The information may be used to training workers in a virtual reality environment. In an AR application, the information may be used to provide instructions to a human using the AR application.

[0016] Technical features introduced by the embodiments of this disclosure include components that achieve a new framework that provides the following:

[0017] Document Processing: First, an improved document or image processing technique of instruction creation and ordering over current image extractions from PDFcoding tools that are aimed at extracting single text, image, and tables from existing documents. The improved process segments single images into different instructions.

[0018] Semantic Task Generation: Next, the solutions of embodiments herein aim to consolidate information into an existing knowledge graph (KG) construct as there will be additional use cases and data in the future, which need to be compatible and interchangeable for any query or device interface to which they are passed.

[0019] Use in Augmented Reality: Finally, AR is incorporated to provide an interactive, intuitive and potentially hands-free system for worker assistance, step validation, and overall visualization without leaving the work scene to resolve questions. Other end-user solutions can display this data as well.

[0020] This is achieved by automatically extracting operational sequences from instruction manuals using machine vision and visualizing them in an XR / AR environment. This visualization is derived from a generalized script that triggers a query to a Knowledge Graph (KG) for the next items to be displayed in the XR / AR field of view (FOV). By way of example, the generalized script may be implemented in UNITY, however, the invention is not so limited. Other utilities may be implemented to query the KG. The AR device may use MICROSOFT HOLOLENS 2 by MICROSOFT CORPORATION, but the system is extendable and reusable for future XR / AR user devices and interfaces as well one nonlimiting example of which is the virtual reality (VR) headset such as an OCULUS QUEST by REALITY LABS.

[0021] FIG. 3 provides an overview of the process of extracting information from source documents and generating AR instructions. The first step is to automatically extract and collect set of images 302 from representative instruction manuals 301. Eachpage (single image file) is divided into multiple image files 302 (depending how many instructions on a given page) using basic image processing and / or semantic segmentation. Any possible text 303 can be extracted with these from raster / pixel form but may also be extracted from other image pixels or locations, that is, not always structured PDFs. After that, the images will have had resource identifiers (like page number, for instance) and instruction numbers extracted and linked to the instruction illustration. This collection can be stored in a reused semantic KG 304 that was created from past projects, which is intermediate to passing semantic query results to a Unity game engine or other utility from where they may be displayed 306 in HoloLens 2 or another AR user device 305. According to an embodiment, the system uses a JSON file format to pass information from KG results to Unity.

[0022] A single reusable script may be used that will automate pulling the next, previous, or a specific instruction number and automatically loading and setting this data up to be the new information displayed in the AR device. This is a generalized enough algorithm to be extensible for use in a more complex Unity user interface (Ul) in the future.

[0023] Programmatic libraries are used for extracting structured information from PDFs; embodiments of the present invention incorporate an image processing upgrade because current image extractions from PDF coding tools are aimed at extracting single text, image, and tables from existing documents and do not address the need to be able to segment single images into different instructions and pull vector text from raster / pixel- based images of text. Using machine learning (ML) classifiers, image element detection, and semantic hierarchies, semantic linking is achievable. However, in some cases this means the sematic structure for that use case must exist in order to align data to.Embodiments of this disclosure do not look for hierarchies for identification, but rather surrounding context for creating ordered instructions (e.g., tags to order steps not just place in a hierarchy), and this typically requires ontologies and semantic models not previously used in combination with the image tagging. In some cases, there are domainspecific implementations of image tags being linked to an ontology or in retrieval systems. More commonly though, these implementations are extracting objects detected in images. Embodiments of this disclosure also extract step ordering information into tagform. These data alignments and step ordering are paired with automated ingestion for use in augmented reality devices.

[0024] An automation pipeline provides a way to create image-sourced instructions and then use them through a knowledge graph in Unity. This leads to faster implementation times for future projects. It is useful in scenarios where there is a need for image analysis in the form of splitting pixel-based images into sub-images and extracting information of varying types. It further provides a general benefit in automation allowing smarter worker assistance technologies. From a broad perspective, information gaps will be bridged for individuals of differing experience or knowledge levels (or accessibility for those fluent in different languages) working in factories. This leads to less risk, less damage, and more safety for all involved. Further, learning impaired individuals may be included in the workforce by leveraging the benefits of digital guidance. Embodiments of this disclosure pave the way to begin linking XR / AR with real user feedback back to KG information for overall quality assurance (e.g., real-time updates, better data quality, etc.). Businesses may use this technology to digitalize existing static paper-based manuals into XR / AR instructions thereby connecting them to productlifecycle management (PLM), digital thread and digital twins of their processes and products and ensuring that their instructions always use the latest information.

[0025] FIG. 4 illustrates an exemplary computing environment 400 within which embodiments of the invention may be implemented. Computers and computing environments, such as computer system 410 and computing environment 400, are known to those of skill in the art and thus are described briefly here.

[0026] As shown in FIG. 4, the computer system 410 may include a communication mechanism such as a system bus 421 or other communication mechanism for communicating information within the computer system 410. The computer system 410 further includes one or more processors 420 coupled with the system bus 421 for processing the information.

[0027] The processors 420 may include one or more central processing units (CPUs), graphical processing units (GPUs), or any other processor known in the art. More generally, a processor as used herein is a device for executing machine-readable instructions stored on a computer readable medium, for performing tasks and may comprise any one or combination of, hardware and firmware. A processor may also comprise memory storing machine-readable instructions executable for performing tasks. A processor acts upon information by manipulating, analyzing, modifying, converting or transmitting information for use by an executable procedure or an information device, and / or by routing the information to an output device. A processor may use or comprise the capabilities of a computer, controller or microprocessor, for example, and be conditioned using executable instructions to perform special purpose functions not performed by a general-purpose computer. A processor may be coupled (electricallyand / or as comprising executable components) with any other processor enabling interaction and / or communication there-between. A user interface processor or generator is a known element comprising electronic circuitry or software or a combination of both for generating display images or portions thereof. A user interface comprises one or more display images enabling user interaction with a processor or other device.

[0028] Continuing with reference to FIG. 4, the computer system 410 also includes a system memory 430 coupled to the system bus 421 for storing information and instructions to be executed by processors 420. The system memory 430 may include computer readable storage media in the form of volatile and / or nonvolatile memory, such as read only memory (ROM) 431 and / or random-access memory (RAM) 432. The RAM 432 may include other dynamic storage device(s) (e.g., dynamic RAM, static RAM, and synchronous DRAM). The ROM 431 may include other static storage device(s) (e.g., programmable ROM, erasable PROM, and electrically erasable PROM). In addition, the system memory 430 may be used for storing temporary variables or other intermediate information during the execution of instructions by the processors 420. A basic input / output system 433 (BIOS) containing the basic routines that help to transfer information between elements within computer system 410, such as during start-up, may be stored in the ROM 431 . RAM 432 may contain data and / or program modules that are immediately accessible to and / or presently being operated on by the processors 420. System memory 430 may additionally include, for example, operating system 434, application programs 435, other program modules 436 and program data 437.

[0029] The computer system 410 also includes a disk controller 440 coupled to the system bus 421 to control one or more storage devices for storing information andinstructions, such as a magnetic hard disk 441 and a removable media drive 442 (e.g., floppy disk drive, compact disc drive, tape drive, and / or solid-state drive). Storage devices may be added to the computer system 410 using an appropriate device interface (e.g., a small computer system interface (SCSI), integrated device electronics (IDE), Universal Serial Bus (USB), or FireWire).

[0030] The computer system 410 may also include a display controller 465 coupled to the system bus 421 to control a display or monitor 466, such as a cathode ray tube (CRT) or liquid crystal display (LCD), for displaying information to a computer user. The computer system includes an input interface 460 and one or more input devices, such as a keyboard 462 and a pointing device 461 , for interacting with a computer user and providing information to the processors 420. The pointing device 461 , for example, may be a mouse, a light pen, a trackball, or a pointing stick for communicating direction information and command selections to the processors 420 and for controlling cursor movement on the display 466. The display 466 may provide a touch screen interface which allows input to supplement or replace the communication of direction information and command selections by the pointing device 461. In some embodiments, an augmented reality device 467 that is wearable by a user, may provide input / output functionality allowing a user to interact with both a physical and virtual world. The augmented reality device 467 is in communication with the display controller 465 and the user input interface 460 allowing a user to interact with virtual items generated in the augmented reality device 467 by the display controller 465. The user may also provide gestures that are detected by the augmented reality device 467 and transmitted to the user input interface 460 as input signals. Augmented reality device 467 may also beconfigured to receive information from a user such as alphanumeric input or through gestures and provide the user's information as input to the computer system 410.

[0031] The computer system 410 may perform a portion or all of the processing steps of embodiments of the invention in response to the processors 420 executing one or more sequences of one or more instructions contained in a memory, such as the system memory 430. Such instructions may be read into the system memory 430 from another computer readable medium, such as a magnetic hard disk 441 or a removable media drive 442. The magnetic hard disk 441 may contain one or more datastores and data files used by embodiments of the present invention. Datastore contents and data files may be encrypted to improve security. The processors 420 may also be employed in a multiprocessing arrangement to execute the one or more sequences of instructions contained in system memory 430. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions. Thus, embodiments are not limited to any specific combination of hardware circuitry and software.

[0032] As stated above, the computer system 410 may include at least one computer readable medium or memory for holding instructions programmed according to embodiments of the invention and for containing data structures, tables, records, or other data described herein. The term “computer readable medium” as used herein refers to any medium that participates in providing instructions to the processors 420 for execution. A computer readable medium may take many forms including, but not limited to, non- transitory, non-volatile media, volatile media, and transmission media. Non-limiting examples of non-volatile media include optical disks, solid state drives, magnetic disks, and magneto-optical disks, such as magnetic hard disk 441 or removable media drive442. Non-limiting examples of volatile media include dynamic memory, such as system memory 430. Non-limiting examples of transmission media include coaxial cables, copper wire, and fiber optics, including the wires that make up the system bus 421 . Transmission media may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0033] The computing environment 400 may further include the computer system 410 operating in a networked environment using logical connections to one or more remote computers, such as remote computing device 480. Remote computing device 480 may be a personal computer (laptop or desktop), a mobile device, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to computer system 410. When used in a networking environment, computer system 410 may include modem 472 for establishing communications over a network 471 , such as the Internet. Modem 472 may be connected to system bus 421 via user network interface 470, or via another appropriate mechanism.

[0034] Network 471 may be any network or system generally known in the art, including the Internet, an intranet, a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a direct connection or series of connections, a cellular telephone network, or any other network or medium capable of facilitating communication between computer system 410 and other computers (e.g., remote computing device 480). The network 471 may be wired, wireless or a combination thereof. Wired connections may be implemented using Ethernet, Universal Serial Bus (USB), RJ- 6, or any other wired connection generally known in the art. Wireless connections may be implemented using Wi-Fi, WiMAX, and Bluetooth, infrared, cellular networks, satellite orany other wireless connection methodology generally known in the art. Additionally, several networks may work alone or in communication with each other to facilitate communication in the network 471.

[0035] An executable application, as used herein, comprises code or machine- readable instructions for conditioning the processor to implement predetermined functions, such as those of an operating system, a context data acquisition system or other information processing system, for example, in response to user command or input. An executable procedure is a segment of code or machine-readable instruction, subroutine, or other distinct section of code or portion of an executable application for performing one or more particular processes. These processes may include receiving input data and / or parameters, performing operations on received input data and / or performing functions in response to received input parameters, and providing resulting output data and / or parameters.

[0036] A graphical user interface (GUI), as used herein, comprises one or more display images, generated by a display processor and enabling user interaction with a processor or other device and associated data acquisition and processing functions. The GUI also includes an executable procedure or executable application. The executable procedure or executable application conditions the display processor to generate signals representing the GUI display images. These signals are supplied to a display device which displays the image for viewing by the user. The processor, under control of an executable procedure or executable application, manipulates the GUI display images in response to signals received from the input devices. In this way, the user may interactwith the display image using the input devices, enabling user interaction with the processor or other device.

[0037] The functions and process steps herein may be performed automatically or wholly or partially in response to user command. An activity (including a step) performed automatically is performed in response to one or more executable instructions or device operation without user direct initiation of the activity.

[0038] The system and processes of the figures are not exclusive. Other systems, processes and menus may be derived in accordance with the principles of the invention to accomplish the same objectives. Although this invention has been described with reference to particular embodiments, it is to be understood that the embodiments and variations shown and described herein are for illustration purposes only. Modifications to the current design may be implemented by those skilled in the art, without departing from the scope of the invention. As described herein, the various systems, subsystems, agents, managers and processes can be implemented using hardware components, software components, and / or combinations thereof.

Claims

CLAIMSWhat is claimed is:1 . A method for producing ordered instructions for display in user devices such as in an augmented reality or virtual reality environment comprising: dividing a page containing a plurality of visual depictions into a plurality of separate images, each separate image files containing one of the visual depictions; extracting textual information from each of the separate images; linking the extracted textual information with the visual depiction in each separate image file; and ingesting the linked information into a semantic knowledge graph (KG).

2. The method of Claim 1 , further comprising: querying the semantic knowledge base to retrieve information about a task relating to one of the separate image files and related contextual information; and formatting the retrieved information into a visualized instruction for display in an augmented reality (AR) device or other user interface.

3. The method of Claim 2, wherein the visualized instruction comprises a graphical depiction and textual information linked from one of the separate image files.

4. The method of Claim 3, wherein the visualized instruction is formatted to guide the user in how to perform the task.

5. The method of Claim 1 , further comprising: extracting the textual information based on raster or pixels in the image.

6. The method of Claim 1 , further comprising: extracting the textual information from a structured source in the separate image file.

7. The method of Claim 6, wherein the structured source is contained in a portable document format (PDF) file.

8. The method of Claim 1 , wherein the page is contained in an instruction manual created by an original equipment manufacturer (OEM).

9. The method of Claim 1 , wherein the textual information comprises numerical representations.

10. The method of Claim 9, wherein the numerical representations correspond to an order of operation for a step for completing a task.

11. A system for producing visual instruction in an augmented reality environment comprising: a computer processor; and a non-transitory computer memory in communication with the computer processor, the non-transitory computer memory storing instructions that when executed by the computer processor cause the computer processor to perform the steps of: dividing a page containing a plurality of visual depictions into a plurality of separate images, each separate image files containing one of the visual depictions; extracting textual information from each of the separate images;linking the extracted textual information with the visual depiction in each separate image file; and ingesting the linked information into a semantic knowledge graph (KG).

12. The system of Claim 11 , the non-transitory computer memory further storing instructions that when executed by the computer processor cause the computer processor to perform the steps of: querying the semantic knowledge base to retrieve information about a task relating to one of the separate image files; and formatting the retrieved information into a visualized instruction for display in an augmented reality (AR) device.

13. The system of Claim 12, wherein the visualized instruction comprises a graphical depiction and textual information linked from one of the separate image files.

14. The system of Claim 13, wherein the visualized instruction is formatted to guide the user in how to perform the task.

15. The system of Claim 11 , the non-transitory computer memory further storing instructions that when executed by the computer processor cause the computer processor to perform the steps of: extracting the textual information based on raster or pixels in the image.

16. The system of Claim 11 , the non-transitory computer memory further storing instructions that when executed by the computer processor cause the computer processor to perform the steps of:extracting the textual information from a structured source in the separate image file.

17. The system of Claim 16, wherein the structured source is contained in a portable document format (PDF) file.

18. The system of Claim 11 , wherein the page is contained in an instruction manual created by an original equipment manufacturer (OEM).

19. The system of Claim 11 , wherein the textual information comprises numerical representations.

20. The system of Claim 19, wherein the numerical representations correspond to an order of operation for a step for completing a task.

Images