Method, media, and system for simultaneous capture of digital and physical notes using augmented reality

Abstract

A method includes receiving digital item location data pertaining to at least one digital item and receiving image data from a camera, wherein the image data includes at least one physical item and the digital item location data is mapped to the image data. The method also includes outputting an augmented reality view including the at least one physical item and the at least one digital item, wherein the at least one digital item differs in how it is depicted in augmented reality. The method further includes receiving an input to capture the augmented reality view with the at least one physical item and the at least one digital item each identified.

Description

METHOD, MEDIA, AND SYSTEM FOR SIMULTANEOUS CAPTURE OF DIGITAL AND PHYSICAL NOTES USING AUGMENTED REALITYTechnical Field

[0001] The present disclosure relates to a non-transitory computer-readable medium, a system, and a method for simultaneous capture of digital and physical notes using augmented reality.Background

[0002] Paper notes have been broadly used in recording, sharing, and communicating ideas and information. For example, during a collaboration session, such as, a brainstorming session, participants write down ideas on repositionable paper notes, whiteboard, or paper, and then share with one another. In addition, people commonly use notes throughout the day to memorialize information or content which the individual does not want to forget. As additional examples, people frequently use notes as reminders of actions or events to take in the future, such as making a telephone call, revising a document or filling out a time sheet.

[0003] Software programs currently exist which permit computer users to create a software-based note in a digital form and to utilize the digital note within a computing environment. For example, a computer user may create a digital note and attach the digital note to an electronic document, a desktop or electronic workspace presented by the computing environment.Summary

[0004] In one embodiment, at least one non-transitory computer-readable medium storing instructions that, when executed, configure at least one processor to receive digital item location data pertaining to at least one digital item and receive image data from a camera, wherein the image data includes at least one physical item and the digital item location data is mapped to the image data. The at least one non-transitory computer-readable medium stores further instructions to output an augmented reality view including the at least one physical item and the at least one digital item, wherein the at least one digital item differs in how it is depicted in augmented reality and receive an input to capture the augmented reality view with the at least one physical item and the at least one digital item each identified.

[0005] In another embodiment, at least one system includes at least one computing device comprising one or more processors and at least one memory coupled to at least one of the one or more processors, wherein the at least one memory comprises instructions that configure the at least one computing device to receive digital item location data pertaining to at least one digital item and receive image data from a camera, wherein the image data includes at least one physical item and the digital item location data is mapped to the image data. The at least one system includes further instructions to output an augmented reality view including the at least onephysical item and the at least one digital item, wherein the at least one digital item differs in how it is depicted in augmented reality and receive an input to capture the augmented reality view with the at least one physical item and the at least one digital item each identified.

[0006] In yet another embodiment, a method includes receiving digital item location data pertaining to at least one digital item and receiving image data from a camera, wherein the image data includes at least one physical item and the digital item location data is mapped to the image data. The method also includes outputting an augmented reality view including the at least one physical item and the at least one digital item, wherein the at least one digital item differs in how it is depicted in augmented reality. The method further includes receiving an input to capture the augmented reality view with the at least one physical item and the at least one digital item each identified.Brief Description of the Drawings

[0007] FIG. 1 illustrates a schematic diagram of an environment for recognizing at least one physical item, according to an embodiment of the present disclosure;

[0008] FIG. 2 illustrates a schematic block diagram of the mobile device of the environment of FIG. 1, according to an embodiment of the present disclosure;

[0009] FIG. 3 illustrates a schematic block diagram of at least one system for simultaneous capture of digital and physical items using augmented reality, according to an embodiment of the present disclosure;

[0010] FIG. 4 illustrates non-limiting examples of the at least one physical item in various configurations, according to an embodiment of the present disclosure;

[0011] FIG. 5 illustrates non-limiting examples of the at least one digital item in various configurations, according to an embodiment of the present disclosure;

[0012] FIG. 6 illustrates an exemplary augmented reality view including at least one physical item and at least one digital item, according to an embodiment of the present disclosure;

[0013] FIG. 7A illustrates an exemplary augmented reality view including at least one physical item, according to an embodiment of the present disclosure;

[0014] FIG. 7B illustrates another exemplary augmented reality view including at least one physical item and at least one digital item, according to an embodiment of the present disclosure;

[0015] FIG. 8 illustrates a flowchart of a method for simultaneous capture of digital and physical items using augmented reality, according to an embodiment of the present disclosure;

[0016] FIG. 9 illustrates a flowchart of a process for simultaneous capture of digital and physical items using augmented reality, according to an embodiment of the present disclosure;

[0017] FIG. 10 illustrates a non-limiting example of a user interface displayed on a screen of a mobile device, according to an embodiment of the present disclosure;

[0018] FIG. 11 illustrates another non-limiting example of a user interface displayed on a screen of a mobile device, according to an embodiment of the present disclosure; and

[0019] FIG. 12 illustrates a flowchart of a process for simultaneous capture of digital and physical items using augmented reality with additional ability to suggest multiple actions based on user- defined actions, according to an embodiment of the present disclosure.Detailed Description

[0020] As used herein, the term “physical item” generally refers to objects with a general boundary and recognizable content. The physical items may include objects resulting after people write, draw, or enter via other type of inputs on the objects, for example, paper, white board, or other objects accepting the inputs. By way of examples, the physical notes may include hand-written repositionable paper notes, paper, or film, whiteboard with drawings, posters, and signs. In some cases, the physical items may be generated using digital means, e.g., printing onto printable repositionable paper notes or printed document. In some cases, one object may include several notes. For example, several ideas may be written on a piece of poster paper or a whiteboard. The physical items may be two-dimensional or three-dimensional. The physical items may have various shapes and sizes. For example, a physical item may be a 3 inches x 3 inches note; a physical note may be a 26 inches x 39 inches poster; and a physical item may be a triangular metal sign. In some cases, the physical notes have known shapes and / or sizes. The term “physical item” may be interchangeably referred to as the “physical note”.

[0021] As used herein, the term “digital item” generally refers to digital objects with information and / or ideas. The digital items may be generated using digital inputs. Digital inputs may include, for example, keyboards, touch screens, digital cameras, digital recording devices, stylus, digital pens, or the like. In some cases, the digital items may be representative of the physical items.

[0022] As used herein, the term “multi-modality notes” generally refers to combination of the physical items and the digital items in a single environment. In such case, an environment is said to have the multi -modality notes. The physical items and the digital items may be collectively referred to as “the items” herein.

[0023] As used herein, the term “workspace” generally refers to a location or an area in a physical space in either two-dimensional or three-dimensional space. The workspace may refer to an office space or a room, in which users work and / or organize collaborative sessions such as meetings and the like and make one or more notes. In some examples, the workspace may refer to a notebook page, in which user(s) make one or more notes.

[0024] Physical notes, such as paper notes have traditionally served as a versatile and vital tool for recording, sharing, and communicating ideas and information. Their tactile nature allows them to be easily manipulated, moved, and referenced, thereby making them effective in bothindividual and collaborative settings. For example, during collaborative sessions, such as brainstorming meetings, workshops, or team strategy sessions, participants often use physical notes to capture their thoughts, ideas, or contributions in real time. Afterward, participants share these written ideas with one another, allowing for enhanced communication and collective decision-making. Beyond collaboration, physical notes also play an important role in an individual’s daily organization and memory. People frequently choose physical notes to document important information, ideas, or observations they want to ensure they retain. This may include recording key points from a meeting, noting a sudden thought, or preserving reminders about concepts they intend to revisit. Physical notes also act as tangible, physical reminders of tasks, appointments, or other actionable operations in one's personal or professional life. For example, someone may use a physical note to remind themselves to make a phone call, revise a document by a specific date, schedule a meeting, or complete a timesheet. This ability to physically write and visually display tasks allows users to stay organized, prioritize responsibilities, and avoid forgetting important deadlines.

[0025] Currently, many software programs exist that allow users to create, store, and manage notes in a digital format, replacing or complementing traditional paper-based or physical methods. These software-based notes provide users with the ability to record and organize their thoughts, ideas, or important information conveniently within a computing environment. Such notes are highly versatile, allowing users to input various types of information, including text, multimedia, links, or even attachments. Unlike physical notes, digital notes may be easily searched, edited, shared, and backed up, making them practical and efficient for both personal and professional use. Another key advantage of digital notes is their integration with other computing tools and electronic workspaces. For instance, a user might create a digital note and attach it to a related electronic document or a specific task within a software application. This feature allows for seamless organization by connecting digital notes directly to projects, presentations, reports, or other digital resources within the computing environment.

[0026] Thus, the method of the present disclosure is configured to advantageously perform simultaneous capture of digital and physical notes using augmented reality. The method of the present disclosure allows displaying digital notes based on physical notes that may act as an anchor, thereby allowing users to seamlessly integrate and organize information from both physical and digital sources. The method of the present disclosure also supports multiple modalities (physical, digital, and mixed-modality notes), thereby enabling users to simultaneously capture and interact with a wider range of notes in various formats. The method of the present disclosure further allows users to view their physical workspace and interact with notes in real time using an augmented reality device. Embodiments herein use the augmented reality device recognizes physical notes and prompts the user to capture them, such that even after a physical note is captured and transformed into digital note, the original note remains asa spatial anchor, preserving its physical reference point within the user's environment and allowing the digital note to be displayed or overlaid in the physical world at the correct location even if the user’s device has moved.

[0027] Referring now to figures, FIG. 1 illustrates a schematic diagram of an environment 100 for recognizing at least one physical item 122, according to an embodiment of the present disclosure. In the non-limiting example of FIG. 1, the environment 100 may include one or more mobile devices 115, in particular one mobile device 115 to capture and recognize the at least one physical note 122 from a physical workspace 120. The mobile device 115 is associated with a user 126. As described herein, the mobile device 115 may provide an execution environment for one or more software applications that may efficiently capture, and extract note content from at least one physical item 122. In one non-limiting example, the at least one physical item 122 may be the result of a collaborative brainstorming session having multiple participants.

[0028] In some embodiments, the mobile device 115 may be a mobile phone. In other embodiments, the mobile device 115 may be a tablet computer, a personal digital assistant (PDA), a laptop computer, a media player, an e-book reader, a wearable computing device (e.g., a watch, eyewear, a glove), or any other type of mobile or non-mobile computing device suitable for performing the techniques described herein.

[0029] In some embodiments, the mobile device 115 may include one or more software executing on the mobile device 115. The mobile device 115 and software executing thereon are configured to perform a variety of note-related operations, including automated creation of at least one digital item representative of the at least one physical item 122 of the physical workspace 120. The mobile device 115, and the software executing thereon, may provide a platform for creating and manipulating the at least one digital item representative of the at least one physical item 122. In some examples, the mobile device 115 may be configured to recognize the at least one physical item 122 by determining a boundary of the at least one physical item 122. After the at least one physical item 122 is recognized, the mobile device 115 may extract the corresponding content of the at least one physical item 122, where the content may be a visual information of the at least one physical item 122.

[0030] The mobile device 115 may implement techniques for automated detection and recognition of the at least one physical item 122 and extraction of information, content or other characteristics associated with the at least one physical item 122. For example, the mobile device 115 may allow the user 126 fine grain control over techniques used by the mobile device 115 to detect and recognize the at least one physical item 122. As one example, the mobile device 115 may allow the user 126 to select between marker-based detection techniques in which the at least one physical item 122 includes a physical fiducial mark on the surface of the physical item and / or non-marker-based techniques in which no fiducial mark is used.

[0031] In some embodiments, the mobile device 115 may include, among other components, an image capture device 118 and a presentation device 128. The mobile device 115 may be configured to process image data captured by the image capture device 118 to detect and recognize the at least one physical item 122 positioned within the physical workspace 120. The image capture device 118 may be a camera or any other suitable component configured to capture image data representative of the physical workspace 120 and the at least one physical item 122 positioned in the physical workspace 120. In other words, the image data may capture one or more visual representations of the physical workspace 120, having the at least one physical item 122. Although discussed as the camera of the mobile device 115, the image capture device 118 may comprise other components capable of capturing the image data, such as, a video recorder, an infrared camera, a Charge Coupled Device (CCD) array, a laser scanner, or the like, which may correspond to in some embodiments to image capture device 118 and / or input device 306 in FIG. 3. The image data includes the at least one physical item 122. The image data may further include at least one of an image, a video, a sequence of images (i.e., multiple images taken within a time period and / or with an order), a collection of images, image portion(s), and / or the like.

[0032] The presentation device 128 may include, but not limited to, an electronically addressable display, such as a liquid crystal display (LCD) or other type of display device capable of use with the mobile device 115 to display the image data. In some embodiments, the presentation device 128 may be an input / output (I / O) device and / or a display / output device(s) 304 (shown in FIG. 3). In some embodiments, the mobile device 115 may generate content to display on the presentation device 128 for the at least one physical item 122 in a variety of formats, for example, a list, grouped in rows and / or column, a flow diagram, or the like. The mobile device115 may, in some cases, communicate display information for presentation by other devices, such as a tablet computer, a projector, an electronic billboard or other external device.

[0033] In addition, the mobile device 115 may provide the user 126 with an improved electronic environment for generating and manipulating corresponding the at least one digital item representative of the at least one physical item 122. By way of non-limiting example, the mobile device 115 may provide mechanisms allowing the user 126 to easily add new digital items to, edit the at least one digital item within, and / or delete the at least one digital item representative of the at least one physical item 122 created during brainstorming activity associated with the physical workspace 120.

[0034] The environment 100 further includes a cloud server 112, a computer system 114, and another mobile device 116. The cloud server 112, the computer system 114, and the other mobile device116 may be communicatively coupled with the mobile device 115 using a network (not shown). In some embodiments, the mobile device 115 may provide functionality by which the user 126 is able to record and manage relationships between the at least one physical item 122. In someembodiments, the mobile device 115 may provide functionality by which the user 126 is able to export the at least one digital note to other systems, such as cloud-based repositories (e.g., cloud server 112) and / or other computing devices (e.g., computer system 114 and / or mobile device 116).

[0035] FIG. 2 illustrates a schematic block diagram of the mobile device 115 of the environment 100 of FIG. 1, according to an embodiment of the present disclosure. In this embodiment, the mobile device 115 may include various hardware components that provide core functionality for operation of the mobile device 115, some or all of which may correspond to the computing hardware components depicted in FIG. 3. The mobile device 115 may include one or more processors 170. In some embodiments, the one or more processors 170 may be embodied in a number of different ways. For example, the one or more processors 170 may be embodied as various processing means, such as one or more of a microprocessor or other processing elements, a coprocessor, or various other computing or processing devices, including integrated circuits, such as, e.g., an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or the like.

[0036] As such, whether configured by hardware or by a combination of hardware and software, the one or more processors 170 may represent an entity (e.g., physically embodied in circuitry - in the form of a processing circuitry) capable of performing operations according to some embodiments while configured accordingly. Thus, for example, when the one or more processors 170 are embodied as an executor of software instructions, the instructions may specifically configure the one or more processors 170 to perform the operations described herein. Alternatively, as another example, when the one or more processors 170 are embodied as the ASIC, FPGA, or the like, the one or more processors 170 may have specifically configured hardware for conducting the operations described herein.

[0037] The one or more processors 170 are configured to operate according to executable instructions (i.e., program code), typically stored in a computer-readable medium or data storage 168 such as a static, random-access memory (SRAM) device or a Flash memory device. The mobile device 115 further includes the input / output (I / O) interface 176. The I / O interface 176 may include one or more devices, such as a keyboard, camera button, power button, volume button, home button, back button, menu button, or the presentation device 128 as described in FIG. 1.

[0038] The mobile device further includes a transmitter 172 and a receiver 174. The transmitter 172 and the receiver 174 are configured to provide wireless communication with other devices, such as the cloud server 112, the computer system 114, or the other mobile device 116 as described in FIG.1, via a wireless communication interface (not shown), such as but not limited to high-frequency radio frequency (RF) signals. The mobile device 115 may include additional discrete digital logic or analog circuitry (not shown), and / or may be embodied in any suitabletype of device such as a smartphone, laptop, tablet, wearable device (such as wearable augmented reality glasses), and the like.

[0039] The mobile device 115 further includes an operating system 164. The operating system 164 may execute on the one or more processors 170 and provide an operating environment for one or more user applications 177 (commonly referred to “apps”). In some embodiments, the one or more user applications 177 include a note management application 178 for managing the at least one physical item 122 and the corresponding at least one digital item.

[0040] In some embodiments, the mobile device 115 includes a data storage 168. The one or more user applications 177 may include executable program code stored in the data storage 168 for execution by the one or more processors 170. As other non-limiting examples, the one or more user applications 177 may include firmware or, in some examples, may be implemented in discrete logic.

[0041] The mobile device 115 is configured to receive the image data and process the image data in accordance with the techniques described herein. The image capture device 118 captures an image of the physical workspace 120 having the at least one physical item 122. In some embodiments, the mobile device 115 may receive the image data from external sources, such as the cloud server 112, the computer system 114, and / or the other mobile device 116, via the receiver 174. In some embodiments, the mobile device 115 may store the image data in the data storage 168 for access and processing by the note management application 178 and / or other of the one or more user applications 177.

[0042] The mobile device 115 further includes a graphical user interface (GUI) 179. In some embodiments, the one or more user applications 177 may invoke kernel functions of the operating system 164 to output the GUI 179 for presenting information to the user 126 of the mobile device 115. In some embodiments, the note management application 178 may construct and / or control the GUI 179 to provide an improved electronic environment for generating and / or manipulating the corresponding one or more digital items representative of the at least one physical item 122. For example, the note management application 178 may construct the GUI 179 to include a mechanism that allows the user 126 to easily add the at least one digital item to and / or deleting the at least one digital item recognized from the image data. In some embodiments, the note management application 178 may provide functionality by which the user 126 is able to record and / or manage relationships between one or more groups of the at least one digital item by way of the GUI 179.

[0043] FIG. 3 illustrates a schematic block diagram of at least one system 300 for simultaneous capture of digital and physical items using augmented reality, according to an embodiment of the present disclosure. The at least one system 300 includes at least one computing device 301. The computing device 301 as described herein is but one example of a suitable computing device and does not suggest any limitation on the scope of any embodiments presented. Nothingillustrated or described with respect to the computing device 301 should be interpreted as being required or as creating any type of dependency with respect to any element or plurality of elements. In some embodiments, the computing device 301 may include, but need not be limited to, a desktop, a laptop, a server, a client, a tablet, a smartphone, a computing cloud, or any other type of device that may utilize data.

[0044] In an embodiment, the at least one computing device 301 includes one or more processors 302, output devices 304, one or more input devices 306, and at least one memory including a nonvolatile memory 308 (e.g., ROM, flash memory, etc.) and / or a volatile memory 310 (e.g., RAM, etc.). The at least one memory is coupled to at least one of the one or more processors 302. The at least one memory comprises instructions that configure the at least one computing device 301 to perform functionalities as discussed herein of the mobile device 115 to perform capture of digital and physical items. The at least one processor 302 may but need not correspond to the processor 170 (shown in FIG. 2). The computing device 301 may include one or more displays, display hardware, and / or output devices 304 such as, for example, AR / VR / MR / XR hardware (which may utilize input devices 306 such as imaging sensors), monitors, speakers, headphones, projectors, wearable-displays, holographic displays, printers, and the like. The output devices 304 may further include, for example, displays and / or speakers, devices that emit energy (radio, microwave, infrared, visible light, ultraviolet, x-ray and gamma ray), electronic output devices (Wi-Fi, radar, laser, etc.), audio (of any frequency), and the like.

[0045] The one or more input devices 306 which may include, by way of example, any type of mouse, keyboard, disk / media drive, memory stick / thumb-drive, memory card, pen, touch-input device, biometric scanner, gaze and / or blink tracker, tracker, voice / auditory input device, motiondetector, camera, scale, and any device capable of measuring data such as motion data (e.g., an accelerometer, GPS, a magnetometer, a gyroscope, etc.), biometric data (e.g., blood pressure, pulse, heart rate, perspiration, temperature, voice, facial-recognition, motion / gesture tracking, gaze tracking, iris or other types of eye recognition, hand geometry, oxygen saturation, glucose level, fingerprint, DNA, dental records, weight, or any other suitable type of biometric data, etc.), video / still images, and audio (including human-audible and human-inaudible ultrasonic sound waves). The one or more input devices 306 may further include any type of device capable of receiving data, whether from another device, visual and / or audio data captured from the real world, object detection data, and the like. The one or more input devices 306 may include cameras (with or without audio recording), such as digital and / or analog cameras, still cameras, video cameras, thermal imaging cameras, infrared cameras, imaging sensors, cameras with a charge-couple display, night-vision cameras, three-dimensional cameras, webcams, audio recorders, and the like. By way of non-limiting example, the one or more input devices 306 and / or the display / output device 304 may correspond to I / O 176 depicted (shown in FIG. 2).

[0046] In some embodiments, the at least one computing device 301 includes a network interface 312. The network interface 312 may facilitate communications over a network 314 with other data source(s) such as a database 318 via wires, a wide area network, a local area network, a personal area network, a cellular network, a satellite network, and the like. Suitable local area networks may include wired Ethernet and / or wireless technologies such as, for example, wireless fidelity (Wi-Fi). Suitable personal area networks may include wireless technologies such as, for example, IrDA, Bluetooth, Wireless USB, Z-Wave, ZigBee, and / or other near field communication protocols. Suitable personal area networks may similarly include wired computer buses such as, for example, USB and FireWire. Suitable cellular networks may include, but are not limited to, technologies such as UTE, WiMAX, UMTS, CDMA, GSM, and the like.

[0047] The network interface 312 is configured to be communicatively coupled to any device capable of transmitting and / or receiving data via one or more network(s) 314. By way of non-limiting example, the network interface 312 may correspond to the transmitter 172 and / or the receiver 174 (shown in FIG. 2). Accordingly, the network interface 312 may include a communication transceiver for sending and / or receiving any wired or wireless communication. For example, the network interface 312 may include an antenna, a modem, LAN port, Wi-Fi card, WiMax card, mobile communications hardware, near-field communication hardware, satellite communication hardware and / or any wired or wireless hardware for communicating with other networks and / or devices. The network interface 312 is configured to facilitate communication with one or more remote devices, which may include, for example, client and / or server devices. The network interface 312 may also be described as a communications module, as these terms may be used interchangeably.

[0048] The at least one computing device 301 further includes a computer-readable medium 316. The computer-readable medium 316 includes at least one non-transitory computer-readable medium. The computer-readable medium 316 is interchangeably referred to as “the non- transitory computer readable medium 316”. The computer readable medium 316 may reside, for example, within the input device 306, the non-volatile memory 308, the volatile memory 310, or any combination thereof. The computer readable storage medium 316 may include tangible media that is able to store instructions associated with, or used by, a device or system. The computer readable medium 316 includes, by way of non-limiting examples: RAM, ROM, cache, fiber optics, EPROM / Flash memory, CD / DVD / BD-ROM, hard disk drives, solid-state storage, optical or magnetic storage devices, diskettes, electrical connections having a wire, or any combination thereof. The non-transitory computer readable medium 316 may also include, for example, a system or device that is of a magnetic, an optical, a semiconductor, or an electronic type. The non-transitory computer readable medium 316 excludes carrier waves and / or propagated signals taking any number of forms such as an optical, an electromagnetic,or combinations thereof. In some embodiments, the at least one non-transitory computer- readable medium 316 is configured to store instructions that, when executed, configure the at least one processor 302 to perform functionalities as discussed herein.

[0049] The database 318 is depicted as being accessible over the network 314 and may reside within a server, the cloud, or any other configuration to support being able to remotely access data and store data in the database 318. By way of non-limiting example, the database 318, the nonvolatile memory 308, the volatile memory 310, and / or the computer readable medium 316 may correspond to the data storage 168 (shown in FIG. 2).

[0050] FIG. 4 illustrates non-limiting examples of the at least one physical item 400 in various configurations. The at least one physical item 400 may be a hand-written note or stationery or one or more sticky notes, which are small pieces of paper with adhesive backing that allow users to attach them to surfaces like desks, computers, or walls. Both handwritten notes and sticky notes are practical tools for managing tasks, capturing ideas, and organizing thoughts in a physical, tangible way.

[0051] FIG. 5 illustrates non-limiting examples of the at least one digital item 500 in various configurations. The at least one digital item 500 may be created by a user directly on digital platforms, such as typing on a computer, tablet, or smartphone. As another non-limiting example, the at least one digital item 500 may correspond to the physical item that was previously captured through digital means, such as scanning, or photographing. In another nonlimiting example, the at least one digital item 500 may be the plurality of digital items including the at least one digital item created by the user and the at least one digital item corresponding to the previously captured physical item. In any of the above non-limiting examples, the at least one digital item 500 is displayed on a mobile device 502, which may be similar to the mobile device 115 as discussed in FIGS. 1 and 2 and may include similar components.

[0052] FIG. 6 illustrates an exemplary augmented reality view 605 including at least one physical item 601 and at least one digital item 602. The augmented reality view 605 further includes environment 600 including a physical workspace 622. In the illustrated example environment 600 of FIG. 6, the physical workspace 622 includes a whiteboard. Further, an augmented reality device 604 (such as augmented glasses, a headset, a head mounted device / display, and the like) may be used as the mobile device 115 shown in FIG. 1. The user 126 may wear the augmented reality device 604 for viewing the augmented reality view 605 including the at least one physical item 601 and the at least one digital item 602. The augmented reality view 605 further includes borders 610 for each of the at least one physical item 601 and the at least one digital item 602. In some embodiments, the at least one physical item 601 includes a note or stationery. The at least one digital item 602 does not physically exist where it is depicted in the augmented reality view 605. In other words, the at least one digital item 602 does not physically exist in the physical workspace 622.

[0053] The augmented reality device 604 may be configured to receive digital item location data pertaining to the at least one digital item 602. In some embodiments, digital item location data may be relative to physical location of physical objects in the physical (coordinates and the like, a physical object, etc.), relative to location data of other digital objects (such as if a digital object has been anchored relative to a user and / or one or more physical world reference points), and / or the physical world generally. In some embodiments, a recognition feature provides for a proposed digital object based upon a detected physical world object. By way of non-limiting example, push-pins in the physical world could categorically trigger suggestions or generation of digital notes to be anchored or otherwise associated with the push-pins. A camera of the augmented reality device 604 may be configured to capture image data of the physical workspace 622. The image data may include the at least one physical item 601. Further, the digital item location data is mapped to the image data. The augmented reality device 604 may also detect at least one anchor 608 in the image data within the physical workspace 622. In some embodiments, the at least one anchor 608 in the image data within the physical workspace 622 may be specified by the user 126. It is to be noted that the anchor 608 is a specific location in the image data. In some embodiments, the user 126 may add the at least one digital item 602 to the anchor 608. In some embodiments, the augmented reality device 604 may automatically add the at least one digital item 602 to the anchor 608.

[0054] FIG. 7A illustrates an exemplary augmented reality view 704 including at least one physical item 706. The augmented reality view 704 may include an environment 700 including a physical workspace 722. The physical workspace 722 includes a part of wall that is used to place the at least one physical item 706. In embodiments, a physical note or other physical object itself may be considered to constitute a physical workspace 722. In the illustrated environment 700, the at least one physical item 706 includes a plurality of sticky notes. In some embodiments, the at least one physical item 706 includes a note or, stationery. The environment 700 further shows a mobile device 702 displaying the augmented reality view 704 including the at least one physical item 706. The mobile device 702 may be similar to the mobile 115 as discussed in FIGS. 1 and 2 and may include similar components. A camera of the mobile device 702 may be configured to capture image data of the physical workspace 722. The image data includes the at least one physical item 706.

[0055] Referring to FIGS. 3 and 7A, image data of the physical workspace 722 may be received from the camera. The image data includes the at least one physical item 706. The at least one processor 302 is further configured to output the augmented reality view 704 including the at least one physical item 706.

[0056] FIG. 7B illustrates an exemplary augmented reality view 704 including at least one physical item 706 and at least one digital item 708. In other words, the augmented reality view 704 includes the plurality of digital items 706 and the plurality of physical items 708. Theaugmented reality view 704 includes the environment 700 of FIG. 7A including the physical workspace 722. The physical workspace 722 includes a part of wall that is used to place the at least one physical item 706. In the illustrated environment 700, the at least one physical item 706 includes a plurality of sticky notes. The environment 700 further shows the mobile device 702 displaying the augmented reality view 704. The mobile device 702 may be similar to the mobile 115 as discussed in FIGS. 1 and 2 and may include similar components.

[0057] Further, as shown in the exemplary embodiment of FIG. 7B, the augmented reality view 704 may include the at least one physical item 706 and at least one digital item 708. The augmented reality view 704 may further include borders 710 for each of the at least on physical item 706 and the at least one digital item 708. The at least one digital item 708 may not physically exist where it is depicted in the augmented reality view 704. In other words, the at least one digital item 708 may not physically exist in the physical workspace 722. In this example, digital items 708 may not be physically present but appear virtually on the mobile device 115 and are in the physical workspace 722 to demonstrate their existence in augmented reality if viewed with the mobile device 702 capable of viewing such augmented reality objects. In other words, in this example the user 126 may not see the at least one digital item 708 on the wall without the aid of augmented reality.

[0058] The mobile device 702 may be configured to receive digital item location data pertaining to the at least one digital item 708. Further, a camera of the mobile device 702 may be configured to capture image data of the physical workspace 722. The image data includes the at least one physical item 706. Further, the digital item location data is mapped to the image data.

[0059] The mobile device 702 may further detect at least one anchor 712 in the image data within the physical workspace 722. In some embodiments, the at least one anchor 712 in the image data within the physical workspace 722 may be specified by the user 126. It is to be noted that the anchor 712 is a specific location in the image data. For example, in the illustrated embodiment of FIG. 7B, a center of the part of wall of the physical workspace 722 may be used as the at least one anchor 712 for the environment 700. The augmented reality view 704 may include the at least one physical item 706, the at least one anchor 712, and the at least one digital items708. In some embodiments, the user 126 may add the at least one digital item 708 to the anchor 712. In some embodiments, the mobile device 702 may automatically add the at least one digital item 708 to the anchor 712. In some embodiments, the user 126 may provide a user input to add a digital item to the anchor 712 when anchor 712 is located remotely from the user 126 (for example, in a remote environment).

[0060] Referring to FIGS. 3, 6 and 7B, the at least one processor 302 may be configured to receive the digital item location data pertaining to the at least one digital item 602, 708. The at least one processor 302 may be further configured to receive the image data of the physical workspace 622, 722 from the camera. The image data may include the at least one physical item 601, 706and the digital item location data is mapped to the image data. The at least one processor 302 may be further configured to output the augmented reality view 605, 704 including the at least one physical item 601, 706 and the at least one digital item 602, 708. The at least one digital item 602, 708 may not physically exist where it is depicted in the augmented reality view 605, 704. Further, the at least one processor 302 may be further configured to receive an input to capture the augmented reality view 704 with the at least one physical item 601, 706 and the at least one digital item 602, 708 each identified.

[0061] FIG. 8 illustrates an exemplary flowchart of a method 800 for simultaneous capture of digital and physical items using augmented reality, according to an embodiment of the present disclosure. It is to be noted that the method 800 may be configured to be performed by the at least one system 300 shown in FIG. 3. Specifically, the method 800 may be configured to be performed by the at least one computing device 301 of the at least one system 300. More specifically, the at least one non-transitory computer-readable medium 316 may be configured to store instructions that, when executed, configure the at least one processor 302 to perform operations of the method 800.

[0062] Referring to FIGS. 1-3, 6, 7B and 8, at operation 802, the method 800 may include receiving digital item location data pertaining to at least one digital item (e.g., the digital item 602, 708). The at least one digital item may be an already existing item in a digital format and may be already stored in the data storage 168 of the mobile device 115 or created using the mobile device 115. The at least one digital item may correspond to a physical item which was previously captured by the camera of the mobile device 115. The at least one digital item may have been previously obtained by another camera or other image capture device. The digital item location data may refer to real / physical world location data used to show virtual objects, e.g., the at least one digital item. Therefore, the digital item location data may include data regarding a location at which the at least one digital item is to be displayed in an augmented reality view (e.g., the augmented reality view 605, 704).

[0063] At operation 804, the method 800 may include receiving image data of a physical workspace (e.g., the physical workspace 120) from a camera (e.g., the image capture device 118), such that the image data includes at least one physical item (e.g., the physical item 601, 706) and the digital item location data may be mapped to the image data. In some embodiments, the physical item is a note or a stationery.

[0064] At operation 806, the method 800 may include outputting the augmented reality view (e.g., the augmented reality view 605, 704) including the at least one physical item and the at least one digital item, such that the at least one digital item differs in how it is depicted in augmented reality from a previous state. In some embodiments, the at least one digital item may be edited in any way from a previous version of itself, or simply may no longer be at the same location as the at least on physical item that it was based on. In some embodiments, the at least onedigital item may be based on a real -world item of which the user 126 may have captured an image of, and at some point later, the digital version may have been relocated in augmented reality to be somewhere else in augmented reality that may correspond to a new physical location (such as near a new physical item).

[0065] At operation 808, the method 800 may include receiving an input to capture the augmented reality view (e.g., the augmented reality view 605, 704) with the at least one physical item and the at least one digital item each identified.

[0066] FIG. 9 illustrates an exemplary flowchart of a process 900 for simultaneous capture of digital and physical items using augmented reality, according to an embodiment of the present disclosure. The process 900 for simultaneous capture of digital and physical items using augmented reality may include the capture of multiple physical items or stationery using an augmented reality or extended reality device. This may be, for example, the multi-modality capture of physical, digital, and mixed-modality items, such that mixed-modality notes may be physical items with digital content attached / overlayed. The process 900 may be embodied as one or more algorithms implemented by the at least one processor 302 of the at least one computing device 301 shown in FIG. 3. Further, the process 900 may be stored in the at least one non-transitory computer-readable medium 316 as instructions executable by the at least one processor 302.

[0067] Referring to FIGS. 1-3, 6, 7B-9, at operation 902, the at least one processor 302 may be configured to display the physical workspace 120 through an augmented reality device, (e.g., the mobile device 115) such that the user 126 views the physical workspace 120 through the augmented reality device. At operation 904, the at least one processor 302 may be configured to display the at least one digital item and / or the at least one physical item placed within the view of operation 902 such that the user 126 views the at least one digital item and / or the at least one physical item placed within the said view. At operation 906, the at least one processor 302 may be configured to recognize the items using the augmented reality device and prompts the user 126 to perform capture. At operation 908, the at least one processor 302 may determine whether the user 126 has performed a capture. At operation 908, if no capture has been performed (condition “no”), then the process 900 may return to operation 906. Otherwise, if so (condition “yes”), then the process 900 may move to operation 910, where the at least one processor 302 may determine that the item is a digital note, and the original note remains as a spatial anchor.

[0068] FIG. 10 illustrates a non-limiting example of a user interface 1004 displayed on a screen 1002 of a mobile device 1000, according to an embodiment of the present disclosure. The mobile device 1000 may be similar to the mobile device 115 shown in FIGS. 1-3 and may include similar components, for example, the screen 1002 may be the display / output device(s) 304. The screen 1002 displays digitally captured portions of a notebook. In other non-limiting examples,Post-it® Notes and / or any other suitable types of objects (including whiteboards and the like) may be captured.

[0069] FIG. 11 illustrates another non-limiting example of a user interface 1104 displayed on a screen 1102 of a mobile device 1100, according to an embodiment of the present disclosure. The mobile device 1100 may be similar to the mobile device 115 shown in FIGS. 1-3 and may include similar components, for example, the screen 1102 may be the display / output device(s) 304. The at least one processor 302 may be configured to execute further instructions to additionally receive input that selects at least two items within the captured augmented reality view on the screen 1102 and output a plurality of context options based upon an interpreted context of the at least two items. In some embodiments, the at least one processor 302 is configured to execute further instructions to receive input that selects at least two items within the captured augmented reality view and output a plurality of context options based upon an interpreted context of the at least two items. In some embodiments, the at least one processor 302 is configured to execute further instructions to receive input that selects at least one of the at least one physical item and the at least one digital item within the captured augmented reality view and output a plurality of context options based upon an interpreted context of the at least one selected physical item, the at least one selected digital item, or both.In some embodiments, the plurality of context options include at least one of organizing a group that includes a plurality of physical items, a plurality of digital items, or a combination thereof; optical character recognition (OCR), pattern recognition, and / or object recognition of at least one digital item or physical item, and outputting suggested data based upon data associated with one or more of the plurality of physical items, a plurality of digital items, or a combination thereof.

[0070] FIG. 12 illustrates a flowchart of a process 1200 for simultaneous capture of digital and physical items using augmented reality with additional ability to suggest multiple actions based on user- defined actions or may be inferred from context, according to embodiments of the present disclosure. For example, star symbol means a particular action and / or detected action (e.g., using computer vision (CV) and / or OCR to detect key words / items. The process 1200 may be embodied as one or more algorithms implemented by the at least one processor 302 of the at least one computing device 301 shown in FIG. 3. Further, the process 1200 may be stored in the at least one non-transitory computer-readable medium 316 as instructions executable by the at least one processor 302.

[0071] At operation 1202, the at least one processor 302 may be configured to display the physical workspace 722 through an augmented reality device with defined borders, such as the mobile device 702 such that the user 126 views physical workspace through augmented reality device with defined borders. At operation 1204, the at least one processor 302 may be configured to prompt the user 126 to manually or automatically (such as with a Visual Attention Model,computer vision, computer vision recognition (OCR, object recognition, pattern recognition, anomaly detection, fiducial recognition (e.g., code detection)), classification, analytics, and the like) choose multiple physical items within view and select to capture the items as digital. At operation 1206, the at least one processor 302 may be configured to prompt the user 126 to manually or automatically (such as with a Visual Attention Model) select an individual item, such that the at least one processor 302 interprets the captured context of the item. At operation 1208, based on the captured context, the at least one processor 302 may be configured to present multiple options to the user 126.

[0072] At operation 1210, the at least one processor 302 may determine if part of context is date based. If the at least one processor 302 determines that the part of context is date based, then the process 1200 may proceed to operation 1212. At operation 1212, the at least one processor 302 may be configured to present the user 126 with calendar integrations, and the process 1200 proceeds to operation 1214. If the at least one processor 302 determines that the part of context is not date based, then the process 1200 may proceed directly to operation 1214.

[0073] At operation 1214, the at least one processor 302 may determine if part of the context is list based. If the at least one processor 302 determines that the part of context is list based, then the process 1200 may proceed to operation 1216. At operation 1216, the at least one processor 302 may be configured to present the user 126 with to do list integrations, and the process 1200 proceeds to operation 1218. If the at least one processor 302 determines that the part of context is not list based, then the process 1200 may proceed directly to operation 1218.

[0074] At operation 1218, the at least one processor 302 may determine if part of the context is contact based or user based. If the at least one processor 302 determines that the part of context is contact based or user based, then the process 1200 may proceed to operation 1220. At operation 1220, the at least one processor 302 may be configured to present the user with contact integrations, and the process 1200 proceeds to operation 1222. If the at least one processor 302 determines that the part of context is not contact based or user based, then the process 1200 may proceed directly to operation 1222.

[0075] At operation 1222, the at least one processor 302 may determine if part of the context is text based. If the at least one processor 302 determines that the part of context is text based, then the process 1200 may proceed to operation 1224. At operation 1224, the at least one processor 302 may be configured to present the user 126 with note integrations, and the process 1200 may proceed to operation 1226. If the at least one processor 302 determines that the part of context is not text, then the process 1200 may proceed directly to operation 1226. At operation 1226, the at least one processor 302 may be configured to present the user 126 with custom integrations (user configured, for example).

[0076] From operation 1226, the process 1200 may return to operation 1206. In some embodiments, other types of context may be utilized. In some embodiments, the types of context may be considered in any order.

[0077] Referring to FIGS. 1-12, the exemplary system 300 and exemplary method 800 for simultaneous capture of digital and physical items using augmented reality is provided with additional ability to suggest multiple actions based on user-defined actions. The system 300 and the method 800 may allow overlaying the at least one digital item onto the at least one physical item, thereby allowing the user 126 to seamlessly integrate and organize information from both physical and digital sources. The system 300 and the method 800 may also support multiple modalities (physical, digital, and mixed-modality items), thereby enabling the user 126 to capture a wider range of items in various formats. The system 300 and the method 800 may also allow the user 126 to view their physical workspace and interact with items in real time using an augmented reality device. The system 300 and the method 800 may allow using the augmented reality device to recognize physical items and prompt the user 126 to capture them, such that even after physical item is captured and transformed into digital item, the original item may remain as a spatial anchor, preserving its physical reference point within the user's environment.

[0078] As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” encompass embodiments having plural referents, unless the content clearly dictates otherwise. As used in this specification and the appended claims, the term “or” is generally employed in its sense including “and / or” unless the content clearly dictates otherwise.

[0079] While particular embodiments have been illustrated and described herein, it should be understood that various other changes and modifications may be made without departing from the scope of the claimed subject matter. Moreover, although various aspects of the claimed subject matter have been described herein, such aspects need not be utilized in combination. It is therefore intended that the appended claims cover all such changes and modifications that are within the scope of the claimed subject matter.

Claims

CLAIMS1. At least one non-transitory computer-readable medium storing instructions that, when executed, configure at least one processor to: receive digital item location data pertaining to at least one digital item; receive image data from a camera, wherein the image data includes at least one physical item and the digital item location data is mapped to the image data; output an augmented reality view including the at least one physical item and the at least one digital item, wherein the at least one digital item differs in how it is depicted in augmented reality; and receive an input to capture the augmented reality view with the at least one physical item and the at least one digital item each identified.

2. The at least one non-transitory computer-readable medium of claim 1, wherein the digital item data was previously obtained by another camera.

3. The at least one non-transitory computer-readable medium of claim 1, wherein the digital item corresponds to a physical item that was previously captured.

4. The at least one non-transitory computer-readable medium of claim 1, wherein the digital item was created by a user.

5. The at least one non-transitory computer-readable medium of claim 1, wherein the augmented reality view includes a plurality of digital items and a plurality of physical items.

6. The at least one non-transitory computer-readable medium of claim 1, wherein an item is a note or stationery.

7. The at least one non-transitory computer-readable medium of claim 1, wherein the plurality of digital items includes at least one digital item created by a user and at least one digital item corresponding to a previously-captured physical item.

8. The at least one non-transitory computer-readable medium of claim 1, further including instructions to: receive input that selects at least two items within the captured augmented reality view; and output a plurality of context options based upon an interpreted context of the at least two items.

9. The at least one non-transitory computer-readable medium of claim 8, further including instructions to: receive input that selects at least one of the at least one physical item and the at least one digital item within the captured augmented reality view; and output a plurality of context options based upon an interpreted context of the at least one selected physical item, the at least one selected digital item, or both.

10. The at least one non-transitory computer-readable medium of claim 8, wherein the plurality of context options include at least one of (i) organizing a group that includes a plurality of physical items, a plurality of digital items, or a combination thereof, (ii) optical character recognition of at least one digital item or physical item, and (iii) outputting suggested data based upon data associated with one or more of the plurality of physical items, a plurality of digital items, or a combination thereof.

11. At least one system including: at least one computing device comprising one or more processors; and at least one memory coupled to at least one of the one or more processors, wherein the at least one memory comprises instructions that configure the at least one computing device to: receive digital item location data pertaining to at least one digital item; receive image data from a camera, wherein the image data includes at least one physical item and the digital item location data is mapped to the image data; output an augmented reality view including the at least one physical item and the at least one digital item, wherein the at least one digital item differs in how it is depicted in augmented reality; and receive an input to capture the augmented reality view with the at least one physical item and the at least one digital item each identified.

12. The at least one system of claim 11, wherein the augmented reality view includes a plurality of digital items and a plurality of physical items.

13. The at least one system of claim 11, wherein an item is a note or stationery.

14. The at least one system of claim 11, wherein the plurality of digital items includes at least one digital item created by a user and at least one digital item corresponding to a previously-captured physical item.

15. The at least one system of claim 11, further including instructions to: receive input that selects at least two items within the captured augmented reality view; andoutput a plurality of context options based upon an interpreted context of the at least two items.

16. The at least one system of claim 15, further including instructions to: receive input that selects at least one of the at least one physical item and the at least one digital item within the captured augmented reality view; and output a plurality of context options based upon an interpreted context of the at least one selected physical item, the at least one selected digital item, or both.

17. The at least one system of claim 15, wherein the plurality of context options include at least one of (i) organizing a group that includes a plurality of physical items, a plurality of digital items, or a combination thereof, (ii) optical character recognition of at least one digital item or physical item, and (iii) outputting suggested data based upon data associated with one or more of the plurality of physical items, a plurality of digital items, or a combination thereof.

18. A method including: receiving digital item location data pertaining to at least one digital item; receiving image data from a camera, wherein the image data includes at least one physical item and the digital item location data is mapped to the image data; outputting an augmented reality view including the at least one physical item and the at least one digital item, wherein the at least one digital item differs in how it is depicted in augmented reality; and receiving an input to capture the augmented reality view with the at least one physical item and the at least one digital item each identified.

19. The method of claim 18, wherein the digital item data was previously obtained by another camera.

20. The method of claim 18, wherein the digital item corresponds to a physical item that was previously captured.

Images