Methods, media, and systems for compartmentalization recognition for multi-modality notes

Abstract

A method includes receiving digital item location data pertaining to at least one digital item and receiving image data of a physical space from a camera, wherein the image data includes at least one physical item and the digital item location data is mapped to the image data. The method further includes outputting an augmented reality view including the at least one physical item and the at least one digital item, wherein the at least one digital item does not physically exist where depicted in augmented reality and generating a group indicator to group together at least one physical item, at least one digital item, at least one anchor, or any combination thereof.

Description

METHODS, MEDIA, AND SYSTEMS FOR COMPARTMENTALIZATION RECOGNITION FOR MULTI-MODALITY NOTESTechnical Field

[0001] The present disclosure relates to media, methods, and systems for compartmentalization recognition for multi -modality notes.Background

[0002] In recent years, use of physical means for taking notes has been combined with the use of digital means. For example, during meetings and collaboration sessions, some participants use software based notes in digital format and some use traditional physical notes, such as, paper notes, whiteboard, or paper, etc. As much work is nowadays preferred in a digital format, the physical notes are transformed to the digital format by a participant by manually typing the physical notes onto a computing environment to combine with the software based notes. The transformation of the physical notes into the digital format may become time consuming and cumbersome for the participant.Summary

[0003] In one embodiment, at least one non-transitory computer-readable medium includes instructions that, when executed, configure at least one processor to receive digital item location data pertaining to at least one digital item and receive image data of a physical space from a camera, wherein the image data includes at least one physical item and the digital item location data is mapped to the image data. The at least one non-transitory computer-readable medium includes further instructions to output an augmented reality view including the at least one physical item and the at least one digital item, wherein the at least one digital item does not physically exist where depicted in augmented reality and generate a group indicator to group together at least one physical item, at least one digital item, at least one anchor, or any combination thereof.

[0004] In another embodiment, at least one system includes at least one computing device including one or more processors and at least one memory coupled to at least one of the one or more processors, wherein the at least one memory includes instructions that configure the at least one computing device to receive digital item location data pertaining to at least one digital item and receive image data of a physical space from a camera, wherein the image data includes at least one physical item and the digital item location data is mapped to the image data. The at least one system of claim 12 includes further instructions to output an augmented reality view including the at least one physical item and the at least one digital item, wherein the at least one digital item does not physically exist where depicted in augmented reality and generate a groupindicator to group together at least one physical item, at least one digital item, at least one anchor, or any combination thereof.

[0005] In yet another embodiment, a method includes receiving digital item location data pertaining to at least one digital item and receiving image data of a physical space from a camera, wherein the image data includes at least one physical item and the digital item location data is mapped to the image data. The method further includes outputting an augmented reality view including the at least one physical item and the at least one digital item, wherein the at least one digital item does not physically exist where depicted in augmented reality and generating a group indicator to group together at least one physical item, at least one digital item, at least one anchor, or any combination thereof.Brief Description of the Drawings

[0006] FIG. 1 shows a schematic diagram of an environment for recognizing physical notes, according to an embodiment of the present disclosure;

[0007] FIG. 2 shows a schematic block diagram of a mobile device, according to an embodiment of the present disclosure;

[0008] FIG. 3 shows a schematic block diagram of at least one system for compartmentalization recognition for multi-modality notes, according to an embodiment of the present disclosure;

[0009] FIG. 4 shows an exemplary augmented reality view including at least one physical item and at least one digital item;

[0010] FIG. 5 shows another exemplary augmented reality view including at least one physical item and at least one digital item;

[0011] FIG. 6 shows a user interface displayed on a screen of a mobile device, according to an embodiment of the present disclosure

[0012] FIG. 7 shows a flowchart of a method for compartmentalization recognition for multi-modality notes, according to an embodiment of the present disclosure;

[0013] FIG. 8 shows a flowchart of a process for compartmentalization recognition for the multimodality notes, according to an embodiment of the present disclosure; and

[0014] FIG. 9 shows a flowchart of a process for compartmentalizing the multi-modality notes, according to an embodiment of the present disclosure.Detailed Description

[0015] As used herein, the term “physical item” generally refers to objects with a general boundary and recognizable content. The physical items can include objects resulting after people write, draw, or enter via other type of inputs on the objects, for example, paper, white board, or other objects accepting the inputs. By way of examples, the physical notes can include hand-writtenrepositionable paper notes, paper, or film, white-board with drawings, posters, and signs. In some cases, the physical items can be generated using digital means, e.g., printing onto printable repositionable paper notes or printed document. In some cases, one object can include several notes. For example, several ideas can be written on a piece of poster paper or a whiteboard. The physical items can be two-dimensional or three dimensional. The physical items can have various shapes and sizes. For example, a physical item may be a 3 inches x 3 inches note; a physical note may be a 26 inches x 39 inches poster; and a physical item may be a triangular metal sign. In some cases, the physical notes have known shapes and / or sizes. The term “physical item” may be interchangeably referred to as the “physical note”.

[0016] As used herein, the term “digital item” generally refers to digital objects with information and / or ideas. The digital items can be generated using digital inputs. Digital inputs can include, for example, keyboards, touch screens, digital cameras, digital recording devices, stylus, digital pens, or the like. In some cases, the digital items may be representative of the physical items.

[0017] As used herein, the term “multi -modality notes” generally refers to combination of the physical items and the digital items in a single environment. In such case, an environment is said to have the multi -modality notes. The physical items and the digital items may be collectively referred to as “the items” herein.

[0018] As used herein, the term “physical workspace” generally refers to a location or an area in a physical space in either two-dimensional or three-dimensional space. The workspace may refer to an office space or a room, in which users work and / or organize collaborative session such as meetings and the like, and make one or more notes. In some examples, the workspace may refer to a notebook page, in which user(s) make one or more notes.

[0019] The present disclosure describes techniques for creating and manipulating software items representative of physical items, such as, physical notes, along with digital notes in a digital environment. For example, techniques are described for recognizing the physical items present within a physical workspace, capturing information therefrom, and creating corresponding digital representations of the physical items. Further, at least some aspects of the present disclosure are directed to techniques for grouping and managing multi-modality notes.

[0020] The present disclosure relates to a method for compartmentalization recognition for multimodality notes. The method may be configured to generate the augmented reality view including a digital representation of the at least one physical item and the at least one digital item. Accordingly, the method may be configured to create a digital twin of the at least one physical item in the augmented reality view. Further, the method may project the at least one digital item into the augmented reality view. Hence, the method may provide access to the multi-modality items, i.e., the multi-modality notes, in the augmented reality view. The method may further generates the group indicator to group together the at least one physical item, theat least one digital item, the at least one anchor, or any combination thereof. Hence, the method may facilitate grouping of the multi-modality notes for easy access by a user.

[0021] Referring to Figures now, FIG. 1 shows a schematic diagram of an environment 100 for recognizing at least one physical item 122, according to an embodiment of the present disclosure. The environment 100 may include a mobile device 115 to capture and recognize the at least one physical item 122 from a physical workspace 120. The mobile device 115 may be associated with a user 126. The mobile device 115 may provide an execution environment for one or more software applications that may efficiently capture and extract content from the at least one physical item 122 from the physical workspace 120. In one non-limiting example, the at least one physical item 122 may be results of a collaborative brainstorming session having multiple participants.

[0022] In some embodiments, the mobile device 115 may be a mobile phone. In other embodiments, the mobile device 115 may be a tablet computer, a personal digital assistant (PDA), a laptop computer, a media player, an e-book reader, a wearable computing device (e.g., a watch, eyewear, a glove), or any other type of mobile or non-mobile computing device suitable for performing the techniques described herein.

[0023] In some embodiments, the mobile device 115 may include one or more software executing on the mobile device 115. The mobile device 115 and software executing thereon are configured to perform a variety of note-related operations, including automated creation of at least one digital item representative of the at least one physical item 122 of the physical workspace 120. The mobile device 115, and the software executing thereon, may provide a platform for creating and manipulating the at least one digital items representative of the at least one physical item 122. In some examples, the mobile device 115 may be configured to recognize the at least one physical item 122 by determining a boundary of the at least one physical item 122. After the at least one physical item 122 is recognized, the mobile device 115 may extract the corresponding content of the at least one physical item 122, where the content may be a visual information of the at least one physical item 122.

[0024] The mobile device 115 may implement techniques for automated detection and recognition of the one or more physical items 122 and extraction of corresponding information, content or other characteristics associated with the at least one physical item 122. For example, the mobile device 115 may allow the user 126 fine grain control over techniques used by the mobile device 115 to detect and recognize the at least one physical item 122. As one example, the mobile device 115 may allow the user 126 to select between marker-based detection techniques in which the one or more of physical items 122 includes a physical fiducial mark on the surface of the physical item 122 and / or non-marker-based techniques in which no fiducial mark is used.

[0025] In some embodiments, the mobile device 115 includes an image capture device 118 and a presentation device 128. The mobile device 115 may be configured to process image data captured by the image capture device 118 to detect and recognize the at least one physical item 122 positioned within the physical workspace 120. The image capture device 118 may be a camera or any other suitable component configured to capture image data representative of the physical workspace 120 and the at least one physical item 122 positioned in the physical workspace 120. In other words, the image data may capture one or more visual representations of the physical workspace 120, having the at least one physical item 122. Although discussed as the camera of the mobile device 115, the image capture device 118 may include other components capable of capturing the image data, such as, a video recorder, an infrared camera, a Charge Coupled Device (CCD) array, a laser scanner, or the like. The image data includes the at least one physical item 122. The image data may further include at least one of an image, a video, a sequence of images (i.e., multiple images taken within a time period and / or with an order), a collection of images, image portion(s), and / or the like.

[0026] The presentation device 128 may include, but not limited to, an electronically addressable display, such as a liquid crystal display (LCD) or other type of display device capable of use with the mobile device 115 to display the image data. In some embodiments, the presentation device 128 may be an input / output device 176 (shown in FIG. 2) and / or a display / output device(s) 304 (shown in FIG. 3). In some embodiments, the mobile device 115 may generate content to display on the presentation device 128 for the at least one physical item 122 in a variety of formats, for example, a list, grouped in rows and / or column, a flow diagram, or the like. In some embodiments, the mobile device 115 may communicate display information for presentation by other devices, such as a tablet computer, a projector, an electronic billboard, or other external device.

[0027] In addition, the mobile device 115 may provide the user 126 with an improved electronic environment for generating and manipulating corresponding the at least one digital item representative of the at least one physical item 122. By way of non-limiting example, the mobile device 115 may provide mechanisms allowing the user 126 to easily add new digital items to, edit the at least one digital items within, and / or delete the at least one digital item representative of the at least one physical item 122 created during brainstorming activity associated with the physical workspace 120.

[0028] The environment 100 further includes a cloud server 112, a computer system 114, and another mobile device 116. The cloud server 112, the computer system 114, and the other mobile device 116 may be communicatively coupled with the mobile device 115 using a network (not shown) . In some embodiments, the mobile device 115 may provide functionality by which the user 126 is able to record and manage relationships between groups of the at least one physical note 122.In some embodiments, the mobile device 115 may provide functionality by which the user 126 is able to export the at least one digital note representative of the at least one physical item 122 to other systems, such as cloud-based repositories (e.g., the cloud server 112) and / or other computing devices (e.g., the computer system 114 and / or the other mobile device 116).

[0029] FIG. 2 shows a schematic block diagram of the mobile device 115 of the environment 100 of FIG. 1, according to an embodiment of the present disclosure. The mobile device 115 may include various hardware components that provide core functionality for operation of the mobile device 115. The mobile device 115 includes one or more processors 170. In some embodiments, the one or more processors 170 may be embodied in a number of different ways. For example, the one or more processors 170 may be embodied as various processing means, such as one or more of a microprocessor or other processing elements, a coprocessor, or various other computing or processing devices, including integrated circuits, such as, e.g., an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or the like.

[0030] As such, whether configured by hardware or by a combination of hardware and software, the one or more processors 170 may represent an entity (e.g., physically embodied in circuitry - in the form of a processing circuitry) capable of performing operations according to some embodiments while configured accordingly. Thus, for example, when the one or more processors 170 are embodied as an executor of software instructions, the instructions may specifically configure the one or more processors 170 to perform the operations described herein. Alternatively, as another example, when the one or more processors 170 are embodied as the ASIC, FPGA, or the like, the one or more processors 170 may have specifically configured hardware for conducting the operations described herein.

[0031] The one or more processors 170 are configured to operate according to executable instructions (i.e., program code), typically stored in a computer-readable medium or a data storage such as a static, random-access memory (SRAM) device or a Flash memory device. The mobile device 115 further includes the input / output (I / O) interface 176. The I / O interface 176 may include one or more devices, such as a keyboard, a camera button, a power button, a volume button, a home button, a back button, a menu button, or the presentation device 128 (shown in FIG. 1).

[0032] The mobile device 115 further includes a transmitter 172 and a receiver 174. The transmitter 172 and the receiver 174 are configured to provide wireless communication with other devices, such as the cloud server 112, the computer system 114, or the other mobile device 116 as described in FIG. 1, via a wireless communication interface (not shown), such as but not limited to high-frequency radio frequency (RF) signals. The mobile device 115 may include additional discrete digital logic or analog circuitry (not shown), and / or may be embodied in any suitabletype of device such as a smartphone, laptop, tablet, wearable device (such as wearable augmented reality device such as augmented reality glasses), and the like.

[0033] The mobile device 115 further includes an operating system 164. The operating system 164 may execute on the one or more processors 170 and provide an operating environment for one or more user applications 177 (commonly referred to “apps”). In some embodiments, the one or more user applications 177 include a note management application 178 for managing the at least one physical item 122 and the corresponding at least one digital item.

[0034] In some embodiments, the mobile device 115 includes a data storage 168. The one or more user applications 177 may include executable program code stored in the data storage 168 for execution by the one or more processors 170. As other non-limiting examples, the one or more user applications 177 may include firmware or, in some examples, may be implemented in discrete logic.

[0035] The mobile device 115 is configured to receive the image data and process the image data in accordance with the techniques described herein. The image capture device 118 captures an image of the physical workspace 120 having the at least one physical item 122. In some embodiments, the mobile device 115 may receive the image data from external sources, such as the cloud server 112, the computer system 114, and / or the other mobile device 116, via the receiver 174. In some embodiments, the mobile device 115 may store the image data in the data storage 168 for access and processing by the note management application 178 and / or other of the one or more user applications 177.

[0036] The mobile device 115 further includes a graphical user interface (GUI) 179. In some embodiments, the one or more user applications 177 may invoke kernel functions of the operating system 164 to output the GUI 179 for presenting information to the user 126 of the mobile device 115. In some embodiments, the note management application 178 may construct and / or control the GUI 179 to provide an improved electronic environment for generating and / or manipulating the corresponding one or more digital items representative of the at least one physical item 122. For example, the note management application 178 may construct the GUI 179 to include a mechanism that allows the user 126 to easily add the at least one digital item to and / or deleting the at least one digital item recognized from the image data. In some embodiments, the note management application 178 may provide functionality by which the user 126 is able to record and / or manage relationships between one or more groups of the at least one digital item by way of the GUI 179.

[0037] FIG. 3 illustrates a schematic block diagram of at least one system 300 for compartmentalization recognition for multi -modality notes, according to an embodiment of the present disclosure.

[0038] The at least one system 300 includes at least one computing device 301. The computing device 301 as described herein is but one example of a suitable computing device and does not suggest any limitation on the scope of any embodiments presented. Nothing illustrated or described with respect to the computing device 301 should be interpreted as being required or as creating any type of dependency with respect to any element or plurality of elements. In some embodiments, the computing device 301 may include, but need not be limited to, a desktop, a laptop, a server, a client, a tablet, a smartphone, a computing cloud, or any other type of device that can utilize data.

[0039] In some embodiments, the at least one computing device 301 includes at least one processor 302, the output devices 304, one or more input devices 306, and a memory including a nonvolatile memory 308 and / or a volatile memory 310. The at least one processor 302 may but need not correspond to the one or more processors 170 (shown in FIG. 2). The computing device 301 may include one or more displays, a display hardware, and / or the output devices 304 such as, for example, AR / VR / MR / XR hardware (which may utilize the one or more input devices 306, such as imaging sensors), monitors, speakers, headphones, projectors, wearabledisplays, holographic displays, printers, and the like. The output devices 304 may further include, for example, displays and / or speakers, devices that emit energy (radio, microwave, infrared, visible light, ultraviolet, x-ray and gamma ray), electronic output devices (Wi-Fi, radar, laser, etc.), audio (of any frequency), and the like.

[0040] The one or more input devices 306 may include any type of mouse, a keyboard, a disk / media drive, memory stick / thumb-drive, a memory card, a pen, a touch-input device, a biometric scanner, a gaze and / or blink tracker, a tracker, a voice / auditory input device, a motion-detector, a camera, a scale, and any device capable of measuring data such as motion data (e.g., an accelerometer, GPS, a magnetometer, a gyroscope, etc.), biometric data (e.g., blood pressure, pulse, heart rate, perspiration, temperature, voice, facial-recognition, motion / gesture tracking, gaze tracking, iris or other types of eye recognition, hand geometry, oxygen saturation, glucose level, fingerprint, DNA, dental records, weight, or any other suitable type of biometric data, etc.), video / still images, and audio (including human-audible and human-inaudible ultrasonic sound waves). The one or more input devices 306 may further include any type of device capable of receiving data, whether from another device, visual and / or audio data captured from the real world, object detection data, and the like. The one or more input devices 306 may include cameras (with or without audio recording), such as digital and / or analog cameras, still cameras, video cameras, thermal imaging cameras, infrared cameras, imaging sensors, cameras with a charge-couple display, night-vision cameras, three-dimensional cameras, webcams, audio recorders, and the like. By way of non-limiting example, the one or more input devices 306 and / or the display / output device 304 may correspond to the I / O 176 (shown in FIG. 2).

[0041] In some embodiments, the at least one computing device 301 includes a network interface 312. The network interface 312 may facilitate communications over a network 314 with other data source(s) such as a database 318 via wires, a wide area network, a local area network, a personal area network, a cellular network, a satellite network, and the like. Suitable local area networks may include wired Ethernet and / or wireless technologies such as, for example, wireless fidelity (Wi-Fi). Suitable personal area networks may include wireless technologies such as, for example, IrDA, Bluetooth, Wireless USB, Z-Wave, ZigBee, and / or other near field communication protocols. Suitable personal area networks may similarly include wired computer buses such as, for example, USB and FireWire. Suitable cellular networks may include, but are not limited to, technologies such as UTE, WiMAX, UMTS, CDMA, GSM, and the like.

[0042] The network interface 312 is configured to be communicatively coupled to any device capable of transmitting and / or receiving data via the network 314. By way of non-limiting example, the network interface 312 may correspond to the transmitter 172 (shown in FIG. 2) and / or the receiver 174 (shown in FIG. 2). Accordingly, the network interface 312 may include a communication transceiver (not shown) for sending and / or receiving any wired or wireless communication. For example, the network interface 312 may include an antenna, a modem, LAN port, Wi-Fi card, WiMax card, mobile communications hardware, near-field communication hardware, satellite communication hardware and / or any wired or wireless hardware for communicating with other networks and / or devices. The network interface 312 is configured to facilitate communication with one or more remote devices, which may include, for example, client and / or server devices. The network interface 312 may also be described as a communications module, as these terms may be used interchangeably.

[0043] The at least one computing device 301 further includes a computer-readable medium 316. The computer-readable medium 316 includes one or more non-transitory computer readable mediums. The computer-readable medium 316 is interchangeably referred to as “the non- transitory computer readable medium 316”. The computer readable medium 316 may reside, for example, within the one or more input devices 306, the non-volatile memory 308, the volatile memory 310, or any combination thereof. The computer readable storage medium 316 may include tangible media that is able to store instructions associated with, or used by, a device or system. The computer readable medium 316 includes, by way of non-limiting examples: RAM, ROM, cache, fiber optics, EPROM / Flash memory, CD / DVD / BD-ROM, hard disk drives, solid-state storage, optical or magnetic storage devices, diskettes, electrical connections having a wire, or any combination thereof. The non-transitory computer readable medium 316 may also include, for example, a system or device that is of a magnetic, an optical, a semiconductor, or an electronic type. The non-transitory computer readable medium 316excludes carrier waves and / or propagated signals taking any number of forms such as an optical, an electromagnetic, or combinations thereof. In some embodiments, the at least one non- transitory computer-readable medium 316 is configured to store instructions that, when executed, configure the at least one processor 302 to perform functionalities as discussed herein.

[0044] The database 318 is depicted as being accessible over the network 314 and may reside within a server, the cloud, or any other configuration to support being able to remotely access data and store data in the database 318. By way of non-limiting example, the database 318, the nonvolatile memory 308, the volatile memory 310, and / or the computer readable medium 316 may correspond to the data storage 168 (shown in FIG. 2).

[0045] FIG. 4 illustrates an exemplary augmented reality view 405 including at least one physical item 401 and at least one digital item 402. The augmented reality view 405 may further include an environment 400 including a physical workspace 422. In the illustrated environment 400 of FIG. 4, the physical workspace 422 includes a whiteboard. Further, an augmented reality device 404 (such as AR glasses) may be used as the mobile device 115 shown in FIG. 1. The user 126 can wear the augmented reality device 404 for viewing the augmented reality view 405 including the at least one physical item 401 and the at least one digital item 402. The augmented reality view 405 may further include borders 410 for each of the at least one physical item 401 and the at least one digital item 402. In some embodiments, the at least one physical item 401 includes a note or stationery. The at least one digital item 402 does not physically exist where depicted in the augmented reality view 405. In other words, the at least one digital item 402 does not physically exist in the physical workspace 422.

[0046] The augmented reality device 404 may be configured to receive digital item location data pertaining to the at least one digital item 402. Further, a camera of the augmented reality device 404 may be configured to capture image data of the physical workspace 422. The image data may include the at least one physical item 401. Further, the digital item location data may be mapped to the image data. The augmented reality device 404 may further detect at least one anchor 408 in the image data within the physical workspace 422. In some embodiments, the at least one anchor 408 in the image data within the physical workspace 422 may be specified by the user 126. It is to be noted that the anchor 408 is a specific location in the image data. In some embodiments, the user 126 may add the at least one digital item 402 to the anchor 408. In some embodiments, the augmented reality device 404 may automatically add the at least one digital item 402 to the anchor 408.

[0047] FIG. 5 shows another exemplary augmented reality view 504 including a plurality of physical items 506 and a plurality of digital items 508. The augmented reality view 504 further includes an environment 500 including a physical workspace 522. The physical workspace 522 includesthe plurality of physical items 506. The environment 500 further shows a mobile device 502 displaying the augmented reality view 504. The mobile device 502 may be similar to the mobile 115 as discussed in FIGS. 1 and 2 and may include similar components. The physical workspace 522 includes a part of wall that is used to place the plurality of physical items 506. In the illustrated environment 500, the plurality of physical items 506 may include a plurality of objects such as Post-it® Notes. In some embodiments, the plurality of physical items 506 includes a note or stationery. The augmented reality view 504 further includes borders 510 for each of the plurality of physical items 506 and the plurality of digital items 508. The plurality of digital items 508 does not physically exist where depicted in the augmented reality view 504. In other words, the plurality of digital items 508 does not physically exist in the physical workspace 522, and in some embodiments is only visible to users on an augmented reality display.

[0048] The mobile device 502 may be configured to receive digital item location data pertaining to the plurality of digital items 508. Further, a camera of the mobile device 502 may be configured to capture image data of the physical workspace 522. The image data may include the plurality of physical items 506. Further, the digital item location data may be mapped to the image data. The mobile device 502 may further detect at least one anchor 512 in the image data within the physical workspace 522. In some embodiments, the at least one anchor 512 in the image data within the physical workspace 522 may be specified by the user 126. It is to be noted that the anchor 512 may be a specific location in the image data. By way of non-limiting example, in the illustrated embodiment of FIG. 5, a center of the part of wall of the physical workspace 522 is used as the at least one anchor 512 for the environment 500. The augmented reality view 504 includes the plurality of physical items 506, the at least one anchor 512, and the plurality of digital items 508. In some embodiments, the user 126 may add the plurality of digital items 508 to the anchor 512. In some embodiments, the mobile device 502 may automatically add the plurality of digital items 508 to the anchor 512. In some embodiments, the user 126 may provide a user input to add a digital item to the anchor 512 when anchor 512 is located remotely from the user 126 (for example, in a remote environment).

[0049] Referring to FIGS. 3, 4 and 5, the at least one processor 302 may be configured to receive the digital item location data pertaining to at least one digital item 402, 508. The at least one processor 302 is further configured to receive the image data of the physical workspace 422, 522 from the camera. The image data may include the at least one physical item 401, 506 and the digital item location data is mapped to the image data. The at least one processor 302 may further be configured to output the augmented reality view 405, 504 including the at least one physical item 401, 506 and the at least one digital item 402, 508. The at least one digital item 402, 508 in this non-limiting example does not physically exist where depicted in the augmented reality view 405, 504.

[0050] The at least one processor 302 may be further configured to generate a group indicator to group together at least one physical item 401, 506, at least one digital item 402, 508, the at least one anchor 408, 512, or any combination thereof. In some embodiments, the group indicator may be a border, a tag, a color, or any combination thereof.

[0051] In some embodiments, the at least one processor 302 may be configured to perform multicapture and / or auto-grouping. In some embodiments, the at least one processor 302 is configured identify potential digital groupings based on physical constructs, variations in the received image data, extracted content (for example, using computer vision (CV) and / or optical character recognition (OCR) techniques) of the plurality of items, or any combination thereof. In some embodiments, the user 126 may select a digital grouping. In some embodiments, a context of the items within the digital grouping is determined based on a collective content of the group.

[0052] In some embodiments, for the multi -capture, broader borders 410, 510 may be used to recognize individual items (i.e., the physical item 401, 506, the digital item 402, 508, and / or the multimodality notes). For auto-grouping, an organization strategy, as well as the items may be digitized. In some embodiments, such features may allow for movement between modalities (i.e., a physical modality, a digital modality, and a mixed-modality). In some embodiments, the use of the at least one physical item 401, 506, the mixed-modality notes, and / or the at least one digital item 402, 508 is provided to detect a theme / group either through a user input or automatic-detection (CV and / or OCR, for example). Anchoring to a physical location allows the digital content to be added remotely.

[0053] Movement between modalities may include by way of non-limiting example, creating a digital twin of a physical or mixed-modality Post-it® Note and / or projecting a digital Post-it® Note into physical space (such as a digital version that exists at a physical location within AR). This may include, for example, embodiments that digitize the organizational strategy (i.e., “borders” in some embodiments). This may utilize context-awareness based on (by way of non-limiting examples), the location of note(s) in the physical space, recognition of color groupings, and / or OCR of text on note(s). Detection of visual grouping may occur through an analysis such as via Visual Attention Model / Visual Attention Software (VAM / VAS such as provided in U.S. Patent 10,176,396), detection of visual borders and relative coordinates in (x,y,z) space, and / or by user manual selection.

[0054] Detection of themes, by way of non-limiting example, may include fine-tuning themes based on user-defined or automatically detected groups. For instance, if the content of the physical items 401, 506 are “bananas, apples, yogurt, pasta, marinara, ground beef’ and the user 126 makes a group, that theme may be “dinner”. If the user 126 then selects a subgroup of pasta,marinara, and ground beef, the at least one processor 302 may suggest tiered themes, which might be “groceries” and level 2 might be “dinner.”

[0055] FIG. 6 shows a user interface 604 displayed on a screen 602 of a mobile device 600, according to an embodiment of the present disclosure. The mobile device 600 may be similar to the mobile device 115 shown in FIG. 3 and may include similar components, for example, the screen 602 may be the display / output device(s) 304. The user interface 604 may illustrate a plurality of groups 606 and corresponding group indicators 608 automatically generated based on context, symbols, and / or extracted content from at least one physical item (e.g., the physical item 401, 506) or at least one digital item (e.g., the digital item 402, 508). The exemplary group indicators 608 are “To Do” and “Work Notes”.

[0056] In some embodiments, the at least one processor 302 is further configured to generate a bulk action 610 for a group 606 of items from the plurality of groups 606. The bulk action 610 includes sharing the group 606 of the items, digitizing the at least one physical item 401, 506 located within the group 606, copying the items within the group 606, or any combination thereof. The bulk action 610 may include sharing the group 606 of the items with another user. In some embodiments, the other mobile device 116 (shown in FIG. 1) may correspond to the other user. This could allow in some embodiments for the bulk action 610 of, for instance, sharing an entire group 606 of work notes with a colleague. Some embodiments could perform the bulk actions 610, such as digitize, copy, and project for the entire group 606.

[0057] In some embodiments, the “To Do” items list may appear next to the user’s work monitor (not shown) when viewing through the AR glasses (e.g., the augmented reality device 404). Some embodiments may provide for adding additional notes that are fully digital. Various embodiments may appear as anchored, mapped, aligned, registered, or the like to an identified physical object.

[0058] FIG. 7 illustrates a flowchart of a method 700 for compartmentalization recognition for multimodality notes, according to an embodiment of the present disclosure. It is to be noted that the method 700 is configured to be performed using the at least one system 300 shown in FIG. 3. Specifically, the method 700 is configured to be performed by the at least one computing device 301 of the at least one system 300. More specifically, the at least one non-transitory computer- readable medium 316 is configured to store instructions that, when executed, configure the at least one processor 302 to perform operations of the method 700. The method 700 will be set forth, by way of example, with reference to FIGS. 1 - 6.

[0059] Referring to FIGS. 1-7, at operation 702, the method 700 includes receiving digital item location data pertaining to at least one digital item (e.g., the digital item 402, 508). The at least one digital item may be an already existing item in a digital format and may be already stored in the data storage 168 of the mobile device 115 or created using the mobile device 115. The atleast one digital item may be a digital representative of a physical note which was previously captured by the camera of the mobile device 115. The digital item location data may refer to real / physical world location data used to show virtual objects, e.g., the at least one digital item. Therefore, the digital item location data may include data regarding a location at which the at least one digital item is to be displayed in an augmented reality view (e.g., the augmented reality view 405, 504).

[0060] At operation 704, the method 700 includes receiving image data of a physical workspace (e.g., the physical workspace 120) from a camera (e.g., the image capture device 118). The image data includes at least one physical item (e.g., the physical item 401, 506). Further, the digital item location data is mapped to the image data. In some embodiments, the physical item is a note or a stationery.

[0061] At operation 706, the method 700 includes outputting the augmented reality view (e.g., the augmented reality view 405, 504) including the at least one physical item and the at least one digital item. As discussed before, the at least one digital item does not physically exist where depicted in the augmented reality view.

[0062] At operation 708, the method 700 includes generating a group indicator (e.g., the group indicator 608) to group together the at least one physical item, the at least one digital item, at least one anchor (e.g., the at least one anchor 408, 512), or any combination thereof. In some embodiments, the method 700 further includes generating the group indicator 608 that is a border, a tag, a color, or any combination thereof.

[0063] In some embodiments, the at least one anchor is a specific location in the image data. The at least one anchor is detected within the physical workspace or specified by the user 126. In some embodiments, the method 700 further includes adding the at least one digital item to the at least one anchor. In some embodiments, adding the at least one digital item to the at least one anchor includes receiving a user input to add the at least one digital item to the at least one anchor when the at least one anchor is located remotely from the user 126.

[0064] In some embodiments, the method 700 further includes identifying potential digital groupings based on physical constructs, variations in the received image data, extracted content of the plurality of items (e.g., the at least one physical item and / or the at least digital item), or any combination thereof. In some embodiments, a potential digital grouping may indicate a group of items that may correspond to the group indicator generated at operation 708. In some embodiments, the method 700 further includes receiving the user input that selects a digital grouping. In some embodiments, the user input may correspond to a confirmation received from the user regarding the potential grouping identified by the at least one processor 302.

[0065] In some embodiments, the method 700 includes determining a context of items within the digital grouping as determined by collective content of the group. Specifically, the method 700includes determining the context of the at least one physical item or the at least one digital item. The context may be determined by OCR of content on the at least one note(s) or using a CV technique. The determination of the context may facilitate in generating the group indicator more efficiently.

[0066] In some embodiments, the method 700 further includes generating a bulk action (e.g., the bulk action 610) for the group of items that includes sharing the group of items, digitizing physical items located within the group, copying items within the group, or any combination thereof. In some embodiments, generating the bulk action for the group of items includes sharing the group of items with another user. In some embodiments, the bulk action may include different actions for different items based on content / context. For example, the bulk action may include sharing one or more group of items to a software application, such a notes application and the other group of items a different software application such as, an email application.

[0067] FIG. 8 shows a flowchart of a process 800 for compartmentalization recognition for multimodality notes, according to an embodiment of the present disclosure. The process 800 is embodied as one or more algorithms implemented by the at least one processor 302 of the at least one computing device 301 shown in FIG. 3. Further, the process 800 may be stored in the at least one non-transitory computer-readable medium 316 as instructions executable by the at least one processor 302.

[0068] Referring to FIGS. 1-8, at operation 802, the at least one processor 302 is configured to display the physical workspace 120 through the mobile device 115. At operation 804, the at least one processor 302 is optionally configured to receive a user input to select potential borders / groups identified in the physical workspace 120. In some embodiments, the user 126 may select one or more options to view potential borders / groups identified in the physical environment. In some cases, the at least one processor 302 is configured automatically identify potential border or group in the physical workspace 120. In such cases, the process 800 directly moves to operation 806 after operation 802 and operation 804 is skipped.

[0069] At operation 806, the at least one processor 302 is configured to capture an augmented reality view and identify potential border or group. In some embodiments, the at least one processor 302 may capture the augmented reality view and identify potential borders / groups based on physical constructs and variations in background, notes, or extracted content (e.g., via OCR, CV, and the like).

[0070] At operation 808, the at least one processor 302 determines if the user 126 selects the border / the group. If at operation 808, the user 126 selects the border / the group, the process 800 moves to operation 814. In some embodiments, determining if the user 126 selects the border / the group operation 808 may include receiving confirmation (e.g., based on a user input) on the automatically identified borders or groups by the user. In other words, the at least one processor302 is configured to receive the confirmation on the automatically identified borders or groups by the user.

[0071] At operation 814, the at least one processor 302 is configured to create a digital border against the at least one anchor (e.g., spatial anchors) created at the time of capturing the augmented reality view.

[0072] If at operation 808, the user 126 does not select the border / the group, the process 800 moves to operation 810. At operation 810, the at least one processor 302 is configured to receive borders from the user 126 by drawing on the augmented reality view. At operation 812, the at least one processor 302 is configured to determine if the drawn borders are saved by the user 126. If the drawn borders are not saved at operation 812, the process 800 moves to operation 802, where the process 800 starts again and the at least one processor 302 is configured to display the physical workspace 120 through the mobile device 115. If the drawn borders are saved at operation 812, the process 800 moves to operation 814.

[0073] FIG. 9 illustrates a flowchart of a process 900 for compartmentalizing the multi -modality notes, according to an embodiment of the present disclosure. For the same, the at least one anchor (e.g., the anchor 512 in FIG. 5) in the physical workspace may be detected automatically or may be specified by the user 126. The digital items may be added to the at least one anchor 512 remotely and can later be accessed when viewing the at least one anchor. For example, a user (e.g., the user 126 of FIG. 1) may attach a reminder note to a computer monitor (i.e., the anchor) when they think of a task they need to complete but they are away from their physical workspace (e.g., the physical workspace 120).

[0074] The process 900 is embodied as one or more algorithms implemented by the at least one processor 302 of the at least one computing device 301. Further, the process 900 may be stored in the at least one memory 310 as instructions executable by the at least one processor 302.

[0075] Referring to FIGS. 1 - 9, at operation 902, the at least one processor 302 is configured to display the physical workspace 120 with defined borders through the mobile device 115. At operation 904, the at least one processor 302 is configured to receive at least one user input corresponding to select the at least one physical item, the at least one anchor captured, or the at least one digital item (e.g., the digital notes), and to arrange them within the digital border creating a group.

[0076] At operation 906, the at least one processor 302 is configured to determine a context of a collective content of notes (i.e., the at least one physical item, the at least one anchor, and the digital items) within the group.

[0077] At operation 908, the processor 302 receives a user feedback regarding the group(s) having the physical item(s), anchor(s), and digital item(s).

[0078] At operation 910, the at least one processor 302 is configured to determine if the user 126 has moved at least one of the at least one physical item, the at least one anchor, or the digital itemsto another group. If, at operation 910, the user 126 has moved the items to the other group, the process 900 moves to operation 906. If, at operation 910, the user 126 has not moved the items to the other group, the process 900 moves to operation 912, where the at least one processor 302 is configured to allow the user 126 to continue working with a combination of the at least one physical item, the at least one anchor, or the digital notes within the group. In some embodiments, moving of the at least one of the at least one physical item, the at least one anchor, or the digital items may include changing at least one of the group indicator, such as, a label, a tag, or other group designation. In other words, moving the at least one of the at least one physical item, the at least one anchor, or the digital items to another group may include nonphysical moves, such as moves to new groups, for example, by changing the label, the tag, or the other group designation.

[0079] Referring to FIGS. 1-9, the system 300 and the method 700 for recognition and compartmentalization for the multi-modality notes is provided. The recognition of the at least one physical item, the at least one digital item, or the at least one anchor may facilitate creating the augmented reality view. The user 126 may easily access the physical as well as the digital items together (i.e., the multi-modality items) in the augmented reality view. Further, the system 300 and the method 700 may further facilitate compartmentalization or grouping of the items (i.e., the physical and / or digital items) based on the context. Hence, the user 126 may readily work with the groups. Further, the system 300 and the method 700 may provide flexibility to the user 126 for modifying the groups which are automatically created, thereby providing more control to the user 126. The at least one anchor 512 may help in creating and organizing the items as per required by the user 126.

[0080] Unless otherwise indicated, all numbers expressing feature sizes, amounts, and physical properties used in the specification and claims are to be understood as being modified by the term “about.” Accordingly, unless indicated to the contrary, the numerical parameters set forth in the foregoing specification and attached claims are approximations that can vary depending upon the desired properties sought to be obtained by those skilled in the art utilizing the teachings disclosed herein.

[0081] Although specific embodiments have been illustrated and described herein, it will be appreciated by those of ordinary skill in the art that a variety of alternate and / or equivalent implementations can be substituted for the specific embodiments shown and described without departing from the scope of the present disclosure. This application is intended to cover any adaptations or variations of the specific embodiments discussed herein. Therefore, it is intended that this disclosure be limited only by the claims and the equivalents thereof.

Claims

CLAIMS1. At least one non-transitory computer-readable medium storing instructions that, when executed, configure at least one processor to: receive digital item location data pertaining to at least one digital item; receive image data of a physical space from a camera, wherein the image data includes at least one physical item and the digital item location data is mapped to the image data; output an augmented reality view including the at least one physical item and the at least one digital item, wherein the at least one digital item does not physically exist where depicted in augmented reality; and generate a group indicator to group together at least one physical item, at least one digital item, at least one anchor, or any combination thereof.

2. The at least one non-transitory computer-readable medium of claim 1, including further instructions to identify potential digital groupings based on physical constructs, variations in the received image data, extracted content of the plurality of items, or any combination thereof.

3. The at least one non-transitory computer-readable medium of claim 1, including further instructions to receive user input that selects a digital grouping.

4. The at least one non-transitory computer-readable medium of claim 1, wherein an anchor is a specific location in the image data, wherein anchors are detected within the physical space or specified by a user.

5. The at least one non-transitory computer-readable medium of claim 4, including further instructions to add at least one digital item to the anchor.

6. The at least one non-transitory computer-readable medium of claim 5, including further instructions to receive user input to add a digital item to the anchor when the anchor is located remotely from the user.

7. The at least one non-transitory computer-readable medium of claim 1, including further instructions to determine context of items within the digital grouping as determined by collective content of the group.

8. The at least one non-transitory computer-readable medium of claim 1, including further instructions to generate a group indicator that is a border, a tag, a color, or any combination thereof.

9. The at least one non-transitory computer-readable medium of claim 1, wherein an item is a note or stationery.

10. The at least one non-transitory computer-readable medium of claim 1, including further instructions to generate a bulk action for a group of items that includes sharing a group of items with another user, digitizing physical items located within a group, copying items within a group, or any combination thereof.

11. The at least one non-transitory computer-readable medium of claim 1, wherein generating a bulk action for a group of items includes sharing a group of items with another user.

12. At least one system including: at least one computing device including one or more processors; and at least one memory coupled to at least one of the one or more processors, wherein the at least one memory includes instructions that configure the at least one computing device to: receive digital item location data pertaining to at least one digital item; receive image data of a physical space from a camera, wherein the image data includes at least one physical item and the digital item location data is mapped to the image data; output an augmented reality view including the at least one physical item and the at least one digital item, wherein the at least one digital item does not physically exist where depicted in augmented reality; and generate a group indicator to group together at least one physical item, at least one digital item, at least one anchor, or any combination thereof.

13. The at least one system of claim 12, including further instructions to identify potential digital groupings based on physical constructs, variations in the received image data, extracted content of the plurality of items, or any combination thereof.

14. The at least one system of claim 12, including further instructions to receive user input that selects a digital grouping.

15. The at least one system of claim 12, wherein an anchor is a specific location in the image data and wherein anchors are detected within the physical space or specified by a user.

16. The at least one system of claim 15, including further instructions to add at least one digital item to the anchor.

17. A method including: receiving digital item location data pertaining to at least one digital item; receiving image data of a physical space from a camera, wherein the image data includes at least one physical item and the digital item location data is mapped to the image data; outputting an augmented reality view including the at least one physical item and the at least one digital item, wherein the at least one digital item does not physically exist where depicted in augmented reality; and generating a group indicator to group together at least one physical item, at least one digital item, at least one anchor, or any combination thereof.

18. The method of claim 17 further including identifying potential digital groupings based on physical constructs, variations in the received image data, extracted content of the plurality of items, or any combination thereof.

19. The method of claim 17 further including receiving user input that selects a digital grouping.

20. The method of claim 17 wherein an anchor is a specific location in the image data and wherein anchors are detected within the physical space or specified by a user.

Images