SYSTEM AND METHOD FOR MANAGING DIGITAL NOTES FOR COLLABORATION - Patent application

Abstract

A method for projecting, detecting, and processing digital notes. The digital notes are displayed on a mixed reality device in combination with a physical note in the same space. The physical notes are automatically detected in the field of view of a camera and converted to a corresponding digital note, and any obstructions in the note are removed in the digital note. The digital notes are projected onto a wall, and when a change in the scene is detected, a new scene is projected onto the wall based on the change. The physical notes are automatically detected in the field of view of a camera and converted to a digital note, and audio associated with the digital note is recorded.

Description

[Background technology]

[0001] Paper notebooks are widely used to record, share, and communicate ideas and information. For example, in a collaboration session (e.g., a brainstorming session), participants write ideas on rearrangeable paper notebooks, whiteboards, or pieces of paper and share them with each other. People also commonly use notebooks on a daily basis to record information or content that they do not want to forget. As a further example, people frequently use notebooks as reminders for future actions or events, such as making a phone call, revising a document, or filling out a timesheet.

[0002] Software programs currently exist that allow computer users to create software-based notes in digital form and utilize the digital notes within a computing environment For example, a computer user can create a digital note and "attach" the digital note to an electronic document, desktop, or electronic workspace presented by the computing environment. Summary of the Invention

[0003] A method for synchronizing digital and physical notes involves displaying the digital notes on a wall or surface, capturing the physical notes via an image capture device, and converting them into corresponding digital notes. For in-person users, the digital notes are displayed alongside the physical notes via projection, mixed reality devices, or mobile devices. For digital users, all notes are also displayed on a synchronized electronic board.

[0004] A method for generating a digital note includes automatically detecting a physical note via an image capture device. The detected physical note is converted into a corresponding digital note. If the note has an obstruction, the obstruction is removed in the digital note.

[0005] A method for displaying a digital note includes projecting the digital note onto a wall or surface for a user and detecting a change in a scene on the wall or surface via an image capture device. The scene is updated based on the detected change. Based on the update, a new scene is projected onto the wall or surface, and the new scene is shared on a common electronic board for the users.

[0006] A method for generating digital notes with audio includes automatically detecting a physical note on a wall or surface via an image capture device, converting the detected physical note into a corresponding digital note, recording audio associated with the digital note, tagging the audio with the digital note, and displaying the digital note with the tag associated with the audio on an electronic board. [Brief description of the drawings]

[0007] [Figure 1A] FIG. 1A illustrates an example of a user using an image capture device of a mobile device to take a picture of notes in a workspace.

[0008] [Figure 1B] FIG. 1B is a block diagram illustrating an example of a mobile device.

[0009] [Figure 1C] FIG. 1C is a block diagram illustrating an example of a note management application running on a mobile device.

[0010] [Figure 1D] FIG. 1D illustrates another embodiment of a note recognition system.

[0011] [Figure 1E] FIG. 1E illustrates a note management system according to another embodiment.

[0012] [Figure 2A]FIG. 2A illustrates a system for integrating physical and digital notes into a single working canvas.

[0013] [Figure 2B] FIG. 2B is a flow chart illustrating a method for integrating physical and digital notes into a single working canvas.

[0014] [Figure 3A] FIG. 3A illustrates a system for note recognition with physical obstructions.

[0015] [Figure 3B] FIG. 3B is a flow chart illustrating a method for note recognition with physical obstructions.

[0016] [Figure 4A] FIG. 4A illustrates a system for reducing data consumption by retrieving only the changed content of a note.

[0017] [Figure 4B] FIG. 4B is a flow chart illustrating a method for reducing data consumption by retrieving only the changed content of a note.

[0018] [Figure 5A] FIG. 5A illustrates a system for synchronizing note taking with a real-time audio and visual presentation.

[0019] [Figure 5B] FIG. 5B is a flow chart illustrating a method for synchronizing note taking with a real-time audio and visual presentation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] overview

[0021] This disclosure describes techniques for creating and manipulating software notes that represent physical notes. For example, techniques are described for recognizing physical notes present in a physical environment, obtaining information therefrom, and creating corresponding digital representations, i.e., digital or software-based notes. Additionally, at least some aspects of this disclosure relate to techniques for managing multiple notes.

[0022] Generally, notes include physical notes and digital notes. A physical note generally refers to an object with a boundary and a recognizable content. A physical note includes objects that are the result of people writing, drawing, or other types of input on the object. For example, paper, whiteboards, or objects that accept other inputs. For example, physical notes may include handwritten, repositionable paper notebooks, paper or film, whiteboards with drawings, posters, and signs. In some cases, physical notes may be generated using digital means, for example, printed on printable, repositionable paper notebooks or printed documents. Also, a single object may contain multiple notes. For example, multiple ideas may be written on poster paper or whiteboards. Physical notes may be two-dimensional or three-dimensional and may have various shapes and sizes. For example, a physical note may be a 3" x 3" notebook, a 26" x 39" poster, or a triangular metal sign. In some cases, a physical note has a known shape and / or size. Digital notes generally refer to digital objects that contain information and / or ideas. Digital notes are generated using digital inputs including, for example, a keyboard, a touch screen, a digital camera, a digital recording device, a stylus, a digital pen, etc. In some cases, a digital note may represent a physical note.

[0023] Note Management System

[0024] FIG. 1A illustrates an example of a note recognition environment 10. In the example of FIG. 1A, the environment 10 includes a mobile device 15 for retrieving and recognizing one or more notes 22 from a workspace 20. As described herein, the mobile device provides an execution environment for one or more software applications that can efficiently retrieve and extract note content from a large number of physical notes, such as a collection of notes 22 from the workspace 20. In this example, the notes 22 may be the result of a collaborative brainstorming session with multiple participants. As described herein, the mobile device 15 and software executing thereon can perform various note-related operations, including the automatic creation of digital notes that represent the physical notes 22 in the workspace 20.

[0025] In this example embodiment, mobile device 15 includes, among other components, image capture device 18 and display device 28. Additionally, although not shown in FIGURE 1A, mobile device 15 may include one or more processors, microprocessors, internal memory and / or data storage, and other electronic circuitry for executing software or firmware to provide the functionality described herein.

[0026] Generally, the image capture device 18 is a camera or other component configured to capture image data representative of the workspace 20 and the notes 22 disposed therein. In other words, the image data captures a visual representation of a plurality of visual notes having an environment, such as the workspace 20. Although described as a camera of the mobile device 15, the image capture device 18 may include other components capable of capturing image data, such as a video recorder, an infrared camera, a CCD (charge-coupled device) array, a laser scanner, a light detection and ranging technology (LiDAR), and the like. Additionally, the captured image data may include images, videos, a series of images (i.e., multiple and / or ordered images taken within a period of time), a collection of images, information from other sensors, data that may include depth data, and the like. The term input image is used herein to refer to these various types of image data.

[0027] Display device 28 may include, but is not limited to, an electronically addressable display (e.g., a liquid crystal display (LCD) or other type of display device). In some embodiments, mobile device 15 generates note content for display on display device 28 in a variety of formats, such as list format, grouped into rows and / or columns, flow diagram format, etc. In some cases, mobile device 15 may also communicate display information for display by other devices, such as a tablet computer, projector, electronic bulletin board, or other external device.

[0028] As described herein, the mobile device 15 and software executing thereon provide a platform for creating and manipulating digital notes representing the physical notes 22. For example, the mobile device 15 is generally configured to process image data generated by the image capture device 18 to detect and recognize at least one of the physical notes 22 disposed within the workspace 20. In some examples, the mobile device 15 is configured to recognize the note by determining the general boundaries of the note. Once the note is recognized, the mobile device 15 extracts at least one content of the one or more notes. This content is visual information of the note 22.

[0029] In some embodiments, mobile device 15 provides functionality that allows user 26 to export digital notes to other systems, such as cloud-based repositories (e.g., cloud server 12) or other computing devices (e.g., computer system 14 or mobile device 16).

[0030] 1A , mobile device 15 is shown as a mobile phone, but in other examples, mobile device 15 may be other types of mobile or non-mobile computing devices suitable for performing the techniques described herein, such as a tablet computer, a personal digital assistant (PDA), a laptop, a media player, an e-reader, a wearable computing device (e.g., a watch, eyewear, a glove), or the like.

[0031] 1B shows a block diagram of an example of a mobile device that operates in accordance with the techniques described herein. By way of example, the mobile device of FIG. 1B will be described with reference to mobile device 15 of FIG. 1A.

[0032] In this example, the mobile device 15 includes various hardware components that provide basic functions necessary for the operation of the device. For example, the mobile device 15 includes one or more programmable processors 70 configured to operate according to executable instructions (i.e., program code), which are typically stored in a computer-readable medium or data storage 68 (e.g., a static random access memory (SRAM) device or a flash memory device). The I / O 76 can include one or more devices, such as a keyboard, a camera button, a power button, a volume button, a home button, a back button, a menu button, or the display device 28 described in FIG. 1A. The transmitter 72 and the receiver 74 provide wireless communication with other devices, such as the cloud server 12, the computer system 14, or other mobile devices 16, as described in FIG. 1A, via a wireless communication interface. This includes, but is not limited to, high frequency radio frequency (RF) signals. The microphone 71 converts audio information into corresponding electrical signals. The speaker 73 converts electrical signals into corresponding audio information. The vibration motor 75 is used to vibrate the mobile device 15 or its housing. Mobile device 15 may include additional discrete digital logic or analog circuitry, not shown in FIG. 1B.

[0033] Generally, operating system 64 executes on processor 70 and provides an operating environment for one or more user applications 77 (commonly referred to as "apps"), including a note management application 78. User applications 77 may include, for example, executable program code stored on a computer-readable storage device (e.g., data storage 68) for execution by processor 70. As another example, user applications 77 may include firmware or, in some examples, may be implemented in discrete logic.

[0034] In operation, the mobile device 15 receives input image data and processes the input image data according to the techniques described herein. For example, the image capture device 18 captures an input image of a note 22 in an environment that includes multiple notes, such as the workspace 20 of FIG. 1A. The mobile device 15 may also receive image data from an external source, such as the cloud server 12, the computer system 14, or the mobile device 16, via the receiver 74. In general, the mobile device 15 stores the image data in the data storage 68 for access and processing by the note management application 78 and / or other user applications 77.

[0035] 1B, the user application 77 may call kernel functions of the operating system 64 to output a graphical user interface (GUI) 79 to present information to a user of the mobile device. As described further below, the note management application 78 may construct and control the GUI 79 to provide an improved electronic environment for creating and manipulating corresponding digital notes that represent the physical notes 22. For example, the note management application 78 may construct the GUI 79 to include mechanisms that allow the user 26 to easily control events that are automatically triggered upon capturing a note with a particular characteristic. Additionally, the note management application 78 may construct the GUI 79 to include mechanisms that allow the user 26 to manage relationships between groups of digital notes.

[0036] 1C is a block diagram illustrating an example embodiment of a note management application 78 that operates in accordance with the techniques described herein. Although described as a user application 77 running on a mobile device 15, the example described herein may be implemented on any computing device, such as a cloud server 12, a computer system 14, or another mobile device.

[0037] In this example, the note management application 78 includes an image processing engine 82 that provides image processing and object recognition functionality. The image processing engine 82 may include an image communication module 90, a note identification module 86, and a digital note generation module 88. Additionally, the image processing engine 82 includes an image processing application programming interface (API) 95, which provides a library of image manipulation functions, such as image thresholding, masking, filtering, edge detection, machine learning / artificial intelligence systems, etc., for use by other components of the image processing engine 82.

[0038] Typically, image data is stored in data storage device 68. In this example, note management application 78 stores images 97 in data storage device 68. Each of images 97 may comprise pixel data for an environment that includes multiple physical images, such as workspace 20 of FIG. 1A.

[0039] As described herein, the note identification module 86 processes the image 97 and identifies (i.e., recognizes) a number of physical notes in the image. The digital note generation module 88 generates digital notes 99 corresponding to the physical notes recognized in the image 97. For example, each of the digital notes 99 corresponds to one of the physical notes identified in the input image 97. During this process, the digital note generation module 88 may update the database 94 to include records of the digital notes and store in the database information (e.g., content) extracted from the input image within the boundaries of the physical notes detected by the note identification module 86. Additionally, the digital note generation module 88 may store metadata in the database 94 that associates the digital notes with one or more groups of digital notes. The note identification module 86 may also perform note tracking to track the digital notes throughout the processes described herein, such as in embodiments 1-4 described below. Tracking may include, for example, the location of the digital notes, the state of the digital notes, changes to the digital notes, the creator of the digital notes, or other people who have made updates to the digital notes.

[0040] Additionally, the note management application 78 may be configured to specify rules 101 that trigger actions in response to detection, for example, by user input 26, of a physical note with a particular characteristic. For example, the user interface 98 may map actions to particular characteristics of a note based on user input. The note management application 78 may output a user interface 98 through which the user may specify actions, including note grouping actions and actions related to other software applications running on the mobile device, for example, actions related to a calendar application. For each rule, the user interface 98 may enable the user to define criteria for triggering the action. During this configuration process, the user interface 98 may prompt the user to capture image data representing the example note for triggering the action, and process the image data to extract features such as color and content. The user interface 98 may then present the determined criteria to the user to define rules corresponding to the example note.

[0041] The image communication module 90 controls the communication of image data between the mobile device 15 and external devices (e.g., the cloud server 12, the computer system 14, the mobile device 16, or the image capture device 18). In some examples, the image communication module 90 may enable a user to communicate, for example, processed or unprocessed images 97 of an environment and digital notes and associated information (including metadata) extracted therefrom. In some examples, the image communication module 90 may export this data to a zip file and communicate it via FTP, HTTP, email, Bluetooth, or other mechanisms.

[0042] In the example of FIG. 1C, the note management application 78 includes a user interface 98, which constructs and controls the GUI 79 (FIG. 1B). As described herein, the user interface 98, in some examples, outputs for display an input image 97 with a plurality of superimposed digital notes 99, each superimposed in the position of a corresponding physical note. The user interface 98 may also display a group of digital notes 99 designated by the user. The group of digital notes 99 may be a subset of the digital notes recognized in a particular input image 97. The user interface 98 displays the designated group (set) of digital notes in another portion of the GUI 79, and allows the user 26 to easily add or remove digital notes 99 from the designated group.

[0043] In some embodiments, the user interface 98 may provide an image editor 96 that allows the user to edit the overlay image and / or the digital note. In another example, the digital note generation module 88 may include a process or processes that enhance the information extracted from the input image.

[0044] FIG. 1D illustrates another example embodiment of a note recognition system 100A. The system 100A may include a processing unit 110, one or more notes 120, a sensor 130, and a note content repository 140. The processing unit 110 may include one or more processors, microprocessors, computers, servers, and other computing devices. The sensor 130 may be, for example, an image sensor and is configured to obtain a visual representation of a scene including one or more notes 120. The sensor 130 may include at least one of a camera, a video recorder, an infrared camera, a CCD (charge-coupled device) array, a scanner, and the like. The visual representation may include an image, a video, a sequence of images (i.e., multiple and / or ordered images taken within a period of time), a collection of images, and the like. The processing unit 110 is connected to the sensor 130 and configured to receive the visual representation. In some cases, the processing unit 110 is electronically connected to the sensor 130. The processing unit 110 is configured to recognize at least one of the one or more notes 120 from the visual representation. In some embodiments, processing unit 110 is configured to recognize a note by determining a general boundary of the note. After the note is recognized, processing unit 110 extracts the content of the note. In some cases, processing unit 110 is configured to recognize and extract the content of multiple notes from a visual representation of a scene that contains those notes.

[0045] In some cases, the processing unit 110 may execute software or firmware stored on a non-transitory computer-readable medium to perform various processes for the system 100A, such as recognizing notes and extracting notes. The note content repository 140 may operate on a single computer, server, storage device, cloud server, etc. In other cases, the note content repository 140 may operate on a series of networked computers, servers, or devices. In some embodiments, the note content repository 140 includes a hierarchy of data storage devices, including local, regional, and central. The notes 120 may include physical notes arranged in an orderly or random manner in the collaboration space, and the sensors 130 generate a visual representation of the notes 120 in the collaboration space.

[0046] In some embodiments, the note recognition system 100A may include a display device (not shown in FIG. 1D ) to indicate to a user which notes have been recognized or whose contents have been extracted. Additionally, the note recognition system 100A may present the extracted contents via the display device. In some embodiments, the processing unit 110 may authenticate the note before extracting the note contents. Once the note is authenticated, its contents are extracted and stored in the note content repository 140.

[0047] FIG. 1E illustrates an embodiment of a note management system 100B. In this embodiment, the note management system 100B includes a processing unit 110, one or more notes 120, one or more note sources 150, and a note content repository 140. In some cases, the system 100B includes a display device 160. The processing unit 110, the notes 120, and the note content repository 140 are similar to the components of the note recognition system 100A illustrated in FIG. 1A. The note sources 150 may include a source providing physical note content, such as a visual representation of a scene including one or more notes, and a source providing digital note content, such as a data stream entered from a keyboard. In some embodiments, the note management system 100B includes a first source and a second source, where the first source is a visual representation of a scene including one or more notes 120. The first source and the second source are generated by different devices. The second source includes at least one of a text stream, an image, a video, a file, and a data entry. Processing unit 110 recognizes one or more notes from a first source and extracts their contents, as described in note recognition system 100A. In some cases, processing unit 110 labels the notes with categories. Processing unit 110 may label the notes based on the particular shape, color, content, and / or other information of the notes. For example, notes in each group may have a different color (e.g., red, green, yellow, etc.).

[0048] In some embodiments, the note management system 100B may include one or more display devices 160 for displaying the contents of the note 120 to a user. The display devices 160 may include, but are not limited to, electronically addressable displays (e.g., liquid crystal displays (LCDs), tablet computers, projectors, electronic bulletin boards, mobile phones, laptops, etc.). In some examples, the processing unit 110 generates content for displaying the contents of the note on the display devices 160 in various formats, such as in a list format, grouped into rows and / or columns, in a flow diagram format, etc.

[0049] Each component of the note recognition system and the note management system, such as the processing unit, the image sensor, and the note content repository, can communicate through a communication interface. The communication interface includes, but is not limited to, short-range and long-range wired or wireless communication interfaces. The short-range communication interface may include, for example, an interface conforming to a known communication standard, such as a local area network (LAN), a Bluetooth standard, an IEEE 802 standard (e.g., IEEE 802.11), a specification based on the ZigBee or IEEE 802.15.4 standard, or other public or proprietary wireless protocols. The long-range communication interface may include, for example, a wide area network (WAN), a cellular network interface, a satellite communication interface, and the like. The communication interface may be in a private computer network, such as an intranet, or on a public computer network, such as the Internet.

[0050] Collaborate on notes with video capture

[0051] The first to fourth embodiments described below can utilize instant communication capabilities by holding one or more physical notes up to a camera on a participant's device or by detecting the physical notes within the camera's field of view and using a software application that converts the physical notes into corresponding digital notes.

[0052] This instant communication feature is a method of instantaneous communication via a physical notebook, dry erase whiteboard, or other physical product that is viewed through a camera (e.g., image capture device 18). The camera may be, for example, a webcam, computer, cell phone, or similar digital device camera. This method allows for faster communication with groups or individuals without having to log into a system or type through a mail system to send notes or content digitally. This method does not require the use of a keyboard or cursor control device, and allows, for example, a sketch to be drawn on a whiteboard or physical notebook that is detected via the camera and converted into a corresponding digital note that is then viewed by other participants.

[0053] For example, by utilizing cameras already in use by participants working remotely or using laptops, this instant communication feature provides a way for participants to simply write in a physical notebook and hold the note up to the camera where it is captured by a software application and converted into a corresponding digital note. That digital note can then be added to the digital tool or sent to the facilitator for continued collaboration. Participants can also point the camera at a whiteboard to add sketches or content, which is then converted into a corresponding digital note and sent to the facilitator and added to the group for further collaboration. The camera can also be a smart camera that is mounted facing the dry erase wall in the conference room. By simply placing a physical note on the wall, the camera automatically captures the note and converts it into a corresponding digital note to share with remote participants.

[0054] In one example of this instant communication method, a participant or other user writes in a physical note and holds the physical note up to a camera that is already on, and a digital version of the note appears in a digital collaboration tool, such as the collaboration feature as described herein. The physical note is converted by a software application that automatically detects the presence of the physical note in the camera view and converts it into a corresponding digital note. This conversion can be done as described in the Note Management System section above.

[0055] The notes are automatically detected by using machine learning techniques to process video frames in a video feed from a camera (e.g., image capture device 18) and detect objects in the video frames that are recognized as notes. The video frames may be captured and processed from the video feed at the server where the video conference is taking place. Alternatively, the video frames may be captured and processed at the client side by using a software application as a virtual camera linked to the camera and routing the video feed through the software application.

[0056] Participants or users may also scan written text and move the scanned text into a video conference "chat" window. Written text may include authentic marks on the writing surface (e.g., physical notebook, dry erase board) and require its own note, such as an App Clip code (or other machine-readable optical label), logo, or printed graphic design. Another option is for participants or users to draw a rectangle around notes in a notebook or other writing surface to indicate the content to be captured as a note. It is also possible to capture multiple physical notes simultaneously. Other content that can be captured along with converting a physical note to a corresponding digital note includes an audio description of the physical note the participant is holding in the camera view, the author of the note, and possibly metadata such as a date and timestamp. Also of interest may be the angle of orientation (e.g., diamond-shaped vs. square), placement on the board, and placement relative to other notes. If a note is created or placed in a group, group metadata such as the size of the group, the time of creation, and the number of notes may be associated with the created note. If a note is moved, further metadata may include the note's route, the driver, the distance traveled, and a Natural Language Processing (NLP) sentiment analysis of the new adjacent notes relative to the previous adjacent notes. Metadata may also include other aspects or characteristics of the note. The captured physical note can also be used to indicate subsequent actions or organize notes under different categories (e.g., new action items, new ideas, or delegation to others).

[0057] The term "instant" is used solely as a label for this instant communication feature, and the conversion of physical notes in the camera view to corresponding digital notes typically occurs in real time, but does not have to be instantaneous.

[0058] 1. Integrate physical and digital notes into a single work canvas

[0059] 2A is a diagram of a system for integrating physical and digital notes into a single working canvas. The system includes a digital camera (e.g., digital image capture device) with a projector 204 that is used to project digital notes 202 onto a wall or other surface 200. A mixed reality device 206 is used to display an enhanced version of the projected digital notes 202, and the digital notes are also provided to a shared synchronized electronic board 208.

[0060] FIG. 2B is a flow chart of a method for integrating physical and digital notes into a single working canvas using the system of FIG. 2A. The method can be implemented in software or firmware modules executed by a processor. In this method, a projector in the camera and projector 204 projects a digital note 202 onto a wall or other surface 200 (step 210). A digital camera in the camera and projector 204 captures a physical note on the wall or surface 200 in the same space and converts it into a corresponding digital note (step 212). The note can be converted, for example, as described in the Note Management System section above. The digital note is displayed in combination with the physical note on the mixed reality device 206 and is displayed in the space when the wall or surface 200 is viewed through the mixed reality device 206 (step 214).

[0061] Hand gestures are optionally tracked (using the mixed reality device's camera and other sensors) and interpreted as commands related to the digital content, allowing the user to interact with the digital content (step 215). The digital notes are also displayed on the common board 208 for the user during the session (step 216). As the user interacts with the digital notes on the common board 208, the common board 208 is updated (step 218). The common board 208 can be implemented via a shared electronic screen or user interface, for example in video conferencing technology. The common board 208 may also exist in a virtual space using Microsoft Corporation's Mesh App.

[0062] The mixed reality device 206 is optional, and the working canvas (common board 208) can be created using a projector without the mixed reality device 206. However, in this embodiment, using a mixed reality device enhances the experience. A mixed reality device may include an augmented reality (AR) device, a virtual reality (VR) device, or both an AR and a VR device. Examples of an AR or VR device include AR or VR goggles and smart glasses.

[0063] In this embodiment, through video capture and display technology, physical and digital rearrangeable notes are integrated into a single interface, which can be shared and manipulated by users in different locations. A physical note in one location is displayed as a digital version in other locations, adjacent to other physical notes in each location by users participating in the video conference. Real-time video capture at each location keeps the integrated collection of notes updated with changes from each user. This integrated collection of notes can be displayed through a variety of display technologies. A projector mounted in the room projects images of the digital notes onto the walls, mixing them with the physical notes on the walls. A mixed reality device displays the digital notes to the user and combines them with the physical notes in the user's space, and images of the physical notes are displayed on the screens of computers and mobile devices, combined with representations of the digital notes from other locations. Notes can also include fiducials, identifiers, or other information to synchronize user devices to the same digital channel. Identifiers can be implemented, for example, as the unique color, position, and location of the notes.

[0064] As distributed teams become the norm or a more common mode of working, there is a demand for technology that helps teams have the feeling that they are working on the same material. This note capture technology, combined with advances in processing and display technology, can provide a unified solution to this need. This applies to both synchronous and asynchronous work, allowing teams working together from different locations to contribute both physical and digital notes simultaneously to the work product and view the combined contributions as a single deliverable from each location. A user in one location can create a collection of physical notes at that location and continue that work in another location, adding new physical notes to the previous collection, which will then be displayed as a digital representation in the new location.

[0065] 2.Note recognition with physical obstacles

[0066] 3A is a diagram of a system for note recognition with physical obstacles. The system includes a digital camera (e.g., a digital image capture device) that includes a physical note 220 held in front of (within the field of view of) the camera 222.

[0067] FIG. 3B is a flow chart of a method for note recognition with physical obstructions. The method may be implemented in a software or firmware module executed by a processor. In this method, a physical note 220 is detected within the field of view of a camera 222 (step 224). This may be done as described above. The process may optionally track the corresponding physical note and determine a moment of capture at which the physical note was detected within the field of view of the camera (step 225). Tracking the note may include, for example, locating the physical note when it is within the field of view of the camera. The moment of capture may include, for example, a date and time at which the physical note was detected to be converted into a corresponding digital note. When a user lifts the physical note 220, a portion of the note may be obstructed, for example, by the user's finger. The physical note 220 is converted into a corresponding digital note with the obstruction (step 226), and the obstruction is removed in the digital note (step 228). The process may automatically detect physical notes in front of the camera, as described above, some of which may include obstructions, such as the user's finger. The process need not only detect physical notes with obstructions.

[0068] Holding a physical notebook in front of a camera is a method for rapidly capturing multiple notes in succession. This function requires the user to hold each note, which may result in portions of the notebook's surface being obstructed from the camera's view. The system recognizes the obstruction as a human finger (or other obstruction) and "fills in" the missing note area, simulating the capture of the full note. In particular, through machine learning, computer vision, or other artificial intelligence, as previously described, the capture engine is trained to recognize that a human finger (or other obstruction) is partially obstructing the entire shape of the note, and to replace the obstructed area with a full note image. Other obstructions may include, for example, clips or other mechanical devices that hold the note in place.

[0069] 3. Reduce data consumption by capturing only the changes in your notes

[0070] 4A is a diagram of a system for reducing data consumption by capturing only the changed content of notes. The system includes a digital camera (e.g., a digital image capture device) with a projector 234 that is used to project digital notes 232 onto a wall or other surface 230. The digital notes are also provided to a shared electronic board 236.

[0071] FIG. 4B is a flow chart of a method for reducing data consumption by capturing only the changed content of the note. The method can be implemented in a software or firmware module executed by a processor. In this method, a projector in the camera and projector 234 projects the digital note 232 onto a wall or other surface 230 (step 238). If a change is detected in the scene on the wall or surface 230 via the camera in the camera and projector 234 (step 240), the scene is updated based on the detected change (step 242). The projector in the camera and projector 234 projects the new updated scene onto the wall or surface 230 and shares the new scene on the common board 236 (step 244). The common board 236 can be implemented via a shared electronic screen or user interface, for example, in video conferencing technology. A change in the scene may include, for example, detection of a new digital note in the scene, removal of one of the digital notes from the scene, a change in the content of the digital note in the scene, or a change in the position of the digital note in the scene.

[0072] Through machine learning, computer vision, or other artificial intelligence, the real-time video capture engine is trained to only recognize physically relocatable notes in the physical environment and to update only when the note content, location, or quantity changes. All other changes in the environment are ignored, greatly reducing the amount of data required to keep the note content updated with the latest work. This embodiment could also be implemented in part using, for example, a human occlusion code API, or the like, to enhance or facilitate detection of changes in the environment.

[0073] As distributed teams become the norm or a more common mode of working, capturing work products in an easily transmittable format may become a more critical need. Real-time video capture of notes in a physical space allows the entire work product to be maintained in a digital format. Capturing the entire environment is data intensive, taxing processing, storage, and energy usage. Training the system to only monitor notes can improve the efficiency and usability of the entire system.

[0074] 4. Capture and sync notes with real-time audio and visual presentations

[0075] 5A is a diagram of a system for synchronizing the capture of notes with real-time audio and visual presentation. The system includes a digital camera (e.g., a digital image capture device) with a microphone 254 that is used to project digital notes 252 onto a wall or other surface 250. Digital notes are also provided to an electronic board 256.

[0076] FIG. 5B is a flow chart of a method for synchronizing the capture of notes with real-time audio and visual presentation. The method can be implemented in software or firmware modules executed by a processor. In this method, a camera in a camera with microphone 254 detects a new physical note in a scene on a wall or surface 250 (step 258). The physical note is converted to a corresponding digital note (step 260). This conversion can be performed as described in the Note Management System section above. Audio associated with the physical note is recorded, optionally converted to text, and tagged to the corresponding digital note (step 262). The recorded audio may be electronically associated with or stored with the corresponding digital note. The digital note is displayed on the board 256 along with a tag for the associated audio (step 264). The tag may include, for example, an icon that the user can select to access the recorded audio or optional corresponding text. The board 256 can be implemented, for example, with a screen or user interface accessible to the user.

[0077] As distributed teams become the norm or more common working mode, capturing the complete meeting experience becomes a more critical need. Notes can be synchronized to real-time audio to provide contextual information that led to the creation of each note. In addition, many work sessions also include visual presentation materials that can be synchronized to notes and audio. This capability allows users in remote locations and different time zones to effectively experience the "real" background behind each physical note. Video capture of environments that include physical, relocatable notes will be enhanced to include simultaneous capture of in-room audio and synchronized visual presentation materials of the user's choosing.

[0078] This embodiment may optionally use speaker recognition to associate only the author's audio data with the note. This embodiment may also optionally use artificial intelligence (AI) techniques to search meeting transcripts for specific timespans of 1-2 minutes before and after the note to find conversations that may be relevant. That audio and image data is then stored with the corresponding note. The user may also be prompted to review associated metadata, and the AI ​​process may improve based on feedback, saving storage space by deleting incorrectly tagged video files.

Claims

1. A method for displaying physical and digital notes, which is performed by a processor. Steps include projecting a digital note onto a wall or surface for the user, The steps include acquiring a physical note on the wall or surface via an image acquisition device, and converting the acquired physical note into a corresponding digital note, The steps include displaying the digital note and the physical note together on the mixed reality device when the wall or surface is displayed through the mixed reality device, The steps include displaying the digital note on a common electronic board for the user, Methods that include...

2. The method according to claim 1, wherein the step of displaying on the mixed reality device includes the step of displaying a combination of the digital notebook and the physical notebook on the augmented reality device.

3. The method according to claim 1, wherein the step of displaying on the mixed reality device includes the step of displaying a combination of the digital notebook and the physical notebook on the virtual reality device.

4. The method according to claim 1, further comprising the steps of detecting a gesture from a user of the mixed reality device via the mixed reality device, and interpreting the detected gesture as a command.

5. The method according to claim 1, further comprising the step of updating the common electronic board when a user interacts with the displayed digital note.

6. A method for generating a digital note, which is executed by a processor. A step of automatically detecting a physical notebook via an image acquisition device, The steps include converting the detected physical notebook into a corresponding digital notebook, If the physical notebook has a defect, the step of removing the defect using the digital notebook, Methods that include...

7. The method according to claim 6, further comprising the step of filling in the area of ​​the note corresponding to the defect in order to simulate a complete note in the digital note.

8. The method according to claim 6, further comprising the step of tracking the detected physical note.

9. The method according to claim 6, further comprising the step of determining the moment when the physical note is detected.

10. A system for collaboration using a notebook, comprising a processor configured to perform any of the methods of claims 1 to 9.

Images