Systems and methods for managing digital records

Abstract

A method for managing either digital records or images of physical records. The method uses optical character recognition to convert handwritten content into characters and icons. The method also determines the size of a non-square physical record when imaged from various angles by a camera. For bent or damaged records, such as records with missing corners, the method detects and processes edges of the physical record in the image and uses the detected edges to convert the image of the physical record into a corresponding digital record that is not bent or damaged.

Description

[Background technology]

[0001] Paper records have been widely used to record, share, and communicate ideas and information. For example, during collaborative sessions (e.g., brainstorming sessions), participants write ideas on repositionable sticky notes, whiteboards, or pieces of paper, and then share them with each other. Additionally, people commonly use records throughout the day to memorialize information or content that the individual does not want to forget. As another example, people frequently use records as reminders for future actions or events, such as making a phone call, revising a document, or filling out a timesheet.

[0002] Software programs now exist that enable computer users to create software-based records in digital form and make the digital records available within a computing environment. For example, a computer user can create a digital record and "attach" the digital record to an electronic document, desktop, or electronic workspace presented by the computing environment. Summary of the Invention

[0003] A first method of managing records includes receiving a plurality of digital records, each including an image of handwritten content; assembling the plurality of digital records into a single electronic document; sending the electronic document to an optical character recognition (OCR) service or an application for using OCR to convert the handwritten content into text; receiving from the OCR service or application an electronic document having results of converting the handwritten content into text using OCR; and electronically separating the plurality of digital records from the electronic document and assigning the results of the OCR to digital records having corresponding handwritten content.

[0004] A second method of managing records includes displaying the digital record on an electronic display device, receiving a command to delete or erase the digital record, and displaying a digital representation and animation of the digital record being crumpled in response to the command.

[0005] A third method of managing records includes receiving an image of a physical record taken from a non-zero angle, detecting corners of the physical record in the image, recalculating the corners according to projections that match a plurality of camera angles, calculating a score for the corner based on the recalculating step at each of the camera angles, selecting a projection angle for the corner based on the calculating step, and calculating a size of the digital record based on an aspect ratio of the digital record at the selected projection angle.

[0006] A fourth method of managing records includes receiving an image of a physical record; detecting an edge of the physical record in the image; selecting a first segment within the edge; selecting a plurality of other segments within the edge relative to the first segment; assigning the first segment and the plurality of other segments to the edge; labeling the edge based on the first segment and the plurality of other segments; and determining whether the physical record in the image is bent or damaged based on the label. [Brief explanation of the drawings]

[0007] [Figure 1A] 1 is a representation illustrating an example of a user using an image capture device on a mobile device to take an image of a workspace where a record is located. [Figure 1B] FIG. 1 is a block diagram illustrating an example of a mobile device. [Figure 1C] FIG. 1 is a block diagram illustrating an example of a records management application running on a mobile device. [Figure 1D]1 illustrates another embodiment of a recording recognition system. [Figure 1E] 1 illustrates another embodiment of a records management system. [Figure 2] 1 is a flow chart of a method of using optical character recognition to convert a record into a corresponding digital record. [Figure 3A] 1 is a flowchart of the digital recording crumple function. [Figure 3B] 10 is an image showing the wireframe of a 3D model in three successive stages of crumpling. [Figure 3C] 10 is an image showing the wireframe of a 3D model in three successive stages of crumpling. [Figure 3D] 10 is an image showing the wireframe of a 3D model in three successive stages of crumpling. [Figure 3E] 3D-3E are images showing rendered views of a crumpled recording at each of a series of stages corresponding to the 3D model shown in FIGS. 3B-3D. [Figure 3F] 3D-3E are images showing rendered views of a crumpled recording at each of a series of stages corresponding to the 3D model shown in FIGS. 3B-3D. [Figure 3G] 3D-3E are images showing rendered views of a crumpled recording at each of a series of stages corresponding to the 3D model shown in FIGS. 3B-3D. [Figure 4A] 1 is a flowchart of a method for converting non-square and damaged or bent recordings into corresponding digital records. [Figure 4B] FIG. 10 shows the output of the resulting projection search using the scoring algorithm. [Figure 4C] It is a captured image of a recording taken at a certain angle. [Figure 4D] This is the captured image of Figure 4C, which has been projectively transformed so that the image appears to have been taken at no angle. [Figure 5]10 is an image showing the conversion of a curved record into a corresponding digital record. [Figure 6] 10 is an image showing the conversion of a curved record into a corresponding digital record. [Figure 7] 10 is an image showing the conversion of a curved record into a corresponding digital record. [Figure 8] 10 is an image showing the conversion of a curved record into a corresponding digital record. [Figure 9] 10 is an image showing the conversion of a curved record into a corresponding digital record. [Figure 10] 10 is an image showing the conversion of a curved record into a corresponding digital record. DETAILED DESCRIPTION OF THE INVENTION

[0008] Overview This disclosure describes techniques for creating and manipulating software records that represent physical records. For example, techniques are described that recognize physical records present in a physical environment, capture information from them, and create digital representations that correspond to the physical records, which representations are referred to herein as digital or software-based records. Additionally, at least some aspects of this disclosure are directed to techniques for managing multiple records.

[0009] In general, records may include physical records and digital records. A physical record generally refers to an object with an overall boundary and recognizable content. A physical record may include an object that results from a person writing, drawing, or other type of input on an object, such as a piece of paper, a whiteboard, or other object that accepts input. For example, a physical record may include handwritten, repositionable sticky notes, paper, film, a whiteboard with a drawing, a poster, or a sign. In some cases, a physical record may be generated using digital means, such as printing on printable, repositionable sticky notes or printing a printed document. In some cases, a single object may contain several records. For example, several ideas may be written on a piece of poster paper or a whiteboard. A physical record may be two-dimensional or three-dimensional. A physical record may have various shapes and sizes. For example, a physical record may be a 3-inch by 3-inch record, a 26-inch by 39-inch poster, or a triangular metal sign. In some cases, a physical record has a known shape and / or size. A digital record generally refers to a digital object that contains information and / or ideas. A digital record can be created using digital input. Digital input can include, for example, a keyboard, a touch screen, a digital camera, a digital recording device, a stylus, a digital pen, etc. In some cases, a digital record can be a representation of a physical record.

[0010] Records Management System FIG. 1A illustrates an example of a record recognition environment 10. In the example of FIG. 1A, the environment 10 includes a mobile device 15 that captures and recognizes one or more records 22 from a work location 20. As described herein, the mobile device provides an execution environment for one or more software applications that can efficiently capture and extract record content from a large number of physical records, such as a collection of records 22 from the work location 20, as described. In this example, the records 22 may be the result of a collaborative brainstorming session with multiple participants. As described, the mobile device 15 and software running thereon may perform various record-related operations, including the automated creation of digital records representing the physical records 22 of the work location 20.

[0011] In this example implementation, mobile device 15 includes, among other components, image capture device 18 and presentation device 28. Additionally, although not shown in FIG. 1A , mobile device 15 may include one or more processors, microprocessors, internal memory and / or data storage, and other electronic circuitry that execute software or firmware to provide the functionality described herein.

[0012] Generally, image capture device 18 is a camera or other component configured to capture image data representative of workspace 20 and recordings 22 located therein. In other words, the image data captures a visual representation of an environment having a plurality of visual recordings, such as workspace 20. While image capture device 18 is described as a camera of mobile device 15, it may include other components capable of capturing image data, such as a video recorder, an infrared camera, a CCD (charge-coupled device) array, a laser scanner, etc. Furthermore, the captured image data may include at least one of an image, a video, a sequence of images (i.e., multiple images taken within a period of time and / or in a certain order), a collection of images, etc., and the term input image is used herein to refer to various exemplary types of image data.

[0013] Presentation devices 28 include, but are not limited to, electronically addressable displays, such as liquid crystal displays (LCDs) or other types of display devices, for use with mobile devices 28. In some implementations, mobile devices 15 generate the contents of recordings for display on presentation devices 28 in various formats, such as lists grouped into rows and / or columns, flow diagrams, etc. In some cases, mobile devices 15 may communicate display information for display on other devices, such as tablet computers, projectors, electronic bulletin boards, or other external devices.

[0014] As described herein, mobile device 15 and software executing thereon provide a platform for creating and manipulating digital records representing physical records 22. For example, mobile device 15 is generally configured to process image data generated by image capture device 18 to detect and recognize at least one of physical records 22 located within workplace 20. In some examples, mobile device 15 is configured to recognize the record by determining the overall boundaries of the record. After recognizing the record, mobile device 15 extracts content of at least one of the one or more records, where content is visual information of record 22.

[0015] In some example implementations, mobile device 15 provides functionality that allows user 26 to export digital records to other systems, such as a cloud-based repository (e.g., cloud server 12) or other computing devices (e.g., computer system 14 or mobile device 16).

[0016] 1A, mobile device 15 is shown as a mobile phone. However, in other examples, mobile device 15 may be a tablet computer, a personal digital assistant (PDA), a laptop computer, a media player, an e-book reader, a wearable computing device (e.g., a watch, eyewear, a glove), or any other type of mobile or non-mobile computing device suitable for implementing the techniques described herein.

[0017] 1B is a block diagram illustrating an example of a mobile device that operates in accordance with the techniques described herein. For purposes of example, the mobile device of FIG. 1B will be described with reference to mobile device 15 of FIG. 1A.

[0018] In this example, mobile device 15 includes various hardware components that provide core functionality for device operation. For example, mobile device 15 includes one or more programmable processors 70 configured to operate according to executable instructions (i.e., program code) typically stored in computer-readable medium or data storage 68, such as a static random access memory (SRAM) device or a flash memory device. As depicted in FIG. 1A , I / O 76 may include one or more devices, such as a keyboard, a camera button, a power button, volume buttons, a home button, a back button, a menu button, or a presentation device 28. Transmitter 72 and receiver 74 provide wireless communication, such as, but not limited to, high-frequency radio frequency (RF) signals, with other devices, such as cloud server 12, computer system 14, or other mobile devices 16, as depicted in FIG. 1A , via a wireless communication interface, as depicted in FIG. 1A . Microphone 71 converts audio information into corresponding electrical signals. Speaker 73 converts electrical signals into corresponding audio information. Vibration motor 75 is used to vibrate mobile device 15 or its housing. Mobile device 15 may include additional discrete digital logic or analog circuitry not shown in FIG. 1B.

[0019] Generally, an operating system 64 runs on the processor 70 to provide an operating environment for one or more user applications 77 (commonly referred to as "apps"), including a records management application 78. The user applications 77 may include, for example, executable program code stored on a computer-readable storage device (e.g., data storage 68) for execution by the processor 70. As another example, the user applications 77 may include firmware, or in some examples, may be implemented in separate logic.

[0020] During operation, mobile device 15 receives input image data and processes the input image data according to the techniques described herein. For example, image capture device 18 may capture an input image of an environment having multiple records, such as workplace 20 having records 22 of FIG. 1A. As another example, mobile device 15 may receive image data via receiver 74 from an external source, such as cloud server 15, computer system 14, or mobile device 16. Generally, mobile device 15 stores the image data in data storage 68 for access and processing by records management application 78 and / or other user applications 77.

[0021] 1B , user application 77 may invoke kernel functions of operating system 64 to output a graphical user interface (GUI) 79 for displaying information to a user of the mobile device. As described further below, records management application 78 may construct and control GUI 79 to provide an improved electronic environment for creating and manipulating corresponding digital records that represent physical records 22. For example, records management application 78 may construct GUI 79 to include mechanisms that allow user 26 to easily control events that are automatically triggered in response to capturing a record with particular characteristics. Additionally, records management application 78 may construct GUI 79 to include mechanisms that allow user 26 to manage relationships between groups of digital records.

[0022] 1C is a block diagram illustrating one exemplary implementation of a records management application 78 that operates in accordance with the techniques described herein. Although user application 77 is described as running on mobile device 15, the examples described herein may be implemented on any computing device, such as cloud server 12, computer system 14, or other mobile device.

[0023] In this example, records management application 78 includes an image processing engine 82 that provides image processing and object recognition functionality. Image processing engine 82 may include an image communication module 90, a record identification module 86, and a digital record generation module 88. Additionally, image processing engine 82 includes an image processing application programming interface (API) 95 that provides a library of image manipulation functions, such as image binarization, masking, filtering, edge detection, etc., for use by other components of image processing engine 82.

[0024] Generally, image data may be stored in data storage device 68. In this example, records management application 78 stores images 97 in data storage device 68. Each of images 97 may include pixel data of an environment having multiple physical images, such as workspace 20 of FIG. 1A.

[0025] As described herein, the record identification module 86 processes the image 97 to identify (i.e., recognize) multiple physical records within the image. The digital record creation module 88 creates digital records 99 corresponding to the physical records identified within the image 97. For example, each of the digital records 99 corresponds to one of the physical records identified in the input image 97. During this process, the digital record creation module 88 updates the database 94 to include records of the digital records and may also store in the database information (e.g., content) extracted from the input image within the determined boundaries of the physical records detected by the record identification module 86. Additionally, the digital record creation module 88 may store metadata in the database 94 that associates the digital records with one or more groups of digital records.

[0026] Additionally, the records management application 78 may be configured to specify, via user input 26, rules 101 that trigger actions in response to the detection of physical records having particular characteristics. For example, the user interface 98 may map actions to particular characteristics of records based on user input. The records management application 78 may output a user interface 98 through which the user can specify rules with actions, such as record grouping actions or actions associated with another software application running on the mobile device, such as an action associated with a calendar application. For each rule, the user interface 98 allows the user to define criteria for triggering the action. During this configuration process, the user interface 98 may prompt the user to capture image data representing an example record for triggering an action and processing the image data to extract characteristics such as color or content. The user interface 98 may then present the determined criteria to the user to assist in defining a rule corresponding to the example record.

[0027] Image communication module 90 controls the communication of image data between mobile device 15 and external devices, such as cloud server 12, computer system 14, mobile device 16, or image capture device 18. In some examples, image communication module 90 may enable a user to communicate, for example, processed or unprocessed images 97 of an environment and / or digital recording and associated information extracted therefrom, including metadata from database 68. In some examples, image communication module 90 exports this data to a zip file that can be communicated via FTP, HTTP, email, Bluetooth, or other mechanisms.

[0028] In the example of FIG. 1C, the records management application 78 includes a user interface 98 that constructs and controls the GUI 79 (FIG. 1B). As described below, in some examples, the user interface 98 may output to display an input image 97 overlaid with multiple digital records 99, with each digital record overlaid in place of a corresponding physical record. Additionally, the user interface 98 may display a group of digital records 99 specified by the user. This group of digital records 99 may be, for example, a subset of digital records recognized within a particular input image 97. The user interface 98 may display this specified group (set) of digital records in a second portion of the GUI 79 and allow the user 26 to easily add or remove digital records 99 from the specified group.

[0029] In some example implementations, the user interface 98 provides an image editor 96 that allows the user to edit the overlay image and / or the digital recording. In another example, the digital recording generation module 88 may include a process or processes that enhance the information extracted from the input image.

[0030] 1D shows another exemplary embodiment of a recording recognition system 100A. The system 100A may include a processing unit 110, one or more recordings 120, a sensor 130, and a recording content repository 140. The processing unit 110 may include one or more processors, microprocessors, computers, servers, and other computing devices. The sensor 130, e.g., an image sensor, is configured to capture a visual representation of a scene having one or more recordings 120. The sensor 130 may include at least one of a camera, a video recorder, an infrared camera, a CCD (charge-coupled device) array, a scanner, etc. The visual representation may include at least one of an image, a video, a sequence of images (i.e., multiple images taken within a period of time and / or in a certain order), a collection of images, etc. The processing unit 110 is coupled to the sensor 130 and configured to receive the visual representation. In some cases, the processing unit 110 is electronically coupled to the sensor 130. Processing unit 110 is configured to recognize at least one of one or more recordings 120 from the visual representation. In some embodiments, processing unit 110 is configured to recognize the recording by determining the overall boundaries of the recording. After recognizing the recording, processing unit 110 extracts the content of the recording. In some cases, processing unit 110 is configured to recognize and extract the content of two or more recordings from the visual representation of the scene having those recordings.

[0031] In some cases, processing unit 110 can execute software or firmware stored on a non-transitory computer-readable medium to implement various processes of system 100A (e.g., recognizing recordings, extracting recordings, etc.). Recording content repository 140 may execute on a single computer, server, storage device, cloud server, etc. In some other cases, recording content repository 140 may execute on a series of networked computers, servers, or devices. In some implementations, recording content repository 140 includes a hierarchy of data storage devices, including local, regional, and central. Recordings 120 can include physical recordings regularly or randomly arranged in a collaborative space, and sensors 130 generate visual representations of recordings 120 within the collaborative space.

[0032] In some implementations, recording recognition system 100A may include a presentation device (not shown in FIG. 1D ) that indicates to a user which recordings have been recognized and / or which recording content has been extracted. Additionally, recording recognition system 100A can present the extracted content via the presentation device. In some embodiments, processing unit 110 can authenticate a recording before extracting the recording's content. If the recording is authenticated, the content is extracted and stored in recording content repository 140.

[0033] FIG. 1E illustrates an embodiment of a record management system 100B. In this embodiment, the record management system 100B includes a processing unit 110, one or more records 120, one or more record sources 150, and a record content repository 140. In some cases, the system 100B includes a presentation device 160. The processing unit 110, the records 120, and the record content repository 140 are similar to the components of the record recognition system 100A shown in FIG. 1A. The record source 150 may include a source that provides physical record content, such as a visual representation of a scene having one or more records, and a source that provides digital record content, such as a data stream entered from a keyboard. In some embodiments, the record management system 100B includes a first source and a second source, where the first source is a visual representation of a scene having one or more records 120. The first source and the second source are generated by different devices. The second source includes at least one of a text stream, an image, a video, a file, and a data entry. Processing unit 110 recognizes at least one of the recordings from the first source and extracts the content of the recording, as described for recording recognition system 100A. In some cases, processing unit 110 labels the recordings with a category. Processing unit 110 may label the recordings based on the recording's particular shape, color, content, and / or other information. For example, each group of recordings may have a different color (e.g., red, green, yellow, etc.).

[0034] In some embodiments, records management system 100B may include one or more presentation devices 160 to present the contents of records 120 to a user. Presentation devices 160 may include, but are not limited to, electronically addressable displays such as liquid crystal displays (LCDs), tablet computers, projectors, electronic bulletin boards, mobile phones, laptops, etc. In some implementations, processing unit 110 generates the contents of records for display on presentation device 160 in various formats, such as, for example, lists grouped into rows and / or columns, flow diagrams, etc.

[0035] Various components of the records recognition system and records management system, such as the processing unit, image sensor, and record content repository, can communicate via a communication interface. The communication interface may include, but is not limited to, any short-range or long-range wired or wireless communication interface. A short-range communication interface may be, for example, a local area network (LAN) or an interface conforming to a known communication standard, such as the Bluetooth standard, the IEEE 802 standard (e.g., IEEE 802.11), ZigBee based on the IEEE 802.15.4 standard, or similar specifications, or other public or proprietary wireless protocols. A long-range communication interface may be, for example, a wide area network (WAN), a cellular network interface, a satellite communication interface, or the like. The communication interface may be either within a private computer network, such as an intranet, or over a public computer network, such as the Internet.

[0036] Optical Character Recognition (OCR) for Records 2 is a flowchart of a method for using optical character recognition to convert a record into a corresponding digital record. The method may be implemented in software or firmware for execution by a processor 70 in a mobile device 15. The method includes receiving (step 200) digital records having handwritten content, such as digital record 210 having content 1, digital record 212 having content 2, and digital record 214 having content 3. The handwritten content may include, for example, characters, text, symbols, icons, or emojis. The handwritten content may include, for example, content handwritten directly on a paper record or electronically written into a digital record, or content that appears to be handwritten. For illustrative purposes, only three digital records are shown, but the method can accommodate a larger number of digital records.

[0037] The digital records are assembled into a grid or other format in a single electronic document or image (step 202). For example, grid 216 includes content from digital records 210, 212, and 214. The grid of digital records is sent to an OCR service or application (step 204), which may include electronically transmitting the grid of digital records to the OCR service or application over a network. OCR results are received (step 206). The results may include, for example, electronic document grid 218, including result 1 from digital record 210, result 2 from digital record 212, and result 3 from digital record 214. The OCR results include a conversion of the content in the received digital records into corresponding characters, such as text, symbols, icons, or emojis.

[0038] The method assigns the OCR results to the original digital record (step 208). For example, result 1 is assigned to digital record 210, result 2 is assigned to digital record 212, and result 3 is assigned to digital record 214. Assigning the results may include, for example, displaying the digital record with the handwritten content replaced with the results, or displaying both the results and the original handwritten content.

[0039] OCR services or applications typically charge a fee for each OCR request to convert content into corresponding characters. By assembling multiple digital records into a single electronic document or image, the number of OCR requests for the digital records can be reduced, resulting in cost savings and more efficient use of the OCR service or application.

[0040] Recording Scraping Function 3A is a flow chart of the digital recording crumpling function. The method may be implemented in software or firmware for execution by processor 70 within mobile device 15.

[0041] The method receives a user command or other instruction to delete or erase a digital record displayed on a display screen, such as the GUI 79 of the mobile device 15 (step 302). The method displays a digital representation and animation of the digital record being crumpled, showing the digital record being crumpled or folded to simulate crumpling a physical record (step 304). The digital representation and animation may include, for example, one or more of visual effects (step 306), sound effects (step 308), and haptic effects (step 310) during at least a portion of the animation. After the animation, the crumpled digital record can optionally be erased from the GUI 79 or other display screen.

[0042] The visual effects of step 306 may include, for example, lighting, texture, shading, and surface effects. The sound effects of step 308 may include, for example, the sound of a physical record being crumpled and played over speaker 73, or another type of sound played over speaker 73 during at least a portion of the animation. The haptic effects may include, for example, vibrating mobile device 15 using vibration motor 75 to create a haptic experience that evokes the sensation of crumpling a physical record in the user's hand. The vibration motor 75 can be triggered in specific patterns to create the haptic experience. One exemplary pattern is a combination of continuous and episodic haptic events configured to synchronize with the sound and animation: one short (100 millisecond) continuous event and seven episodic events with finely tuned intensity levels, closely spaced over 300 milliseconds.

[0043] Table 1 provides the algorithmic steps for implementing the method of FIG. 3A. [Table 1]

[0044] As shown in the algorithm steps of Table 1, Figures 3B-3D are images showing wireframes of a 3D model at three sequential stages of crumpling an exemplary digital recording, and Figures 3E-3G are images showing rendered views of the exemplary digital recording being crumpled at each successive stage corresponding to the 3D model shown in Figures 3B-3D.

[0045] Non-square records 4A is a flowchart of a method for converting non-square and damaged or bent recordings into corresponding digital recordings. The method can be implemented in software or firmware for execution by processor 70 in mobile device 15.

[0046] The method includes receiving an image of a non-square or damaged or bent recording (step 402), such as a non-square recording 410, a damaged recording 412 (with a torn, missing, or bent corner), or a bent recording 414. The method applies a scoring algorithm to the received image of the recording (step 404) and a detection algorithm to the received image of the recording (step 406). The method uses the scoring algorithm and the detection algorithm to convert the received recording into a corresponding digital recording (step 408), such as a non-square digital recording 416, a digital recording without missing corners 418, and a non-bent digital recording 420.

[0047] The method of Figure 4A can be used with paper products of various shapes and sizes, including repositionable sticky notes. The method can detect rectangular (non-square) and non-rectangular products such as recording paper, as well as non-paper products such as whiteboards. Some examples of these products are 5x8 inch, 2x2 inch, and 8x6 inch repositionable sticky notes.

[0048] An algorithm can be used to implement the method of Figure 4A so that the size of the physical records in the captured image can be determined. After running the algorithm, the method will have a collection of "records" that are roughly rectangular in shape, defined by four corners. Even if all of the physical records are the same size, because the shapes of the records are distorted by projection in the captured image, these rectangles will vary in shape and size, and the corner locations reported by the algorithm may not be 100% accurate.

[0049] To be able to correctly determine what physical size the digital recording corresponds to, the digital recording is perspectively rectified, i.e., the corner positions are recalculated so that the recording appears to have been taken head-on (rather than at a non-zero angle). This is done using the following search:

[0050] 1. For every camera angle from the minimum to the maximum along the X and Y axes, the corners of the recording are recalculated according to the projection that matches the camera angle, and a score is calculated. (Example: starting from -55 degrees and ending at 55 degrees, recalculating the corners of the recording in 2-degree steps, i.e., starting at -55 degrees, then at -53 degrees, then at -51 degrees). See Figure 4B.

[0051] 2. The score is designed to maximize the "squareness" of the digital record, i.e., corner angles as close to 90 degrees as possible, vertical edge lengths as similar as possible, and horizontal edges as similar as possible.

[0052] Figure 4B shows the output from the projection search. Each point is the result of scoring a combination of X and Y values. Darker colors mean better scores. The lighter colored points in the search space are those with the best scores.

[0053] Figure 4C is a captured image of a recording taken at an angle (non-zero angle from the front). Figure 4D is the captured image of Figure 4C, but with a projective transformation so that the image appears as if viewed from the front at no angle.

[0054] The projection angle with the best score is selected for the next step, which involves finding the combination of recording sizes that best corresponds to the aspect ratios of the individual recordings and their relative sizes as follows:

[0055] 1. For each recording, calculate the closest recording size(s) based on the aspect ratio. The closest size class is one that has an aspect ratio similar to the digital recording. For example, a recording with an aspect ratio of 0.98 would be very close to all square recording sizes, which have an aspect ratio of 1. A recording with an aspect ratio of 1.28 would be close enough to both 8x6 (which has an aspect ratio of 1.25) and 6x4 (which has an aspect ratio of 1.33). For each size class, calculate the pixels per inch, i.e., how much physical area each pixel represents if the recording size is correct.

[0056] 2. For each calculated pixel-per-inch value, calculate the best matching size class for each recording and calculate the total error based on the difference in area and aspect ratio between the calculated values ​​of the physical and digital recordings.

[0057] 3. Select the segment with the smallest total error. The selected segment can be assigned to the record to generate a corresponding digital record having a size based on the selected segment.

[0058] Broken and bent records Damaged records (i.e., records with bent or torn corners) and bent records (i.e., records that are not flush with the surface to which they are attached) can cause suboptimal detection results, either missing records entirely or records with incorrect shapes. To properly correct for these problems, the method of FIG. 4A using the detection algorithm of step 406 can also be used to process damaged or bent records, allowing the software algorithm to find the exact shape of the record.

[0059] The detection algorithm finds out if a record is bent by examining additional metadata generated about the corners of the digital record. Since the supported record size is rectangular, the final result contains the locations of the four corners of the record. However, the detection algorithm itself uses more detailed contours, i.e., contours with five or more edges (see Figure 5).

[0060] In the detailed outline, the longest segment is first selected as the starting point. Given this segment, three more segments are determined to be "major edges" based on their length (e.g., at least 25% of the longest segment) and their relative position to the longest segment (see Figure 6). The remaining segments in the detailed outline are assigned to four edges based on the similarity of their angles to the major edges (see Figure 7).

[0061] The algorithm thus detects four edges, each consisting of one or more segments.

[0062] 1. If an edge consists of only one segment, it is labeled as a straight edge.

[0063] 2. If an edge has two or more segments, the segments are labeled as either straight (if the segments form a more or less straight line) or inwardly or outwardly curved, with a curvature index, which is a value that indicates how curved the edge is. The sum of the angles between the segments that make up the edge determines whether the edge is curved or straight. Curvature is recorded simply as a binary value, i.e., the record is either curved or not. The fact that the record is curved is then used as a hint that the size and shape may be less accurate than a non-curved record. The curvature index can, for example, be based on the sum of the angles between the segments that make up the edge.

[0064] If the two edges at either end of the record are bent inward and outward, respectively, the record is a curved record, and the record candidate is tagged with this information (see Figure 9).

[0065] The algorithm can also handle corrupted records, e.g., records with portions missing, by examining the edges of the record in the same or similar manner to find a digital record of corresponding size without the missing portion.

[0066] Figures 5-10 are images showing the conversion of a bend record into a corresponding digital record: Figure 5 is the record with the input contour, Figure 6 is the main edge, Figure 7 is the segment assignment, Figure 8 is the edge termination, Figure 9 is the edge division, and Figure 10 is the final corner position of the record. In addition to the embodiments, the following aspects will be noted. (Appendix 1) 1. A method for managing records, the method being performed by a processor, comprising: receiving a plurality of digital records, each of the digital records including an image of handwritten content; assembling the plurality of digital records into a single electronic document; sending the electronic document to an optical character recognition (OCR) service or an application for using the OCR to convert the handwritten content into text; receiving the electronic document from the OCR service or application, the electronic document having the results of converting the handwritten content into text using the OCR; electronically separating the plurality of digital records from the electronic document and assigning the results of the OCR to the digital records having the corresponding handwritten content; A method comprising: (Appendix 2) 2. The method of claim 1, wherein the assembling step includes assembling the plurality of digital records into a grid. (Appendix 3) 10. The method of claim 1, wherein receiving the plurality of digital records comprises receiving the digital records converted from images of corresponding physical records. (Appendix 4) 2. The method of claim 1, wherein the results include text. (Appendix 5) 2. The method of claim 1, wherein the result includes an icon. (Appendix 6) 1. A method for managing records, the method being performed by a processor, comprising: displaying the digital record on an electronic display device; receiving a command to delete or erase said digital recording; displaying a digital representation and animation of the digital recording being crumpled in response to the command; A method comprising: (Appendix 7) 7. The method of claim 6, wherein the step of displaying the digital record comprises receiving the digital record converted from an image of a corresponding physical record. (Appendix 8) 7. The method of claim 6, further comprising providing a visual effect during the animation. (Appendix 9) 9. The method of claim 8, wherein the visual effect comprises a texture on the digital recording. (Appendix 10) 9. The method of claim 8, wherein the visual effect comprises shading on the digital recording. (Appendix 11) 7. The method of claim 6, further comprising providing a crunchy sound effect during the animation. (Appendix 12) 7. The method of claim 6, further comprising providing a haptic effect via vibration during the animation. (Appendix 13) said displaying said digital representation further comprising: placing said digital record within a spherical object; shrinking the spherical object during the animation; 7. The method of claim 6, comprising: (Appendix 14) 1. A method for managing records, the method being performed by a processor, comprising: displaying the digital record on an electronic display device; receiving a command to delete or erase said digital recording; displaying a digital representation and animation of the digital recording being crumpled in response to the command; providing both a sound effect of a particular sound and a haptic effect via vibration during at least a portion of said animation; A method comprising: (Appendix 15) 1. A method for managing records, the method being performed by a processor, comprising: receiving an image of the physical record taken from a non-zero angle; detecting corners of the physical record in the image; recalculating the corners according to projections consistent with multiple camera angles; calculating a score for the corner based on the recalculating step at each of the camera angles; selecting a projection angle for the corner based on the calculating step, and calculating a size of the digital recording based on an aspect ratio of the digital recording at the selected projection angle; A method comprising: (Appendix 16) 16. The method of claim 15, wherein the recalculating step includes using a maximum camera angle and a minimum camera angle. (Appendix 17) 16. The method of claim 15, wherein the recalculating step includes recalculating the corners at incremental angles among the plurality of camera angles. (Appendix 18) The calculating step calculating pixels per inch for the selected size; determining the difference between the calculated pixels per inch value and the area of ​​a physical record corresponding to the selected size; 16. The method of claim 15, comprising: (Appendix 19) 1. A method for managing records, the method being performed by a processor, comprising: receiving an image of the physical record; detecting an edge of the physical record in the image; selecting a first segment within the edge; selecting a plurality of other segments within the edge relative to the first segment; assigning the first segment and the plurality of other segments to the edge; labeling the edge based on the first segment and the plurality of other segments; determining whether the physical record in the image is bent or damaged based on the label; A method comprising: (Appendix 20) 20. The method of claim 19, wherein the label of the edge is selected from straight, curved inward, and curved outward. (Appendix 21) The determining step includes: determining that the physical record is a curved record if both edges of the physical record have the inward or outward bending labels; determining that the physical record is not a curved record if both edges of the physical record do not have the inward or outward bending labels; 21. The method of claim 20, comprising: (Appendix 22) 21. The method of claim 20, further comprising determining a bending coefficient for the edge labeled as an in-fold or an out-fold, the bending coefficient being a value indicative of an amount of bending of the corresponding edge. (Appendix 23) 23. A system for managing records, comprising a processor configured to perform any of the methods described in appendixes 1 to 22. (Appendix 24) A method for displaying a user interface on an electronic display device, the user interface displaying a record that has been crumpled according to the method of any one of claims 6 to 14.

Claims

1. 1. A method for managing records, the method being performed by a processor, comprising: receiving a plurality of digital records, each of the digital records including an image of handwritten content; assembling the plurality of digital records into a single electronic document; sending the electronic document to an optical character recognition (OCR) service or application for using the OCR to convert the handwritten content into text; receiving the electronic document from the OCR service or application, the electronic document having the results of converting the handwritten content into text using the OCR; electronically separating the plurality of digital records from the electronic document and assigning the results of the OCR to the digital records having the corresponding handwritten content; A method comprising:

2. The method of claim 1 , wherein the assembling step comprises assembling the plurality of digital records into a grid.

3. The method of claim 1 , wherein the step of receiving the plurality of digital records comprises receiving the digital records converted from images of corresponding physical records.

4. The method of claim 1 , wherein the results include text.

5. The method of claim 1 , wherein the result comprises an icon.

Images