Digital Image Transcription and Manipulation

Abstract

A computer server configured to perform digital image processing is disclosed herein. In one embodiment, upon receiving a transcription command, the computer server performs text recognition based on patterns in a digital image to generate digital text data corresponding to the digital image. The computer server can then determine a content format of the digital text data based on the imported digital image, and automatically apply the determined content format to the generated digital text data. The digital data can then be inserted into an electronic message and sent to a destination along with the inserted typeset digital text data.

Description

Background Art

[0001] Writing on a whiteboard is an effective way to communicate ideas between conference participants. For example, conference participants can use a marker to write down action items, draw schematics, or reduce other suitable information in written form. Although physical whiteboards have been used for a long time, interactive whiteboards are also suitable today. For example, an interactive whiteboard can include a computer with a large touch screen. Device drivers are usually installed on the computer so that the interactive whiteboard can act as a human input device (HID). In one application, an interactive whiteboard can receive input via, for example, a touch screen, and display the received input as an image on the touch screen. Summary of the invention

[0002] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0003] While whiteboard discussions can be an effective way to develop strategies, solve problems, explain design details, and the like, summaries of these discussions or meeting notes can be difficult to disseminate electronically as text transcripts. For example, during a whiteboard meeting, a meeting participant may handwrite notes of action items as a list with multiple bullet points. A digital image of the bulleted notes can be captured using a camera or the functionality of an interactive whiteboard. The image of the notes can then be attached to an email and transmitted to the meeting participants. However, in order to convert the notes in the captured image into text bullet points or other suitable forms of digital data, one of the meeting participants (or another person) must manually type the notes and format the typed notes as bullet points based on the captured image. This manual transcription can be cumbersome and inefficient, and therefore negatively impacts the productivity of the meeting participants. In addition, manual transcription is also prone to human error.

[0004] Several embodiments of the disclosed technology can address at least some of the above difficulties by enabling automatic transcription of meeting notes for electronic dissemination based on images of the meeting notes. In one implementation, a digital image of the meeting notes can be captured from a whiteboard via a camera or the functionality of an interactive whiteboard. The user can then begin composing a new email (or other suitable type of electronic message) about the meeting on a client device, such as in an email client. The email client can be a stand-alone application on a client device or a web-based application executed in a web browser. The email can be in the format of Hypertext Markup Language (HTML), Extensible Markup Language (XML), or other suitable types of formats.

[0005] The user can then use functionality in an email client to import the captured digital image of the meeting notes into the email body of the email being composed. In one example, the digital image can be pasted or inserted into the email body as an inline picture. In another example, the digital image can be attached to the email as an attachment. In other examples, the user can import the digital image into the email via other suitable techniques. During email composition, the content in the email client can be synchronized with the content in the corresponding email server via a computer network. Therefore, after the digital image is imported, the email server can also contain a copy of the imported digital image via synchronization with the email client.

[0006] Then, the user can provide a transcription command to convert the imported digital image into a file or data in a text format. The text file can contain a stream of characters, including possible control characters, and can be encoded in one of various character encoding schemes, such as HTML, XML, etc. In one embodiment, the user can activate a user interface element (e.g., a button) on the email client to provide a command. In other embodiments, the user can activate a hot key, a function key, or other suitable input to provide a command. In other embodiments, the email client can be configured to automatically detect the imported digital image and ask the user to confirm the automatic transcription of the digital image. Upon receiving the command, the email client and / or the corresponding email server can transmit the imported digital image together with other suitable information (e.g., a request for transcription, an email identification of the email, the user's authentication credentials, etc.) to a text recognizer to perform text recognition.

[0007] The text recognizer can be configured to perform pattern recognition to recognize text in a digital image. In one embodiment, the text recognizer can be hosted on a remote server in, for example, a data center. In other embodiments, the text recognizer can be hosted in an email server, a client device, or other suitable type of computing environment. In some implementations, the text recognizer can be configured to perform pattern recognition based on a set of pattern models, which are developed by analyzing a set of digital images with identified text using a "neural network" or "artificial neural network" configured to "learn" or gradually improve the performance of a task by learning known examples. In some implementations, a neural network can include a multi-layer object commonly referred to as a "neuron" or "artificial neuron". Each neuron can be configured to perform a function such as a nonlinear activation function based on one or more inputs connected via a corresponding connection. Artificial neurons and connections usually have contribution values ​​that are adjusted as learning proceeds. The contribution value increases or decreases the strength of the input at the connection. Typically, artificial neurons are organized in layers. Different layers can perform different types of transformations on corresponding inputs. Signals usually travel from the input layer to the output layer after traversing one or more intermediate layers. Thus, by using a neural network, the text recognizer can provide a set of pattern models that the text recognizer can use to recognize text in a received digital image. In other implementations, the text recognizer can be configured to perform pattern recognition based on user-provided rules or via other suitable techniques.

[0008] When completing the recognition of the text in the received digital image, the text recognizer can also be configured to recognize the content structure or format contained in the digital image. Example content structures can include lists, tables, paragraphs, etc. with bullet points. In some embodiments, the recognition of such content structures can be based on trigger words or keywords (e.g., "1", "2", "3", etc.), punctuation marks (e.g., bullet points, spaces, etc.), content appearance (e.g., bold, italics, underlines, etc.), or other characteristics of the recognized text specified in a structural model developed using, for example, a neural network. In other embodiments, the recognition of the content structure can be based on the relative position, relative size, or other profiles of the recognized text. In other embodiments, the recognition of the content structure can be based on any one of the combinations of the above-mentioned technologies.

[0009] When identifying the content structure, the text recognizer can be configured to automatically apply the content structure to the recognized text to derive the text corresponding to the layout of the digital image. For example, the text recognizer can be configured to format the recognized text into columns and insert bullets or numbers in front of each entry in the column. In another example, the text recognizer can be configured to format the recognized text into a table, column, paragraph, or other suitable type of format.

[0010] The text recognizer can then be configured to provide data representing the typeset text to the email server. The email server and / or email client on the client device can then automatically display the received data in the body of the email being composed, in addition to or in lieu of the imported digital image of the meeting notes. The user can then selectively modify the displayed text, reformat the displayed text, or perform other suitable operations. Upon receiving a command from the user to send an email, the email client and email server can cooperate to transmit a copy of the email to the target destination.

[0011] Several embodiments of the disclosed technology can therefore efficiently transcribe handwritten notes into digital text data with little or no user intervention. Rather than a user manually typing meeting notes from a digital image, a single click by the user can cause the digital image to be automatically transcribed into digital text data during composition and inserted into the body of an email. By eliminating human error during transcription, several embodiments of the disclosed technology can also be more accurate than manually typing meeting notes. Thus, productivity of meeting participants can be increased compared to manual transcription of meeting notes. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figures 1A-1D is a schematic diagram illustrating a computing system that implements digital image transcription and manipulation during certain stages of operation in accordance with an embodiment of the disclosed technology.

[0013] Figure 2 is a schematic diagram illustrating certain hardware / software components of the computing system of FIG. 1 , in accordance with an embodiment of the disclosed technology.

[0014] Figures 3A-3C is a flow chart illustrating an example process of digital image transcription and manipulation according to an embodiment of the disclosed technology.

[0015] Figure 4 A computing device that is suitable for certain components of the computing system in FIG. 1 . DETAILED DESCRIPTION

[0016] Certain embodiments of systems, devices, components, modules, routines, data structures, and processes for digital image transcription and manipulation in a computing system are described below. In the following description, specific details of the components are included to provide a thorough understanding of certain embodiments of the disclosed technology. A person skilled in the relevant art will also understand that the technology may have additional embodiments. The technology may also be used without the following references. Figure 1A-4 The embodiments described are practiced without the need for several details.

[0017] As used herein, the term "email server" generally refers to a computer dedicated to running applications that are configured to receive incoming email from senders and forward outgoing email to recipients via a computer network (e.g., the Internet). Examples of such applications include Microsoft An email server may maintain and / or access one or more inboxes for a corresponding user. As used herein, an "inbox" is a folder configured to contain data representing a user's incoming emails. An email server may also maintain and / or access one or more draft folders and / or outboxes and / or other suitable mailboxes configured to store outgoing emails.

[0018] As also used herein, "digital image" generally refers to digital data representing an image, a picture, or other suitable type of graphic entity. Example digital images can be bitmaps, JPEGs, or other suitable image formats. The term "digital text data" generally refers to digital data representing characters of readable material rather than corresponding graphical representations. Digital text data may include plain text, formatted text, structured text, or other suitable types of text data. Digital text data may be encoded according to various encoding rules, and may also include human-readable metadata (or tag data), such as HTML, XML, etc.

[0019] Additionally, the term "text recognition" generally refers to the technology of electronically converting images of typed, handwritten, or printed text into machine-encoded text. For example, a photograph of a document may be converted into machine-encoded text via text recognition to allow the characters in the document to be copied, edited, or otherwise digitally manipulated. The term "content format" or "content structure" generally refers to an encoding format according to which data may be manipulated as displayable information. Content formats may be used to record, transmit, and / or prepare digital data for viewing or interpretation.

[0020] Although whiteboard discussions can be an effective way to develop strategies, solve problems, explain design details, etc., summaries of these discussions or meeting notes may be difficult to disseminate electronically as text transcripts. For example, during a whiteboard meeting, meeting participants may handwrite notes of action items as a list with multiple bullet points. A digital image of the bulleted notes can be captured using a camera or the functionality of an interactive whiteboard. The image of the note can then be attached to an email and transmitted to the meeting participants. However, in order to convert the notes in the captured image into text bullet points or other suitable forms of digital data, one of the meeting participants (or another person) must manually type the notes and format the typed notes as bullet points based on the captured image. This manual transcription can be cumbersome and inefficient, and therefore has a negative impact on the productivity of the meeting participants. In addition, manual transcription is also prone to human error.

[0021] Several embodiments of the disclosed technology can address at least some of the above-mentioned difficulties by enabling automatic transcription of meeting notes based on images of the meeting notes for electronic dissemination. For example, upon receiving a user command, pattern recognition can be performed on a digital image imported into an email to convert the digital image into digital text data. In addition to or in lieu of the digital image, the converted digital text data can then be automatically inserted into the email for transmission to a suitable destination. In this way, summaries or meeting notes of whiteboard or other types of discussions can be easily disseminated electronically as text transcripts with little or no involvement of human error, as described below with reference to Figure 1A-4 Described in more detail.

[0022] Figures 1A-1D is a schematic diagram illustrating a computing system 100 that implements digital image transcription and manipulation during certain stages of operation according to an embodiment of the disclosed technology. Figure 1A As shown, computing system 100 may include a computer network 104 interconnected with a client device 102 of a user 101, one or more email servers 106 (referred to herein for simplicity as "email servers 106") interconnected with a network storage device 112 including one or more inboxes 114, and a text recognizer 107 interconnected to a data repository 108 including a schema model 110 and a structural model 115. Computer network 104 may include an intranet, a wide area network, the Internet, or other suitable type of network.

[0023] Although Figure 1ASpecific components of computing system 100 are shown in FIG. 1 , but in other embodiments, computing system 100 may include additional and / or different components or arrangements. For example, computing system 100 may also include additional network storage devices, additional hosts, and / or other suitable components (not shown). In other embodiments, network storage device 112 and / or data repository 108 may be integrated into email server 106.

[0024] As described in more detail below, various components of computing system 100 may be configured to collaborate and perform automatic transcription and manipulation of a digital image, such as digital image 109 captured by user 101 of whiteboard 102 holding visible content. In the illustrated example, the content on whiteboard 102 includes a handwritten numbered list of action items as follows:

[0025] 1. Clear the field

[0026] 2. Install fence

[0027] 3. Erecting buildings

[0028] 4. Install the robot

[0029] 5. Car manufacturing

[0030] In other examples, the content on whiteboard 102 may also include lists with bullet points, paragraphs, sentences, graphic sketches, or other suitable types of content.

[0031] In some embodiments, the whiteboard 102 may be a physical whiteboard 102 on which content may be handwritten. The user 101 may then use, for example, a camera 103, a smartphone, or other suitable device to capture a digital image 109 of the content on the whiteboard 102. The digital image 109 may then be uploaded or otherwise transmitted from the camera 103 to the client device 102 via a wireless or wired connection for further processing. In other embodiments, the whiteboard 102 may be an interactive whiteboard that is configured to directly generate the digital image 109, for example, by capturing a photo of the content on the whiteboard 102 and sending the captured photo as the digital image 109 to the client device 102. Although in Figure 1A Whiteboard 102 is used as an example of a physical medium for storing handwritten content (e.g., the list shown above), but in other embodiments, computer system 100 can also perform automatic transcription and manipulation of display content captured from a blackboard, monitor screen, or other suitable type of output device.

[0032] Client devices 102 may each comprise a computing device that facilitates user 101 to access computing services provided by email server 106 via computer network 104. For example, in the illustrated embodiment, client device 102 comprises a desktop computer. In other embodiments, client device 102 may also comprise a smartphone, laptop computer, tablet computer, game console, or other suitable computing device. Although for illustration purposes only, the present invention is not limited to the embodiment of FIG. Figure 1A One user 101 is shown in FIG. 1 , but in other embodiments, computing system 100 may facilitate any suitable number of users 101 accessing suitable types of computing services provided by email server 106 .

[0033] like Figure 1A As shown, the client device 102 can be configured to execute appropriate instructions to provide the user 101 with an email client 120. In some embodiments, the email client 120 can be a stand-alone application, such as Microsoft In other embodiments, the email client 120 may be a web-based application such as Google In the example shown, email client 120 may include a user interface 121 having a plurality of control elements, such as a “Compose” button 122 , an “Import” button 124 , a “Transcribe” button 126 , and a “Send” button 127 .

[0034] The user interface 121 may also include a work area 123 for facilitating composition, editing, viewing, or other suitable operations associated with the email service. For example, the work area 123 may be configured to facilitate composition of a draft email 116 from the user 101 regarding the content on the whiteboard 102 when the user 101 activates the "compose" button 122. The email 116 may include a header 117 having, for example, a "To:" field, a "Reply:" field, a "From:" field, a "Cc:" field, and / or other suitable data fields. For simplicity, Figure 1A Only the To: field and the Reply: field are shown. In the example shown, the To: field could contain a distribution list of "Project Team" and the Reply: field could contain the subject line, "Project Summary."

[0035] In some implementations, the user 101 may activate the “Import” button 124, as represented by the cursor 128, to insert the digital image 109 into the body portion 118 of the draft email 116. In one example, the camera 103 may be connected to the client device 102 via a wired or wireless connection and viewed as an external drive by the client device 102. Thus, upon detecting activation of the “Import” button 124, the email client 120 may be configured to display a pop-up window (not shown) that allows the user 102 to browse files on the camera 103 and select the digital image 109 for import. In other examples, the user 101 may copy and paste the digital image 109 from the camera 103 into the body portion 118 of the draft email 116. In other examples, the email client 120 may be configured to insert the digital image 109 into the body portion 118 of the draft email 116 in other suitable ways.

[0036] Email server 106 may be configured to facilitate email reception, storage, forwarding, and other related functions. Figure 1A As shown, user 101 may have a corresponding email folder 114 in network storage 112. Email folder 114 may be configured to contain copies of emails 116 that are received, sent, deleted, etc. Email folder 114 may also be configured to contain copies of draft emails 116 composed on client device 102 via, for example, periodic synchronization with email client 120 on client device 102. For example, user 101's email folder 114 on network storage 112 may have a server copy of draft email 116 and a copy of inserted digital image 109 in email client 120.

[0037] When inserting the digital image 109 into the body portion 118 of the draft email 116, in some embodiments, the user 101 can activate a "transcribe" button 126 to automatically transcribe the content in the digital image 109 (i.e., the list of action items) into digital text data 114 ( Figure 1C ), without requiring the user 101 to manually enter the list. Figure 1BAs shown, upon receiving activation of the “transcribe” button 126 represented by the cursor 128′, the email client 120 and / or other suitable components of the client device 102 may be configured to send a transcription command 112 to the email server 106 via the computer network 104. In some embodiments, the transcription command 112 may include a request for transcription of the digital image 109, an image identification or copy of the digital image 109, an email identification of the draft email 116, authentication credentials of the user 101, and / or other suitable information. In other embodiments, the email client 120 may be configured to automatically detect the imported digital image 109 and query the user 101 to confirm the automatic transcription of the digital image 109.

[0038] Upon receiving the transcription command 112, the email server 106 may provide a copy of the digital image 109' to the text recognizer 107 along with a request to transcribe the content in the digital image 109'. Figure 1B In the illustrated example, text recognizer 107 is shown as a server separate from email server 106. In another example, text recognizer 107 may be part of email server 106. In other examples, text recognizer 107 may include one or more computing services hosted on other suitable servers (not shown), such as in a data center.

[0039] The text recognizer 107 can be configured to perform pattern and structure recognition based on the pattern model 110 and the structure model 115 in the data repository 108 to recognize text in the digital image 109. In some embodiments, the pattern model 110 and the structure model 115 can be developed via machine learning, as described below with reference to Figure 2 More detailed description. In other embodiments, the mode model 110 and the structural model 115 may also include rules configured by the user 101 or other suitable entities. In some implementations, the mode model 110 may include parameters of shape, position, relative size and / or other suitable values ​​corresponding to specific characters. For example, a shape with a dot on top of a vertical line may correspond to the letter "i". The structural model 115 may include trigger words or keywords (e.g., "1", "2", "3", etc.), punctuation marks (e.g., bullets, spaces, etc.), content appearance (e.g., bold, italics, underline, etc.) or other characteristics of the recognized text that correspond individually to the text structure or format. In other implementations, the mode model 110 and / or the structural model 115 may include other suitable rules, parameters, features or attributes for identifying characters and text structures.

[0040] like Figure 1CAs shown, the text recognizer 107 can be configured to utilize the pattern model 110 to transcribe the content in the digital image 109' into digital text data 114 and recognize the content structure or format included in the digital image 109'. The text recognizer 107 can then apply the recognized content format to the digital text data 114 to generate typeset digital text data 114. The text recognizer 107 can then send the typeset digital text data 114 to the email server 106 for insertion into the body portion 118 of the draft email 116. Example components and operations of the text recognizer 107 are described below with reference to Figure 2 Describe in more detail.

[0041] Upon receiving the typeset digital text data 114, the email server 106 may insert the digital text data 114 into the body portion 118 of the server copy of the draft email 116 in addition to or in place of the digital image 109. Through periodic synchronization or other suitable techniques, in addition to or in place of the digital image 109, the digital text data 114 may also be transmitted to the client device 102 and displayed to the user 101 in the body portion 118 of the email client 120. The user 101 may then optionally modify the displayed digital text, reformat the displayed digital text, or perform other suitable operations. For example, the user 101 may now copy and paste some or all of the digital text data 114 into another portion of the body portion 118. Upon receiving a command from user 101 to send an email, for example, by activating a “Send” button represented by cursor 128 ”, the email client 120 and email server 106 may cooperate to transmit a copy of the email 116 to a target destination, such as another client device 102 of another user 101 b having digital text data 114 instead of a digital image 109 .

[0042] Several embodiments of the disclosed technology can therefore efficiently transcribe handwritten notes contained in digital images 109 into digital text data 114 with little or no user intervention. Unlike user 101 manually typing meeting notes from a digital image, a single click by the user can cause digital image 109 to be automatically transcribed into digital text data 114 during composition and inserted into the body 118 of email 116. By eliminating human errors during transcription, several embodiments of the disclosed technology can also be more accurate than manually typing meeting notes. Therefore, when compared to manual transcription of meeting notes, the productivity of meeting participants (e.g., users 101a and 101b) can be improved.

[0043] Figure 2 is a schematic diagram illustrating certain hardware / software components of a computing system 100 according to an embodiment of the disclosed technology. Figure 2For the sake of clarity, only Figures 1A-1D Certain components of the computing system 100. Figure 2 In other figures of this document, individual software components, objects, classes, modules, and routines may be computer programs, procedures, or processes written as source code in C, C++, C#, Java, and / or other suitable programming languages. A component may include, but is not limited to, one or more modules, objects, classes, routines, properties, processes, threads, executable files, libraries, or other components. A component may be in source form or in binary form. A component may include aspects of source code (e.g., classes, properties, procedures, routines) prior to compilation, compiled binary units (e.g., libraries, executable files), or artifacts (e.g., objects, processes, threads) that are instantiated and used at runtime.

[0044] Components within the system may take different forms within the system. As an example, a system including a first component, a second component, and a third component may, but is not limited to, encompass a system in which the first component is a property in source code, the second component is a binary compiled library, and the third component is a thread created at runtime. A computer program, procedure, or process may be compiled into object code, intermediate code, or machine code and presented for execution by one or more processors of a personal computer, network server, laptop, smart phone, and / or other suitable computing device.

[0045] Likewise, a component may include a hardware circuit. One of ordinary skill in the art will recognize that hardware may be considered to be petrochemical software, while software may be considered to be liquefied hardware. To cite just one example, the software instructions in a component may be burned into a programmable logic array circuit, or may be designed as a hardware circuit with an appropriate integrated circuit. Likewise, hardware may be simulated by software. Various implementations of source code, intermediate code, and / or object code and associated data may be stored in a computer memory, which may include a read-only memory, a random access memory, a disk storage medium, an optical storage medium, a flash memory device, and / or other suitable computer-readable storage media in addition to a propagating signal.

[0046] like Figure 2 As shown, the computing system 100 may include a model developer 130 and a text recognizer 107 operatively coupled to each other. Figure 2 Although shown as a separate component from the text recognizer 107, in other embodiments, the model developer 130 may be integrated with the text recognizer 107. In other embodiments, the model developer 130 may be provided by one or more other online or offline servers (not shown) separate from the text recognizer 107.

[0047] The model developer 130 may be configured to develop and generate pattern models 110 and structure models 115 via various machine learning techniques based on a training data set 111 comprising a digital image 109″ and associated digital text 114″ and structure 117. The associated digital text 114″ and structure 117 may be generated manually, automatically via unstructured learning, or via other suitable techniques. In one implementation, the model developer 130 may be configured to use a neural network, which includes multiple layers of objects, generally referred to as “neurons” or “artificial neurons,” to perform machine learning based on the training data set 111. By using a neural network, the model developer 130 may provide a collection of pattern models 110 and structure models 115 that may be used by the text recognizer 107 for automatic transcription and formatting operations. In one example, the pattern model 110 may include various variable values ​​associated with the digital image 109″. Example variables may include geometry, size, spatial arrangement, and / or other suitable parameters. Thus, one example pattern model 110 may include an indication that a portion of the digital image 109" represents the letter "o" when the portion of the digital image has a generally circular shape. In other examples, the pattern model 110 may have other suitable features and attributes. In the illustrated embodiment, the model developer 130 provides the pattern model 110 and the structural model 115 for storage at the data repository 108. In other embodiments, the model developer 130 may provide the pattern model 110 and / or the structural model 115 directly to the text recognizer 107, or store the pattern model 110 in other suitable locations.

[0048] like Figure 2 As shown, the text recognizer 107 may include a character identifier 132, a structure identifier 134, and a text formatter 136 that are operatively coupled to each other. Figure 2 Specific components or modules of the text recognizer 107 are shown in FIG. 1 , but in other embodiments, the text recognizer 107 may also include an interface, a network, or other suitable types of components and / or modules. In other embodiments, at least one of the above components may be provided by an external application / server separate from the text recognizer 107.

[0049] In some embodiments, in the first stage of the transcription process, the character identifier 132 can identify individual characters in the received digital image 109 based on the pattern model 110 in the data repository 108. In one embodiment, the character identifier 132 can scan individual shapes or shape combinations in the received digital image 109 and convert them into one or more ASCII or other suitable types of characters, punctuation marks, special characters, etc. based on the pattern model 110. In other embodiments, the character identifier 132 can also be configured to identify invisible characters such as lines, paragraph breaks, spaces, or other suitable types. After completing the identification of the characters in the digital image 109, the character identifier 132 forwards the digital text data 114 to the structure identifier 134 for further processing.

[0050] After receiving the digital text data 114, the structure identifier 134 can be configured to identify one or more content structures or formats in the digital text data 114 based on the structural model 115. Example content structures may include lists, tables, paragraphs, etc. with bullet points. In some embodiments, the identification of such content structures can be based on trigger words or keywords (e.g., "1", "2", "3", etc.), punctuation (e.g., bullet points, spaces, etc.), content appearance (e.g., bold, italics, underline, etc.), or other characteristics of the recognized text specified in a structural model developed using, for example, a neural network. In other embodiments, the identification of the content structure can be based on the relative position, relative size, or other profiles of the recognized text. In other embodiments, the identification of the content structure can be based on any one of the combinations of the above-mentioned technologies.

[0051] After identifying the content structure, the structure identifier 134 can forward the digital text data 114 and the identified content structure 119 to the text formatter 136 for further processing. The text formatter 136 can be configured to automatically apply the content structure 119 to the identified text data 114 to derive the typeset text data 114' corresponding to the digital image 109. For example, the text formatter 136 can be configured to format the identified text data 114 into a column and insert bullets or numbers in front of each entry in the column. In another example, the text formatter 136 can be configured to format the identified text into a table, column, paragraph, or other suitable type of format.

[0052] The text recognizer 107 may then be configured to provide the typeset text data 114' to the email server 106. Figures 1A-1DAs described in more detail, in addition to or in lieu of the imported digital image 109, the email server 106 and / or the email client 120 on the client device 102 can then automatically insert and render the received typeset digital text data in the body portion 118 of the email 116 being composed. The user 101 can then optionally modify the rendered text, reformat the rendered text, or perform other suitable operations. Upon receiving a command from the user 101 to send the email 116, the email client 120 and the email server 106 can cooperate to send a copy of the email 116 to the target destination.

[0053] Figures 3A-3C is a flow chart illustrating an example process for automatic summarization of content in an electronic message according to an embodiment of the disclosed technology. Figures 1A-1D The processes are described with reference to computing system 100 of FIG. 1 , but in other embodiments, the processes may also be implemented in computing systems having additional and / or different components.

[0054] like Figure 3A As shown, process 200 may include importing a digital image into the body of a draft email at stage 202. Process 200 may then include a decision stage 204 to determine whether a transcription command has been received to convert the digital image into text. In response to determining that a transcription command has been received, process 200 continues to automatically transcribe the digital image into text in the body of the draft email at stage 206. Figure 3B Example operations for performing such automatic transcription are described in more detail. Process 200 may then proceed to another decision stage 208 to determine whether a send command has been received. In response to determining that a send command has been received, process 200 may include sending the draft email to the target destination. Otherwise, process 200 may include saving the email as a draft or performing other suitable operations at stage 211. In response to determining that a transcription command has not been received, process 200 proceeds directly to decision stage 208.

[0055] like Figure 3B As shown, the operation may include transmitting a copy of the digital image to a text recognizer at stage 212. The operation may then include receiving digital text data from the text recognizer at stage 214. Figure 3C Example operations performed by the text recognizer to transcribe the digital image are described in more detail.Then, at stage 216, in addition to or in lieu of the digital image, the operations may include automatically inserting the digital text into the body portion of the draft email.

[0056] Figure 3Cis a flow chart illustrating an example operation of automatically transcribing a digital image by a text recognizer according to an embodiment of the disclosed technology. Figure 3C As shown, the operation includes performing text pattern recognition at stage 220. The operation may then include performing structure recognition at stage 222. Figure 1A-2 Example implementations of such text pattern recognition and content structure recognition are described. Operations may then include, at stage 224, applying the recognized content structure to the recognized text.

[0057] Figure 4 is a computing device 300 applicable to certain components of the computing system 100 in FIG. 1. For example, the computing device 300 may be applicable to Figure 1A The computing device 300 may be an email server 106, a text recognizer 107, or a client device 102. In a very basic configuration 302, the computing device 300 may include one or more processors 304 and a system memory 306. A memory bus 308 may be used to communicate between the processor 304 and the system memory 306.

[0058] Depending on the desired configuration, the processor 304 can be of any type, including but not limited to a microprocessor (μP), a microcontroller (μC), a digital signal processor (DSP), or any combination thereof. The processor 304 can include one or more levels of cache, such as a level 1 cache 310 and a level 2 cache 312, a processor core 314, and registers 316. The example processor core 314 can include an arithmetic logic unit (ALU), a floating point unit (FPU), a digital signal processing core (DSP core), or any combination thereof. An example memory controller 318 can also be used with the processor 304, or in some implementations, the memory controller 318 can be an internal part of the processor 304.

[0059] Depending on the desired configuration, system memory 306 may be of any type, including but not limited to volatile memory (e.g., RAM), non-volatile memory (e.g., ROM, flash memory, etc.), or any combination thereof. System memory 306 may include an operating system 320, one or more applications 322, and program data 324. The depicted basic configuration 302 is shown by those components within the inner dashed line.

[0060] The computing device 300 may have additional features or functions and additional interfaces to facilitate communication between the basic configuration 302 and any other devices and interfaces. For example, a bus / interface controller 330 may be used to facilitate communication between the basic configuration 302 and one or more data storage devices 332 via a storage interface bus 334. The data storage device 332 may be a removable storage device 336, a non-removable storage device 338, or a combination thereof. Examples of removable storage devices and non-removable storage devices include magnetic disk devices such as floppy disk drives and hard disk drives (HDDs), optical disk drives (such as compact disk (CD) drives or digital versatile disk (DVD) drives), solid state drives (SSDs), and tape drives, etc. Example computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer readable instructions, data structures, program modules, or other data). The term "computer-readable storage medium" or "computer-readable storage device" does not include propagation signals and communication media.

[0061] System memory 306, removable storage device 336, and non-removable storage device 338 are examples of computer-readable storage media. Computer-readable storage media include, but are not limited to: RAM, ROM, EEPROM, flash memory or other memory technology; CD-ROM, digital versatile disk (DVD) or other optical storage; cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices; or any other medium that can be used to store the desired information and can be accessed by computing device 300. Any such computer-readable storage media can be part of computing device 300. The term "computer-readable storage media" does not include propagating signals and communication media.

[0062] The computing device 300 may also include an interface bus 340 for facilitating communication from various interface devices (e.g., output devices 342, peripheral interfaces 344, and communication devices 346) to the basic configuration 302 via the bus / interface controller 330. Example output devices 342 include a graphics processing unit 348 and an audio processing unit 350, which may be configured to communicate with various external devices (e.g., a display or speakers) via one or more A / V ports 352. Example peripheral interfaces 344 include a serial interface controller 354 or a parallel interface controller 356, which may be configured to communicate with external devices such as input devices (e.g., keyboards, mice, pens, voice input devices, touch input devices, etc.) or other peripheral devices (e.g., printers, scanners, etc.) via one or more I / O ports 358. Example communication devices 346 include a network controller 360, which may be arranged to facilitate communication with one or more other computing devices 362 via one or more communication ports 364 over a network communication link.

[0063] A network communication link can be an example of a communication medium. Communication media can generally be embodied by computer-readable instructions, data structures, program modules, or other data in a modulated data signal (e.g., a carrier wave or other transmission mechanism), and can include any information transfer medium. A "modulated data signal" can be a signal that sets or changes one or more of its characteristics in a manner that encodes information in the signal. By way of example and not limitation, communication media can include wired media such as a wired network or a direct wired connection, and wireless media such as acoustic, radio frequency (RF), microwave, infrared (IR), and other wireless media. The term computer-readable media used herein can include storage media and communication media.

[0064] The computing device 300 may be implemented as part of a small form factor portable (or mobile) electronic device such as a cell phone, a personal digital assistant (PDA), a personal media player device, a wireless network watch device, a personal headset device, a dedicated device, or a hybrid device including any of the above functions. The computing device 300 may also be implemented as a personal computer including laptop and non-laptop configurations.

[0065] According to the above, it can be understood that the specific embodiments of the present disclosure have been described herein for the purpose of illustration, but various modifications can be made without departing from the present disclosure. In addition, in addition to or replacing the elements of other embodiments, many elements in the elements of an embodiment can be combined with other embodiments. Therefore, the present technology is not subject to restrictions other than the attached claims.

Claims

1. A method for automatic digital image transcription during composition of an electronic message, the method comprising: At a client device, importing the digital image into a body portion of an electronic message being composed in an email client; receiving, at the client device, a transcription command to automatically transcribe the imported digital image in the electronic message; In response to receiving the transcription command, transmitting, via a computer network, a copy of the imported digital image to an email server interconnected to the client device; performing, at the email server, text recognition based on patterns in the imported digital image to generate digital text data corresponding to the imported digital image; determining, at the email server, a content format of the digital text data based on the imported digital image by identifying one or more keywords or punctuation marks that correspond individually to a text structure; At the email server, automatically applying the determined content format to the generated digital text data so that the generated digital text data has the text structure corresponding to the identified one or more keywords or punctuation marks, thereby typeset the digital text data; providing a copy of the typeset digital text data from the email server to the client device via the computer network; and inserting and appearing, at the client device, the provided copy of the typeset digital text data into a copy of the electronic message being composed in the email client; as well as Upon receiving user input for sending the electronic message being composed at the email client, a copy of the electronic message with the inserted typeset digital text data is transmitted to a destination.

2. The method according to claim 1, further comprising: receiving a request from the user to import the digital image into the electronic message; as well as In response to receiving the request, importing the digital image into a body portion of the electronic image; and Wherein performing text recognition includes performing text recognition based on patterns in the imported digital image in the body portion of the electronic message to generate digital text data corresponding to the imported digital image.

3. The method according to claim 1, wherein: Performing pattern recognition includes performing text recognition based on a pattern model for recognizing handwriting patterns of characters in the imported digital image to generate digital text data corresponding to the imported digital image.

4. The method according to claim 1, wherein: Determining the content format includes determining one or more of a list, a table, or a paragraph according to which the digital text data is to be arranged; and Automatically applying the determined content format includes automatically arranging the digital text data into one or more of a list, a table, or paragraphs.

5. The method according to claim 1, wherein: Determining the content format includes determining one or more of a list, table, or paragraph according to which the digital text data is to be arranged based on one or more of keywords, punctuation, or content appearance; and Automatically applying the determined content format includes automatically arranging the digital text data into one or more of a list, a table, or a paragraph based on the one or more of keywords, punctuation, or content appearance.

6. The method according to claim 1, wherein: The determined content format is a list; and The content formats determined by the automatic application include: automatically arranging the digital text data into columns; as well as Insert one or more of a bullet point or a numbered heading before each item in the column.

7. The method according to claim 1, wherein: Inserting the typeset digital text data includes inserting the typeset digital text data into a copy of the electronic message being composed in place of the imported digital image.

8. The method according to claim 1, wherein: Inserting the typeset digital text data includes inserting the typeset digital text data into a copy of the electronic message being composed in addition to the imported digital image.

9. The method according to claim 1, further comprising: After inserting said typeset digital text data into a copy of said electronic message being composed, presenting the inserted typeset digital text data to the user; and Upon receiving the user input for sending the electronic message being composed, one or more edits to the rendered layout digital text data are received prior to transmitting the electronic message with the layout digital text data to the destination.

10. A computing device for processing electronic messages, the computing device being interconnected to a remote server via a computer network, wherein: The computing device comprises: Processor; and A memory containing instructions executable by the processor to cause the computing device to perform a process according to one of claims 1-9.

Images