Autocorrection candidate selection

Abstract

A data processing system implements receiving textual content from a first application on a first client device associated with a first user for analysis by a writing assistant configured to provide suggestions for improving the textual content. The textual content is analyzed by one or more NLP models configured to receive the textual content as input and to provide candidate suggestions for correcting errors and / or otherwise improving the textual content. The candidate suggestions are associated with a suggestion category, and an auto-correction model makes a determination as to whether auto-correction is suitable for the suggestion category. The textual content is automatically corrected based on those candidate suggestions that are automatically more appropriate, while the remainder of the candidate suggestions may be presented to the user for approval before the candidate suggestions are applied to the textual content.

Description

Background Art

[0001] There are various automated tools for proofreading the text content of electronic documents and for providing suggestions for improving the grammar, spelling, vocabulary, punctuation or other aspects of the text content. Typically, these tools present these suggestions to the user, and the user can accept, reject or ignore these suggestions. However, presenting such suggestions and receiving feedback from the user may interrupt the user's workflow. In contrast, other tools automatically correct the text content of electronic content to implement these suggestions. However, the user may not want and / or disagree with the automatic correction and must manually undo the automatic correction. Therefore, the automatic correction feature may become intrusive, interrupting the workflow, and reducing the user experience by automatically correcting in places where the automatic correction is not considered useful by the user. Therefore, it is necessary to provide an improved system and method for implementing a technical solution for a context-adaptive writing assistant that adapts when applying automatic correction. Summary of the Invention

[0002] An example data processing system according to the present disclosure may include a processor and a machine-readable medium storing executable instructions. The instructions, when executed, cause the processor to perform operations comprising: receiving textual content from a first application on a first client device associated with a first user for analysis by a writing assistant, the writing assistant configured to provide suggestions for improving the textual content; analyzing the textual content using one or more natural language processing (NLP) models to identify one or more candidate suggestions for improving the textual content; determining that a first candidate suggestion of the one or more candidate suggestions is associated with a first suggestion category, the first suggestion category being associated with first model performance information indicating the performance of a first NLP model of the one or more NLP models that generated the first candidate suggestion, the first suggestion category being further associated with first user acceptance information indicating a user acceptance rate of previous candidate suggestions of the first suggestion category; determining that the first model performance information and the first user acceptance information associated with the first suggestion category meet an auto-correction threshold criterion; in response to determining that the model performance threshold and the user acceptance information associated with the first suggestion category meet the auto-correction threshold criterion, generating auto-corrected textual content by updating the textual content based on the first candidate suggestion; and providing the auto-corrected textual content to the first client device.

[0003] An example data processing system according to the present disclosure may include a processor and a machine-readable medium storing executable instructions. The instructions, when executed, cause the processor to perform operations comprising: receiving textual content from a first application on a first client device associated with a first user for analysis by a writing assistant, the writing assistant configured to provide suggestions for improving the textual content; analyzing the textual content using one or more natural language processing (NLP) models to identify one or more candidate suggestions for improving the textual content; causing the first application of the first client device to display the one or more candidate suggestions; receiving feedback from the first client device including user acceptance information associated with the one or more candidate suggestions; aggregating the user acceptance information included in the feedback with historical user acceptance information to obtain aggregated user acceptance information; obtaining model performance information associated with the one or more NLP models; analyzing the aggregated user acceptance information and the model performance information to determine whether the aggregated user acceptance information and the model performance information meet an auto-correction threshold criterion; and training an auto-correction model based on the determination of whether the aggregated user acceptance information and the model performance information meet the auto-correction threshold criterion.

[0004] An example method implemented in a data processing system for providing suggestions for improving text content includes: receiving text content from a first application on a first client device associated with a first user for analysis by a writing assistant, the writing assistant configured to provide suggestions for improving the text content; analyzing the text content using one or more natural language processing (NLP) models to identify one or more candidate suggestions for improving the text content; determining that a first candidate suggestion of the one or more candidate suggestions is associated with a first suggestion category, the first suggestion category being associated with first model performance information indicating the performance of a first NLP model of the one or more NLP models that generated the first candidate suggestion, the first suggestion category being further associated with first user acceptance information indicating a user acceptance rate of previous candidate suggestions of the first suggestion category; determining that the first model performance information and the first user acceptance information associated with the first suggestion category meet an auto-correction threshold criterion; in response to determining that the model performance threshold and the user acceptance information associated with the first suggestion category meet the auto-correction threshold criterion, generating auto-corrected text content by updating the text content based on the first candidate suggestion; and providing the auto-corrected text content to the first client device.

[0005] This Summary is provided to introduce a set of concepts in a simplified form that are further described in the Detailed Description below. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The accompanying drawings depict one or more embodiments of the present teachings by way of example only and not limitation. In the accompanying drawings, like reference numerals represent identical or similar elements. In addition, it should be understood that the accompanying drawings are not necessarily drawn to scale.

[0007] 1 is a diagram illustrating an example computing environment in which the techniques disclosed herein for providing a context-adaptive writing assistant may be implemented.

[0008] 2 is a diagram illustrating another example computing environment in which the techniques disclosed herein may be implemented.

[0009] FIG. 3 is a diagram illustrating additional features of the writing assistant service, client device, and application service shown in FIG. 2 .

[0010] 4A-4G are diagrams of example user interfaces illustrating examples of the auto-correction techniques disclosed herein.

[0011] 5A-5E are examples of data structures that may be used to store data for various aspects of the auto-correction techniques disclosed herein.

[0012] 6A is a flow diagram of an example process for providing automatic corrections in accordance with the techniques disclosed herein.

[0013] 6B is a flow diagram of an example process for training an auto-correction model, such as the auto-correction model shown in the previous examples.

[0014] Figure 7 is a block diagram illustrating an example software architecture, portions of which may be used in conjunction with the various hardware architectures described herein, which may implement any of the features described.

[0015] 8 is a block diagram illustrating components of an example machine configured to read instructions from a machine-readable medium and perform any of the features described herein. DETAILED DESCRIPTION

[0016] Technologies are provided for providing automatic correction candidate selection. These technologies address technical issues associated with current writing assistants and proofing tools that provide automatic correction features to improve a user's handwriting. Current writing assistants are designed to provide suggestions for correcting and / or automatically correcting certain types of issues detected in text content drafted by the user. These issues may include, but are not limited to, spelling, grammar, punctuation, capitalization, and / or style issues detected by the writing assistant. Suggestions are typically presented to the user via a pop-up window or other user interface element that identifies a portion of the text content associated with the suggestion and presents the suggested improvements to the text content. The user can then choose to accept, reject, or ignore the suggestions. Current writing assistants also implement autocorrection to automatically apply certain types of suggested improvements without user input. When applied correctly, autocorrection improves user workflow and user experience by helping users automatically improve the text content they are producing. However, in some cases, users may not want or agree to autocorrection. As a result, autocorrection features can become intrusive, interrupting workflow, and degrading the user experience by automatically correcting text in areas where the user would not consider it useful.

[0017] The technology disclosed herein provides a writing assistant that implements autocorrection candidate selection, which identifies candidates for autocorrection, determines whether autocorrection is appropriate for each candidate, and performs autocorrection on candidates for which autocorrection is appropriate. The autocorrection technology herein utilizes a quantitative approach to determine whether autocorrection is appropriate for a specific scenario by utilizing several criteria for determining when autocorrection is appropriate. These criteria may include model performance information and user acceptance information. Model performance information includes accuracy and suggestion precision. Accuracy is a quantitative measure of confidence that a suggestion candidate representing an error or potential improvement in text content has been correctly detected. Suggestion precision is a quantitative measure of confidence that a suggestion for improving a suggestion candidate is an appropriate suggestion. User acceptance information indicates whether the user has previously adopted the suggestion when presented to the user, and / or whether the user has previously rejected the autocorrection if the suggestion was automatically applied to the suggestion candidate. These and / or other factors are used to generate an autocorrection model that determines when to apply autocorrection in various use cases. The technical benefit of this method is that it improves user experience and workflow by applying autocorrection that is appropriate for a specific use case. This approach reduces the likelihood that the user will reject the autocorrection and restore the text content to the state it was in before the autocorrection was applied.These and other technical benefits of the technology disclosed herein will be apparent from the following discussion of example implementations.

[0018] FIG1 is a diagram illustrating an example computing environment 100 in which the techniques disclosed herein for providing automatic correction candidate selection may be implemented. The computing environment 100 illustrates various elements associated with an implementation of a writing assistant service 190 that is configured to analyze text content to detect spelling, grammar, punctuation, capitalization, style issues, and / or other issues with the text content. The writing assistant service 190 is further configured to provide suggestions for improving these issues presented to the user and / or automatically corrected by the writing assistant service 190. The text content is received from a client-side interface unit 105.

[0019] In some embodiments, the client-side interface unit 105 is implemented on the client device of the user who is creating the text content. In such an embodiment, the client-side interface unit 105 can be implemented on the user's client device to provide support for a writing assistant service in one or more native applications installed on the client device, or can be implemented by a native application to provide support for a writing assistant service for the application. In other embodiments, the client-side interface unit 105 is implemented by a cloud-based application service. In such an embodiment, the client-side interface unit 105 provides support for a writing assistant service of one or more web applications available to users of the cloud-based application service. The writing assistant service can provide proofreading and suggestions for text content that is being created and / or modified using native applications and web-based applications.

[0020] When a user creates or edits text content in a native application or a web-based application, the client-side interface unit 105 provides at least a portion of the text content to the client-side interface unit 105 for analysis by the writing assistant service 190. The client-side interface unit 105 sends the text content to the writing assistant service 190 via a network connection. The text content may include a sentence, multiple sentences, paragraphs, pages, or other portions of the text content to be analyzed. The text content may be a portion of an electronic document, an email, a text message, a chat message, and / or other types of electronic content that may be improved through suggested improvements and / or automatic corrections provided by the writing assistant service 190. The specific type of text content depends on the native application or web application from which the text content has been received. The writing assistant service may be provided to various different types of native applications and / or web-based applications, such as, but not limited to, word processing applications, collaboration platforms, messaging applications, web browsers, presentation design applications, and / or other types of applications in which text content can be created. In response to sending the text content to the writing assistant service 190 , the client-side interface unit 105 receives suggestions from the writing assistant service 190 that can be presented to the user via the application's user interface and / or autocorrected to be automatically applied to the text content without involving the user.

[0021] The client-side interface unit 105 is also configured to provide feedback to the writing assistant service 190 regarding the suggestions and / or auto-corrections provided. Regarding the suggestions, the user can choose to accept the suggestion, reject the suggestion, or simply ignore the rejection. The native application or web application provides a user interface that provides controls that enable the user to accept, reject, or ignore the suggestions provided by the writing assistant service 190. The user interface also provides an indication of the user's selection to the client-side interface unit 105, which sends the indication to the writing assistant service 190 for processing by the model development pipeline 110. This information is used to train and / or fine-tune the training of the auto-correction model 135.

[0022] The writing assistant service 190 includes a model development pipeline 110, a proof pipeline 140, an auto-correction model 135, and a writing assistant microservice 160. The model development pipeline 110 is used to train and / or fine-tune the auto-correction model 135 used by the writing assistant service 190 to determine whether auto-correction is appropriate for a specific problem identified in textual content. The proof pipeline 140 is configured to analyze textual content to identify candidate suggestions for improving the textual content and to make auto-correction decisions regarding these candidate suggestions. The writing assistant microservice 160 is configured to analyze textual content and provide suggestions for improving the textual content.

[0023] The model development pipeline 110 includes a feedback unit 115, a data aggregation unit 120, a data analysis unit 125, and a performance evaluation unit 130. The feedback unit 115 is configured to receive feedback from the writing assistant service 190, provided by the client-side interface unit 105, on whether the user accepts, rejects, or ignores the suggestion. The feedback includes an identifier for the suggestion category associated with the suggestion for which the feedback is provided and a user response indicator indicating whether the user accepts, rejects, or ignores the suggestion. The indication provided by the client-side unit 105 does not include any user-specific information that can be associated with a particular user, to avoid any privacy issues associated with collecting such information from users. The collected information is used to improve the determination of whether a suggestion of a particular category can be automatically corrected or should be presented to the user for consideration instead. The data aggregation unit 120 is configured to aggregate the data received by the feedback unit 115 by suggestion type. The data can be aggregated in various ways. In some embodiments, the data associated with each suggestion type is aggregated into a cumulative total of the number of suggestions presented to the user and the corresponding number of accept, reject, and ignore responses associated with those suggestions. In some embodiments, the sum of the accept, reject, and ignore responses is expressed as a percentage. Examples of aggregated user data obtained through such an aggregation process are shown in Figures 5B and 5C, which are discussed in detail in the examples below. In some embodiments, the feedback unit 115 writes the feedback to the feedback data structure shown in Figure 5A, and the data aggregation unit 120 is configured to read data from the data structure to determine the cumulative value shown in the data structure shown in Figures 5B and 5C.

[0024] The performance evaluation unit 130 is configured to obtain information indicating the accuracy and suggestion precision associated with a specific suggestion category. In the example shown in Figure 1, suggestions for improving text content are generated by a writing assistant microservice 160. The writing assistant microservice includes a style service 165, a spelling service 170, a punctuation service 175, a grammar service 180, and a capitalization service 185, each of which is configured to identify specific types of problems in the text content and provide suggestions for correcting these problems. The accuracy and suggestion precision of the suggestions provided by each of these microservices can be determined in various ways. In some embodiments, the accuracy and suggestion precision of the suggestions provided are determined by providing a set of reference data to the microservice and determining, based on the reference data, how close the result provided by the microservice is to the expected result. In some embodiments, the accuracy and suggestion precision determined for each suggestion category are stored in the data structure shown in Figure 5D, which is described in more detail in the example below.

[0025] The data analysis unit 125 combines the data received from the data aggregation unit 120 and the performance evaluation unit 130. The data analysis unit 125 is configured to determine whether the accuracy rate, suggestion precision, and acceptance rate associated with a particular category of suggestions exceed an autocorrection threshold. The autocorrection threshold sets minimum values ​​for accuracy rate, suggestion precision, and minimum or maximum values ​​for one or more user factors that must be met for a particular suggestion category in order for the writing assistant service 190 to perform autocorrection for that suggestion category. In some embodiments, the user factors provide a minimum user acceptance threshold percentage and / or a maximum user ignore percentage threshold. If the accuracy rate associated with the suggestion category meets the accuracy threshold, the suggestion precision associated with the suggestion category meets the suggestion precision threshold, and one or more user factor thresholds are also met, then autocorrection can be performed. In some embodiments, the same autocorrection threshold is applied to all suggestion categories. In other embodiments, different autocorrection thresholds can be associated with different suggestion categories. The autocorrection threshold can be configured by one or more administrators of the writing assistant service 190. Furthermore, changes in the auto-correction threshold and / or aggregated user data associated with a particular suggestion category may trigger data analysis unit 125 to re-evaluate whether auto-correction is appropriate for that suggestion category.

[0026] Data analysis unit 125 is configured to update auto-correction model 135 in response to determining whether the auto-correction is suitable for a particular suggestion category. Data analysis unit 125 is configured to update auto-correction model 135 by updating the auto-correction model to identify that the auto-correction is suitable for a particular suggestion category. How the update is implemented depends on the specific implementation of auto-correction model 135.

[0027] In some embodiments, the auto-correction model 135 is implemented as a rule-based model that is configured to provide a determination of whether a specified suggestion category is a suggestion category for which the writing assistant service 190 can automatically correct an issue in the text content. In some embodiments, the data analysis unit 125 adds or updates rules to the auto-correction model 135 in response to determining whether auto-correction is appropriate for a particular suggestion category. In some embodiments, the data analysis unit 125 is configured to store the rules in a data structure similar to the data structure shown in FIG5E . By default, auto-correction is not allowed for every auto-correction category. However, in response to the data analysis unit 125 determining that auto-correction is allowed for a particular suggestion category, the data structure 530 shown in FIG5E is updated to indicate that auto-correction is allowed for that suggestion category. Similarly, in response to the data analysis unit 125 determining that auto-correction is not allowed for a particular suggestion category, the data structure shown in FIG5E is updated to indicate that auto-correction is not allowed for that suggestion category. When the auto-correction model 135 receives a request from the auto-correction determination unit 150 regarding whether a suggestion category associated with a candidate suggestion can be automatically corrected by the writing assistant service 190, the auto-correction model 135 accesses the data structure 530 to determine whether auto-correction is allowed. In some embodiments, the data structure 530 can be sparsely populated, meaning that records for particular suggestion categories are only added as the model development pipeline 110 receives and analyzes feedback. In such embodiments, if no record for a suggestion category exists in the data structure 530, the auto-correction model 135 returns a determination that auto-correction is not allowed for the suggestion category. As described above, the data in the data structure 530 can continue to be updated by the model development pipeline 110 as feedback is received from the user via the client-side interface unit 105 and processed by the model development pipeline 110.

[0028] In some embodiments, the auto-correction model 135 is implemented as a machine learning model that is configured to classify candidate suggestions as permissible or impermissible based on the suggestion category. In some embodiments, the data analysis unit 125 is configured to generate training data that associates the suggestion category with an indication of whether the auto-correction is permissible.

[0029] The proof pipeline 140 is configured to analyze text content received from the client-side interface unit 105 to identify candidate suggestions for improving the text content and determine whether auto-correction is appropriate for each candidate suggestion. The proof pipeline 140 sends the auto-corrected text to the client-side interface unit 105 to obtain candidate suggestions for which auto-correction is predicted to be appropriate. The proof pipeline 140 includes a content inspection unit 145, an auto-correction determination unit 150, and an auto-correction execution unit 155.

[0030] The content inspection unit 145 is configured to receive text content provided by the client-side interface unit 105 and submit the text content to the writing assistant microservice 160 for analysis. The writing assistant microservice 160 analyzes the text content to obtain candidate suggestions for improving the text content. The content inspection unit 145 provides the candidate suggestions to the automatic correction determination unit 150. For each candidate suggestion, the automatic correction determination unit 150 queries the automatic correction model for the suggestion category associated with the candidate suggestion to the automatic correction model to obtain a determination of whether the writing assistant service 190 is allowed to automatically correct the text content using the candidate suggestion. The automatic correction determination unit 150 provides the candidate suggestion and the automatic correction indicator for the candidate suggestion to the execution automatic correction unit 155. The automatic correction indicator indicates whether automatic correction is authorized.

[0031] For each candidate suggestion, if the autocorrect indicator indicates that autocorrection is authorized, the autocorrection unit 155 automatically corrects the text input according to the candidate suggestion. The autocorrection unit 155 automatically corrects the text content by updating the text content according to the candidate suggestion. If the candidate suggestion is a spelling-related problem, the autocorrection unit 155 updates the text content using the corrected spelling provided in the candidate suggestion. The autocorrection unit 155 similarly updates the text content to correct grammar, style issues, punctuation, capitalization and / or other types of problems associated with candidate suggestions of other categories. In some cases, the text content is associated with multiple candidate suggestions that are authorized for autocorrection, and the autocorrection unit 155 updates the text content according to each of these candidate selections. The automatically corrected text content is sent to the client-side interface unit 105, which is configured to cause a native application or a web-based application to replace the text content at the application with the automatically corrected text content. The client-side interface unit 105 is configured to determine whether the original text content has been modified at the native application or web-based application since the text content was sent to the writing assistant service 190 for processing. If the original text content has been modified, the client-side interface unit 105 discards the automatically corrected text content and resubmits the modified text content from the native application or web-based application for processing. This protection ensures that changes introduced by the user at the native application or web-based application are not lost by replacing the modified text content with the automatically corrected text content.

[0032] The perform auto-correction unit 155 sends candidate suggestions that cannot be automatically corrected to the client-side interface 105 for presentation to the user. The candidate suggestions include offset information and / or other information that can be used by the client-side user interface 105 and / or the native application or web-based application to present the candidate suggestions to the user. In the event that some candidate selections can be automatically corrected by the writing assistant service 190 and other candidate selections cannot be automatically corrected, the perform auto-correction unit 155 performs auto-correction for the candidate suggestions that allow auto-correction and sends the auto-corrected text content and the remaining candidate suggestions to the client-side interface unit 105. The perform auto-correction unit 155 updates the offset information and / or other information that can be used by the client-side user interface 105 and / or the native application or web-based application to present the candidate suggestions to the user based on the auto-corrected text content, so that the candidate suggestions are aligned with the auto-corrected text content when presented by the native application or web-based application.

[0033] The writing assistant microservice 160 is implemented by one or more cloud-based services accessible to the user's client device via a network connection. In the example embodiment shown in Figure 1, the writing assistant microservice 160 includes a style service 165, a spelling service 170, a punctuation service 175, a grammar service 180, and a capitalization service 185. However, other embodiments may include a different number of such microservices. In addition, in some embodiments, the functionality of two or more of these microservices can be combined into a single microservice. In some embodiments, the writing assistant microservice 160 utilizes a natural language processing (NLP) model that is trained to analyze text input and output suggestions for improving the text input.

[0034] 2 is a diagram illustrating an example computing environment 200 in which the techniques disclosed herein for providing a writing assistant service may be implemented. The computing environment 200 includes the writing assistant service 190. The example computing environment 200 also includes client devices 205a, 205b, 205c, and 205d (collectively, client devices 205) and an application service 225. The client devices 205a, 205b, 205c, and 205d communicate with the writing assistant service 190 and / or the application service 225 via a network 220. In addition, the application service 225 communicates with the writing assistant service 190 via the network 220. The network 220 may be a combination of one or more public and / or private networks and may be implemented, at least in part, by the Internet.

[0035] 2 , the writing assistant service 190 is implemented as a cloud-based service or set of services. The writing assistant service 190 implements functionality for providing suggestions for improving text content and / or for automatically correcting text content based on these suggestions, as discussed with respect to FIG. 1 .

[0036] The writing assistant service 190 is configured to provide writing assistant services to applications provided by the application service 225 and / or native applications on the client devices 205a-205d. The application service 225 and / or the applications of the client devices 205a-205d send text content created or edited in the applications to the writing assistant service 190. As discussed in the aforementioned examples, the writing assistant service 190 can provide writing assistant functionality to various different types of applications on the client devices, such as, but not limited to, word processing applications, collaboration platforms, messaging applications, web browsers, presentation design applications, and / or other types of applications in which text content can be created. The text content can include text from a word processing document, a presentation, a text message, an email message, a chat message, and / or other types of text content.

[0037] The application service 225 provides cloud-based software and services that are accessible to users via client devices 205a-205d. The application service 225 provides one or more software applications, including but not limited to communication platforms and / or collaboration platforms, word processing applications, presentation design applications and / or other types of applications in which users can create and / or access text content. The text content can be stored on the writing assistant service 190 and / or the client devices 205a-205d. The term "text content" as used herein can represent any document or component in electronic form that can be created by a computing device, stored in a machine-readable storage medium and / or transmitted between computing devices via a network connection or via a machine-readable storage medium. Examples of such electronic documents include but are not limited to word processing documents, program code, presentations, websites (e.g., Microsoft sites), digital drawings, media files, components thereof, etc. One or more software applications provided by application service 225 are configured to utilize the services provided by writing assistant service 190 to analyze text content and provide suggestions for improving the text content.

[0038] Each of the client devices 205a, 205b, 205c, and 205d is a computing device that can be implemented as a portable electronic device, such as a mobile phone, a tablet computer, a laptop computer, a portable digital assistant device, a portable game console, and / or other such devices. The client devices 205a, 205b, 205c, and 205d can also be implemented in computing devices having other form factors, such as a desktop computer, an in-vehicle computing system, an information kiosk, a point-of-sale system, a video game console, and / or other types of computing devices. Although the example embodiment shown in FIG2 includes four client devices, other embodiments can include a different number of client devices that utilize the application service 225 and / or the writing assistant service 190. In addition, in some embodiments, the application functionality provided by the application service 225 is implemented by native applications installed on the client devices 205a, 205b, 205c, and 205d, and the client devices 205a, 205b, 205c, and 205d communicate directly with the writing assistant service 190 via a network connection.

[0039] In some embodiments, the writing assistant service 190 or at least a portion of its functionality is implemented by the application service 225 to provide writing assistant-related services to users of the application service. In other embodiments, the writing assistant service 190 or at least a portion of its functionality is implemented by a native application on the client device 205a-205d.

[0040] 3 is a diagram illustrating additional features of the writing assistant service 190, the client device 205, and the application service 225. The writing assistant service 190 includes a request processing unit 305 and an authentication unit 325. The writing assistant service 190 also includes the model development pipeline 110, the certification pipeline 140, the auto-correction model 135, and the writing assistant microservice 160 shown in FIG1 .

[0041] The request processing unit 305 receives feedback and / or text content to be analyzed from the client-side interface unit 105. The request processing unit 305 also sends suggested and / or automatically corrected text content to the client-side interface unit 105. The request processing unit 305 provides feedback to the model development pipeline 110, as discussed with respect to FIG1. ​​The request processing unit 305 provides the text content to the proof pipeline 140 for analysis.

[0042] The authentication unit 325 provides functionality for verifying whether a user is allowed to access the services provided by the writing assistant service 190. In some implementations, the authentication unit 325 provides functionality for receiving the user's authentication credentials from the user's corresponding client device 205 and / or from the application service 225. The authentication unit 325 can be configured to verify that the authentication credentials are valid and, in response to the authentication credentials being valid, allow the user to access the services provided by the writing assistant service 190.

[0043] The application service 225 includes an application service unit 360, an authentication unit 365, a web application 370, and a client-side interface unit 385. The application service unit 360 provides users with the ability to consume, create, share, collaborate, and / or modify various types of text content, including text content that can be analyzed by the writing assistant service 190. The application service unit 360 can utilize the writing assistant service 190 to analyze text content being created or edited by the user of the client device 205. In some embodiments, the application service unit 360 provides a web-based application 370 to enable users to access at least a portion of the services provided by the application service 225. In other embodiments, users can access the services provided by the application service 225 via one or more native applications 350 installed on the client device 205. The application service unit 360 can then obtain the services provided by the writing assistant service 190.

[0044] The authentication unit 365 provides functionality for verifying whether a user is permitted to access services provided by the application service 225 and / or the writing assistant service 190. In some embodiments, the authentication unit 365 provides functionality for receiving a user's authentication credentials from the user's corresponding client device 205. In such embodiments, the authentication unit 365 verifies that the authentication credentials are valid and, in response to the authentication credentials being valid, allows the user to access services and / or documents provided by the application service 225 and / or the writing assistant service 190.

[0045] The client device 205 may include one or more native applications 350 and / or browser applications 355. In some embodiments, the one or more native applications 350 include native applications configured to communicate with the application service 225 to enable a user to consume, create, share, collaborate, and / or modify text content using services provided by the application service 225. In some embodiments, the one or more native applications 350 include native applications configured to communicate with the writing assistant service 190. In such embodiments, the native applications provide an interface for the user to interact with the writing assistant service 190 and / or the application service 225.

[0046] The browser application 355 is an application for accessing and viewing web-based content, which may be provided by the application service 225 and / or the writing assistant service 190. The application service 225 may provide a web application 370 that enables users to consume, create, share, collaborate, and / or modify content. A user of the client device 205 may access the web application 370 via the browser application 355, and the browser application 355 presents a user interface for interacting with the application service 225 in the browser application 355. The browser application 355 may be used to access services provided by the application service 225 and / or the writing assistant service 190.

[0047] The application service 225 and / or the writing assistant service 190 can support both one or more web-enabled native applications 350 and one or more web applications 370, and the user can choose which approach best suits their needs. The writing assistant service 190 can also provide support for one or more native applications 350, browser applications 355, or both to provide the user of the client device 205 with functionality for obtaining services provided by the writing assistant service 190.

[0048] Figures 4A through 4G illustrate examples of user interfaces in which the auto-correction techniques provided herein may be implemented. The embodiments shown in Figures 4A through 4C illustrate an example user interface 402 for a messaging application. The embodiments shown in Figures 4D through 4G illustrate an example user interface 420 for an email application. User interfaces 402 and 420 may be generated by a native application 350 of a client device 205 or a web application 370 of an application service 225.

[0049] The examples shown in Figures 4A, 4C, and 4D illustrate examples of improved text messaging applications that utilize a writing assistant service 190 to provide suggestions for the text content of a text message, wherein the context-adaptive writing assistant functionality is disabled. The examples shown in Figures 4B and 4E illustrate examples of text messaging applications that utilize a writing assistant service provided by a writing assistant service 190. The writing assistant service 190 provides suggestions for improving the text content of a message being written in a messaging application, and / or automatically corrects the text content according to the techniques provided herein. In cases where the user may have accepted the suggested improvements to the text content of the message, the accuracy rate indicates that the writing assistant service 190 correctly detected errors in the text content or potential improvements to the text content, and the suggestion accuracy indicates that the writing assistant service 190 has provided appropriate suggestions for errors or potential improvements to the text content. The technical benefit of this method is that the writing assistant provides fewer notifications, which may distract the user and interrupt the user's workflow.

[0050] The user interface 402 shown in Figure 4A includes a text input field 404 in which a user can enter text messages to one or more recipients. The text of these text messages is sent to the writing assistant service 190 for analysis by the client-side interface unit 105. In the example shown in Figure 4A, the writing assistant service 190 has detected an error in the text content of the message. The error is a missing comma after the greeting "Hey" in the text message. The proof pipeline 140 of the writing assistant service 190 analyzes the text content as discussed in the aforementioned examples to identify candidate suggestions for correcting the error. The automatic correction determination unit 150 then queries the automatic correction model 135 to determine whether the writing assistant service 190 can automatically correct the detected error or whether the candidate suggestions for correcting the error should be presented to the user to allow the user to decide whether to accept, reject, or ignore the suggestion.

[0051] The auto-correction information 406 shown in FIG4A illustrates example auto-correction determination information 406 used by writing assistant service 190 to determine whether writing assistant service 190 is permitted to automatically correct text content using candidate suggestions. In the example shown in FIG4A , the judgment type is the "Comma after Greeting" judgment type. For this question, the accuracy associated with this suggestion category is 95%, the suggestion accuracy is 98%, the user acceptance rate is 98%, the user rejection rate is 1%, and the user ignore rate is 1%. The auto-correction thresholds for this example include an accuracy greater than 95%, a suggestion accuracy greater than 95%, a user acceptance rate greater than 75%, and a user ignore rate less than 5%. In this example embodiment, the "Comma after Greeting" suggestion category meets the auto-correction thresholds. Therefore, writing assistant service 190 is permitted to automatically correct the question without presenting a suggestion to the user to correct the question and receiving user input regarding the suggestion. FIG4B illustrates the textual content of text input field 404 after performing the auto-correction.

[0052] The autocorrection determination information 406 will typically be maintained internally by the writing assistant service 190 and will not be presented to the user on the user interface 402. The autocorrection determination information 406 is provided in FIG4A to illustrate an example of information that may be used by the writing assistant service 190 to determine whether an autocorrection is appropriate.

[0053] Figure 4C shows another example in which the autocorrection information 406 indicates that the writing assistant service 190 is not allowed to autocorrect the comma after the greeting suggestion category. In the example shown in Figure 4C, the accuracy and precision of the suggestion category are below the autocorrect threshold. For this question, the accuracy associated with the suggestion category is 93%, the suggestion precision is 94%, the user acceptance rate is 98%, the user rejection rate is 1%, and the user ignore rate is 1%. The 93% accuracy for the suggestion category is below the autocorrect threshold accuracy of greater than 95%, and the 94% suggestion precision for the suggestion category is below the autocorrect threshold suggestion precision of greater than 95%. In this example embodiment, the comma after the greeting suggestion category does not meet the autocorrect threshold. Therefore, the writing assistant service 190 presents the suggestion to the user in the suggestion pane 410.

[0054] In some embodiments, the corresponding portion of the text content associated with the suggested improvements is underlined, highlighted, or otherwise emphasized to provide a visual indication to the user that there are associated suggested improvements. The location information included with the suggested improvements is used to ensure that the correct portion of the text content is emphasized to the user. In some embodiments, the user can then interact with the emphasized text to cause a suggestion pane 410 with the suggested improvements to be displayed. The technical benefit of this method is that clutter on the user interface of the application is reduced by displaying only the suggested improvements that the user interacts with.

[0055] The suggestion pane 410 includes controls for the user to accept, reject, or ignore the suggestions provided. The model development pipeline 110 receives the feedback, analyzes the feedback, and can update the auto-correction model 135. The model used by the writing assistant microservice 160 can change over time. The performance evaluation unit 130 is configured to periodically re-evaluate the performance of the writing assistant microservice 160 and provide updated accuracy and suggestion precision information to the data analysis unit 125. The data analysis unit 125 uses this information to determine whether to update the auto-correction model 135. A technical benefit of this approach is that the auto-correction model 135 continues to evolve automatically as user feedback about particular corrections changes over time and / or the performance of the NLP model changes over time.

[0056] Figures 4D to 4G illustrate an example of an example user interface 420 for an email application. The user interface 420 includes a content pane 424 in which the body of the email message can be composed. In the example shown in Figure 4D, the text of the email includes grammatical and spelling issues. Autocorrection determination information 426 is information used by the writing assistant service 190 to determine whether the autocorrection is suitable for the concise suggestion category. Autocorrection determination information 428 is information used by the writing assistant service 190 to determine whether the autocorrection is suitable for the spelling suggestion category. In the example shown in Figure 4D, the autocorrection threshold is met for both the concise suggestion and the spelling suggestion, and the writing assistant service 190 automatically corrects the text content of the message, as shown in Figure 4E.

[0057] Figures 4F and 4G show examples in which one of the candidate suggestion categories can be automatically corrected by the writing assistant service 190. In Figure 4F, the auto-correction information 430 for the Concise category does not meet the auto-correction threshold because the user acceptance rate is too low. However, in this example, the auto-correction information 428 for the Spelling category indicates that the writing assistant service 190 can automatically correct spelling errors in the text. Therefore, the automatically corrected text with corrected spelling errors but including grammatical errors is presented in the content pane 424, and the concise suggestion is highlighted to indicate that the user can click on the highlighted text or otherwise interact with the highlighted text to cause a suggestion pane 438 to be displayed. The suggestion pane 438 provides controls for the user to accept, reject, or ignore the suggestion. In Figure 4G, the auto-correction information 432 for the Spelling suggestion category does not meet the auto-correction threshold because the user acceptance rate is too low. However, in this example, the auto-correction information 426 for the Concise suggestion category indicates that the writing assistant service 190 can automatically correct grammatical errors in the text. Thus, the autocorrected text with corrected grammatical errors but including spelling errors is presented in the content pane 424, and the spelling suggestion is highlighted to indicate that the user can click on or otherwise interact with the highlighted text to cause the suggestion pane 438 to be displayed.

[0058] The specific values ​​of accuracy rate, suggestion precision, user acceptance rate, user rejection rate, user ignore rate, and auto-correction threshold value shown in Figures 4A to 4G are non-limiting examples intended to illustrate the concepts herein. Other embodiments may have different accuracy rates, suggestion precision, user acceptance rate, user rejection rate, user ignore rate, and auto-correction threshold values.

[0059] Figures 5A, 5B, 5C, and 5D are examples of data structures used in some implementations to store data associated with the auto-correction feature provided by the writing assistant service 190. The data structures shown in Figures 5A through 5D are maintained in persistent storage of the writing assistant service 190. In some implementations, the persistent storage is implemented as a database that can be queried by various components of the writing assistant service 190 that utilize the data to provide the services of the writing assistant service 190.

[0060] In response to the user accepting, rejecting, or ignoring the suggestions for correcting errors or improving the text content provided by the writing assistant service 190, the feedback data structure 510 is populated with data obtained from the feedback sent by the client-side interface unit 105 to the writing assistant service 190. In the example embodiment shown in Figure 5A, the user feedback 505 includes a suggestion category and a user response indicator. In this example, the user response indicator is a numerical value of zero, one, or two, where a zero value indicates ignoring the response, a one value indicates accepting the response, and a two value indicates rejecting the response. The feedback unit 115 uses the information from the feedback to populate the feedback data structure 510. The feedback unit 115 also adds a timestamp indicating when the feedback was received by the writing assistant service 190.

[0061] FIG5B illustrates an example of a data structure 515 for aggregating user acceptance data. Data structure 515 stores a cumulative total of feedback received from users in response to suggestions provided by writing assistant service 190. This includes the total number of responses for each suggestion category, as well as how many suggestions for that suggestion category the user accepted, rejected, or ignored. In some embodiments, the values ​​stored in data structure 515 are determined by data aggregation unit 120. Furthermore, in some embodiments, data aggregation unit 120 aggregates all data from feedback data structure 510 to generate the values ​​stored in data structure 515. In other embodiments, data aggregation unit 120 aggregates a subset of data from feedback data structure 510 to generate the values ​​stored in data structure 515. For example, data aggregation unit 120 may select data from the previous month, quarter, year, or other time period to determine the cumulative values ​​stored in data structure 515. Data aggregation unit 120 may utilize this latter approach to ensure that the total reflects the user's current usage patterns and / or to reduce the computational complexity of the calculation by selecting only a recent subset of feedback data. FIG. 5C is an example of a data structure 520 that is similar to data structure 515 , but with the accept, reject, and ignore actions expressed as percentages of the total feedback.

[0062] 5D is an example of a data structure 525 storing accuracy and suggestion precision values ​​for each suggestion category. The values ​​stored in data structure 525 are stored by performance evaluation unit 130. Performance evaluation unit 130 re-evaluates these percentages by testing the performance of writing assistant microservice 160 for detecting errors or potential improvements associated with each suggestion category, and generates candidate suggestions for updating text content to address these issues.

[0063] 5E is an example of a data structure 530 storing an indication of whether auto-correction is permitted for each suggestion category. In some embodiments, the indication of whether auto-correction is permitted is determined by the data analysis unit 125. The data analysis unit 125 is configured to periodically and / or in response to user feedback information and / or changes in the accuracy rate and / or suggestion precision of each suggestion category, and to re-determine these values.

[0064] 6A is an example flow diagram of an example process 600 for providing suggestions for improving text content. The process 600 can be implemented by the writing assistant service 190 shown in the previous examples.

[0065] The process 600 includes an operation 605 of receiving text content from a first application on a first client device 205 associated with a first user. As discussed in the previous example, the native application 350 on the client device 205 or the application provided by the application service 225 provides a user interface for composing and / or editing text content. The text content is sent to the writing assistant service 190 for analysis by the client-side interface unit 105.

[0066] The process 600 includes an operation 610 of analyzing text content using one or more natural language processing (NLP) models to identify one or more candidate suggestions for improving the text content. The proof pipeline 140 of the writing assistant service 190 sends the text content received from the client-side interface unit 105 to the writing assistant microservice 160 for analysis. Each microservice implements an NLP model that is configured to identify errors or potential improvements in the text content and provide candidate suggestions for resolving these issues.

[0067] The process includes an operation 615 of determining that a first candidate suggestion from one or more candidate suggestions is associated with a first suggestion category. The first suggestion category is associated with first model performance information indicating the performance of a first NLP model from the one or more NLP models that generated the first candidate suggestion. The first suggestion category is also associated with first user acceptance information indicating a user acceptance rate of previous candidate suggestions of the first suggestion category. In some embodiments, the writing assistant microservice 160 provides a suggestion category for each candidate suggestion provided to the content inspection unit 145. In other embodiments, the content inspection unit 145 determines the suggestion category associated with the suggestion candidate based on the microservice of the writing assistant microservice 160 from which the suggestion candidate was received. In other embodiments, the content inspection unit 145 implements an NLP model that is trained to receive candidate suggestions from the writing assistant microservice 160 and output a suggestion category for the suggestion candidate.

[0068] The process includes an operation 620 of determining whether first model performance information and first user acceptance information associated with a first suggestion category meet auto-correction threshold criteria. In some embodiments, auto-correction determination unit 150 provides the suggestion category associated with each candidate suggestion to auto-correction model 135 for analysis to determine whether auto-correction is allowed for the suggestion category. Auto-correction model 135 provides an indication to auto-correction determination unit 150 regarding whether each candidate suggestion can be auto-corrected. The candidate suggestions and the indication from auto-correction model 135 are provided to perform auto-correction unit 155 for processing.

[0069] The process includes, in response to determining that the model performance threshold and user acceptance information associated with the first suggestion category meet the auto-correction threshold criteria, generating auto-corrected text content by updating the text content based on the first candidate suggestion 625. The execution auto-correction unit 155 applies the candidate suggestion to the suggestion category associated with the suggestion category that allows auto-correction.

[0070] The process includes an operation 630 of providing the automatically corrected text content to the first client device. The first client device displays the automatically corrected text content in the first application on the first client device, and replaces the original text content in the first application. The technical benefit of this approach is that it automatically improves the text content that the user is composing or modifying without interrupting the user's workflow.

[0071] FIG6B is a flowchart of an example process 650 for providing suggestions for improving text content. Process 650 can be implemented by the writing assistant service 190 shown in the previous example to generate the auto-correction model 135. In some embodiments, process 650 is used to train the auto-correction model 135 used in process 600 shown in FIG6A.

[0072] The process 650 includes an operation 655 of receiving text content from a first application on a first client device 205 associated with a first user. As discussed in the previous example, the native application 350 on the client device 205 or the application provided by the application service 225 provides a user interface for composing and / or editing text content. The text content is sent to the writing assistant service 190 for analysis by the client-side interface unit 105.

[0073] Process 650 includes an operation 660 of analyzing text content using one or more NLP models to identify one or more candidate suggestions for improving the text content. The proof pipeline 140 of the writing assistant service 190 sends the text content received from the client-side interface unit 105 to the writing assistant microservice 160 for analysis. Each microservice implements an NLP model that is configured to identify errors or potential improvements in the text content and provide candidate suggestions for resolving these issues.

[0074] Process 650 includes an operation 665 of sending one or more candidate suggestions to the first client device. During the process of developing the auto-correction model 135, the proof pipeline 140 sends the candidate suggestions to the first client device for presentation to the user, who can accept, reject, or ignore the suggestions. Auto-correction is not performed until sufficient user feedback is received to determine the user acceptance rate for the suggestion category.

[0075] The process 650 includes an operation 670 of receiving feedback from a first client device including user acceptance information associated with one or more candidate suggestions, and an operation 675 of aggregating the user acceptance information included in the feedback with historical user acceptance information to obtain aggregated user acceptance information. As discussed in the previous example, the feedback unit 115 receives feedback from the client-side interface unit 105, and the data aggregation unit 120 aggregates the feedback data. The feedback data is aggregated with historical user acceptance data previously received from users who accepted, rejected, or ignored the candidate suggestions provided by the writing assistant service 190.

[0076] Process 650 includes obtaining model performance information associated with one or more NLP models at operation 680. As discussed in the previous example, performance evaluation unit 130 obtains performance information of various NLP models used by writing assistant microservice 160 to analyze text content.

[0077] Process 650 includes analyzing the aggregated user acceptance information and model performance information to determine whether the aggregated user acceptance information and model performance information meet the auto-correction threshold criteria 685. The data analysis unit 125 analyzes the aggregated user acceptance information and model performance information to determine whether the auto-correction of the text content is appropriate for the various suggestion categories for which feedback has been received.

[0078] Process 650 includes an operation 690 of training an auto-correction model based on a determination of whether the aggregated user acceptance information and model performance information meet the auto-correction threshold criteria. As discussed in the previous example, data analysis unit 125 updates auto-correction model 135. Auto-correction model 135 is trained to receive a suggestion category as input, and auto-correction model 135 outputs an indication of whether a candidate suggestion for the suggestion category is allowable for auto-correction.

[0079] To illustrate the present disclosure and its benefits, detailed examples of the systems, devices and techniques described in conjunction with Figures 1 to 6B are presented herein. Such usage examples should not be construed as limitations on the logical process embodiments of the present disclosure, nor should variations from the user interface methods described herein be considered outside the scope of the present disclosure. It should be understood that references to displaying or presenting items (such as, but not limited to, presenting an image on a display device, presenting audio via one or more speakers, and / or a vibration device) include issuing instructions, commands and / or signals that cause or reasonably are expected to cause a device or system to display or present items. In some embodiments, the various features described in Figures 1 to 6B are implemented in corresponding modules, which may also be referred to as and / or include logic, components, units and / or mechanisms. Modules may constitute software modules (e.g., code embodied on a machine-readable medium) or hardware modules.

[0080] In some examples, the hardware module can be implemented mechanically, electronically, or using any suitable combination thereof. For example, a hardware module may include dedicated circuits or logic configured to perform certain operations. For example, a hardware module may include a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). A hardware module may also include programmable logic or circuits temporarily configured by software to perform certain operations, and may include a portion of machine-readable media data and / or instructions for such configuration. For example, a hardware module may include software contained within a programmable processor configured to execute a set of software instructions. It should be understood that the decision to implement a hardware module mechanically, in a dedicated and permanently configured circuit, or in a temporarily configured circuit (e.g., configured by software) may be driven by cost, time, support, and engineering considerations.

[0081] Therefore, the phrase "hardware module" should be understood to encompass a tangible entity that can perform certain operations and can be configured or arranged in a physical manner to be an entity that can be physically constructed, permanently configured (e.g., hardwired) and / or temporarily configured (e.g., programmed) to operate in some manner or perform certain operations described herein. As used herein, a "hardware-implemented module" refers to a hardware module. Considering examples in which hardware modules are temporarily configured (e.g., programmed), each hardware module does not need to be configured or instantiated at any one moment. For example, in the case where a hardware module includes a programmable processor that is configured by software to become a special-purpose processor, the programmable processor can be configured as different special-purpose processors (e.g., including different hardware modules) at different times. Software can configure one or more processors accordingly, for example, to constitute a specific hardware module at one moment and to constitute different hardware modules at different moments. Hardware modules implemented using one or more processors can be referred to as "processor-implemented" or "computer-implemented."

[0082] A hardware module can provide information to other hardware modules and receive information from other hardware modules. Therefore, the described hardware modules can be considered to be communicatively coupled. In the case where multiple hardware modules are present at the same time, communication can be achieved by signal transmission (e.g., through appropriate circuits and buses) between or among two or more hardware modules. In an embodiment where multiple hardware modules are configured or instantiated at different times, communication between such hardware modules can be achieved, for example, by storing and retrieving information in a memory device that multiple hardware modules can access. For example, a hardware module can perform an operation and store the output in a memory device, and another hardware module can then access the memory device to retrieve and process the stored output.

[0083] In some examples, at least some of the operations of the method can be performed by one or more processors or processor-implemented modules. In addition, one or more processors can also operate to support the execution of related operations in a "cloud computing" environment or as "software as a service" (SaaS). For example, at least some of the operations can be performed by multiple computers (as an example of a machine including a processor) and / or between multiple computers, where these operations are accessible via a network (e.g., the Internet) and / or via one or more software interfaces (e.g., application program interfaces (APIs)). The execution of certain operations can be distributed among the processors, not only residing within a single machine, but also deployed across several machines. The processor or processor-implemented module can be in a single geographic location (e.g., in a home or office environment or a server farm), or can be distributed across multiple geographic locations.

[0084] Figure 7is a block diagram 700 illustrating an example software architecture 702, portions of which may be used in conjunction with the various hardware architectures described herein, which may implement any of the features described above. Figure 7 8, including a processor 810, a memory 830, and input / output (I / O) components 850. A representative hardware layer 704 is illustrated and may represent, for example, the machine 800 of FIG. 8. The representative hardware layer 704 includes a processing unit 706 and associated executable instructions 708. The executable instructions 708 represent the executable instructions of the software architecture 702, including implementations of the methods, modules, etc. described herein. The hardware layer 704 also includes a memory / storage device 710, which also includes the executable instructions 708 and accompanying data. The hardware layer 704 may also include other hardware modules 712. The instructions 708 maintained by the processing unit 706 may be part of the instructions 708 maintained by the memory / storage device 710.

[0085] The example software architecture 702 can be conceptualized as layers, each providing various functionalities. For example, the software architecture 702 may include layers and components such as an operating system (OS) 714, libraries 716, frameworks 718, applications 720, and a presentation layer 744. In operation, applications 720 and / or other components within a layer may invoke API calls 724 to other layers and receive corresponding results 726. The layers shown are representative in nature, and other software architectures may include additional or different layers. For example, some mobile or dedicated operating systems may not provide framework / middleware 718.

[0086] OS 714 can manage hardware resources and provide public services. OS 714 may include, for example, a kernel 728, services 730, and drivers 732. Kernel 728 may serve as an abstraction layer between hardware layer 704 and other software layers. For example, kernel 728 may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. Services 730 may provide other public services for other software layers. Driver 732 may be responsible for controlling or interfacing with the underlying hardware layer 704. For example, depending on the hardware and / or software configuration, driver 732 may include a display driver, a camera driver, a memory / storage device driver, a peripheral device driver (e.g., via a universal serial bus (USB)), a network and / or wireless communication driver, an audio driver, etc.

[0087] Libraries 716 may provide common infrastructure that can be used by applications 720 and / or other components and / or layers. Libraries 716 generally provide functionality used by other software modules to perform tasks, rather than interacting directly with OS 714. Libraries 716 may include system libraries 734 (e.g., C standard libraries) that may provide functionality such as memory allocation, string manipulation, and file operations. In addition, libraries 716 may include API libraries 736, such as media libraries (e.g., supporting the presentation and manipulation of image, sound, and / or video data formats), graphics libraries (e.g., OpenGL libraries for rendering 2D and 3D graphics on a display), database libraries (e.g., SQLite or other relational database functions), and web libraries (e.g., WebKit, which may provide web browsing functionality). Libraries 716 may also include a variety of other libraries 738 to provide numerous functions for applications 720 and other software modules.

[0088] Framework 718 (sometimes also referred to as middleware) provides a higher-level common infrastructure that can be used by applications 720 and / or other software modules. For example, framework 718 can provide various graphical user interface (GUI) functions, advanced resource management, or advanced location services. Framework 718 can provide a wide range of other APIs for applications 720 and / or other software modules.

[0089] Applications 720 include built-in applications 740 and / or third-party applications 742. Examples of built-in applications 740 may include, but are not limited to, contact applications, browser applications, location applications, media applications, messaging applications, and / or game applications. Third-party applications 742 may include any application developed by an entity other than the vendor of a particular platform. Applications 720 may use functionality available via OS 714, libraries 716, frameworks 718, and presentation layer 744 to create a user interface for interacting with the user.

[0090] Some software architectures use virtual machines, such as virtual machine 748. Virtual machine 748 provides an execution environment in which applications / modules can execute as if they were executing on a hardware machine (e.g., machine 800 of FIG. 8 ). Virtual machine 748 can be hosted by a host OS (e.g., OS 714) or a hypervisor and can have a virtual machine monitor 746 that manages the operation of virtual machine 748 and interoperation with the host operating system. Software architecture that may be different from software architecture 702 outside the virtual machine is executed within virtual machine 748, such as OS 750, libraries 752, frameworks 754, applications 756, and / or presentation layer 758.

[0091] 8 is a block diagram illustrating components of an example machine 800 configured to read instructions from a machine-readable medium (e.g., a machine-readable storage medium) and perform any of the features described herein. The example machine 800 is in the form of a computer system in which instructions 816 (e.g., in the form of software components) can be executed to cause the machine 800 to perform any of the features described herein. In this way, the instructions 816 can be used to implement the modules or components described herein. The instructions 816 cause the unprogrammed and / or unconfigured machine 800 to operate as a specific machine configured to perform the described features. The machine 800 can be configured to operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 800 can operate in the capacity of a server machine or a client machine in a server-client network environment, or as a node in a peer-to-peer or distributed network environment. The machine 800 can be embodied as, for example, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a gaming and / or entertainment system, a smartphone, a mobile device, a wearable device (e.g., a smartwatch), and an Internet of Things (IoT) device. Further, while only a single machine 800 is illustrated, the term "machine" includes any collection of machines that individually or jointly execute instructions 816.

[0092] The machine 800 may include a processor 810, a memory 830, and an I / O component 850, which may be communicatively coupled via, for example, a bus 802. The bus 802 may include multiple buses that couple the various elements of the machine 800 via various bus technologies and protocols. In an example, the processor 810 (including, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, or a suitable combination thereof) may include one or more processors 812a to 812n that may execute instructions 816 and process data. In some examples, one or more processors 810 may execute instructions provided or recognized by one or more other processors 810. The term "processor" includes multi-core processors, which include cores that can execute instructions simultaneously. Although FIG. 8 illustrates multiple processors, the machine 800 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors each having a single core, multiple processors each having multiple cores, or any combination thereof. In some examples, the machine 800 may include multiple processors distributed across multiple machines.

[0093] The memory / storage 830 may include a main memory 832, a static memory 834 or other memory, and a storage unit 836, both of which may be accessed by the processor 810, for example, via the bus 802. The storage unit 836 and the memories 832, 834 store instructions 816 that embody any one or more of the functions described herein. The memory / storage 830 may also store temporary, intermediate, and / or long-term data for the processor 810. The instructions 816 may also reside, completely or partially, within the memories 832, 834, within the storage unit 836, within at least one of the processors 810 (e.g., within a command buffer or cache memory), within at least one of the I / O components 850, or any suitable combination thereof during execution thereof. Thus, the memories 832, 834, the storage unit 836, the memory in the processor 810, and the memory in the I / O components 850 are examples of machine-readable media.

[0094] As used herein, a "machine-readable medium" refers to a device that is capable of temporarily or permanently storing instructions and data that cause the machine 800 to operate in a particular manner, and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical storage media, magnetic storage media and devices, cache memory, network-accessible or cloud storage devices, other types of storage devices, and / or any suitable combination thereof. The term "machine-readable medium" applies to a single medium or a combination of multiple media used to store instructions (e.g., instructions 816) for execution by the machine 800, such that the instructions, when executed by one or more processors 810 of the machine 800, cause the machine 800 to perform one or more features described herein. Thus, a "machine-readable medium" may refer to a single storage device, as well as a "cloud-based" storage system or storage network comprising multiple storage devices or devices. The term "machine-readable medium" does not include the signals themselves.

[0095] The I / O components 850 may include various hardware components suitable for receiving input, providing output, generating output, sending information, exchanging information, capturing measurements, and the like. The specific I / O components 850 included in a particular machine will depend on the type and / or function of the machine. For example, a mobile device such as a mobile phone may include a touch input device, while a headless server or IoT device may not include such a touch input device. The specific examples of I / O components shown in FIG8 are in no way limiting, and other types of components may be included in the machine 800. The grouping of I / O components 850 is merely used to simplify this discussion and is in no way limiting. In various examples, the I / O components 850 may include a user output component 852 and a user input component 854. The user output component 852 may include, for example, a display component (e.g., a liquid crystal display (LCD) or a projector) for displaying information, an acoustic component (e.g., a speaker), a haptic component (e.g., a vibration motor or force feedback device), and / or other signal generators. The user input components 854 may include, for example, an alphanumeric input component (e.g., a keyboard or touch screen), a pointing component (e.g., a mouse device, touchpad, or another pointing instrument), and / or a tactile input component (e.g., a physical button or touch screen that provides the location and / or force of a touch or touch gesture) that is configured to receive various user inputs, such as user commands and / or selections.

[0096] In some examples, the I / O components 850 may include a biometric component 856, a motion component 858, an environmental component 860, and / or a positioning component 862, as well as various other physical sensor components. The biometric component 856 may include, for example, components for detecting body expressions (e.g., facial expressions, vocal expressions, hand or body gestures, or eye tracking), measuring biosignals (e.g., heart rate or brain waves), and identifying people (e.g., via voice, retina, fingerprint, and / or facial recognition). The motion component 858 may include, for example, an acceleration sensor (e.g., an accelerometer) and a rotation sensor (e.g., a gyroscope). The environmental component 860 may include, for example, a light sensor, a temperature sensor, a humidity sensor, a pressure sensor (e.g., a barometer), an acoustic sensor (e.g., a microphone for detecting ambient noise), a proximity sensor (e.g., infrared sensing of nearby objects), and / or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. Positioning component 862 may include, for example, a position sensor (e.g., a Global Positioning System (GPS) receiver), an altitude sensor (e.g., a barometric pressure sensor from which altitude can be derived), and / or an orientation sensor (e.g., a magnetometer).

[0097] The I / O components 850 may include a communication component 864 that implements various technologies operable to couple the machine 800 to a network 870 and / or a device 880 via respective communication couplings 872 and 882. The communication component 864 may include one or more network interface components or other suitable devices to interface with the network 870. The communication component 864 may include, for example, components suitable for providing wired communication, wireless communication, cellular communication, near field communication (NFC), Bluetooth communication, Wi-Fi, and / or communication via other modalities. The device 880 may include other machines or various peripheral devices (e.g., via USB coupling).

[0098] In some examples, the communication component 864 can detect an identifier or include a component suitable for detecting an identifier. For example, the communication component 864 can include a radio frequency identification (RFID) tag reader, an NFC detector, an optical sensor (e.g., a one-dimensional or multi-dimensional barcode or other optical code), and / or an acoustic detector (e.g., a microphone for identifying an audio signal of a tag). In some examples, location information can be determined based on information from the communication component 862, such as, but not limited to, a geographic location via an Internet Protocol (IP) address, a location via Wi-Fi, cellular, NFC, Bluetooth, or other wireless station identification, and / or signal triangulation.

[0099] In the preceding detailed description, numerous specific details have been set forth by way of example in order to provide a thorough understanding of the relevant teachings. However, it will be apparent that the present teachings can be practiced without such details. In other instances, well-known methods, procedures, components, and / or circuits have been described at a relatively high level without detail in order to avoid unnecessarily obscuring aspects of the present teachings.

[0100] Although various embodiments have been described, the description is intended to be exemplary rather than restrictive, and it should be understood that more embodiments and implementations within the scope of the embodiments are possible. Although many possible combinations of features are shown in the drawings and discussed in this detailed description, many other combinations of the disclosed features are possible. Unless otherwise specifically limited, any feature of any embodiment can be used in combination with any other feature or element in any other embodiment or replace any other feature or element in any other embodiment. Therefore, it should be understood that any feature shown and / or discussed in this disclosure can be implemented together in any suitable combination. Therefore, the embodiments are not limited except in accordance with the appended claims and their equivalents. Moreover, various modifications and changes can be made within the scope of the appended claims.

[0101] While the foregoing has described what is believed to be the best mode and / or other examples, it should be understood that various modifications may be made therein, and the subject matter disclosed herein may be implemented in a variety of forms and examples, and the teachings may be applied to many applications, only some of which are described herein. It is intended that the appended claims claim any and all applications, modifications, and variations that fall within the true scope of the present teachings.

[0102] Unless otherwise indicated, all measurements, values, ratings, positions, quantities, sizes and other specifications set forth in this specification (including in the appended claims) are approximate and not exact. They are intended to have a reasonable range consistent with the functions to which they relate and with customary practice in the art to which they pertain.

[0103] The scope of protection is limited solely by the appended claims. When interpreted in light of this specification and subsequent prosecution history, the scope is intended and should be interpreted to be consistent with the ordinary meaning of the language used in the claims and to encompass all structural and functional equivalents. Nevertheless, none of the claims are intended to include subject matter that fails to meet the requirements of Sections 101, 102, or 103 of the Patent Act, nor should they be interpreted in such a manner. Therefore, protection is not claimed for any unexpected encirclement of such subject matter.

[0104] Except as stated immediately above, nothing stated or shown is intended or should be construed as causing exclusive use to the public of any component, step, feature, object, benefit, advantage, or equivalent, whether or not recited in the claims.

[0105] It should be understood that the terms and expressions used herein have ordinary meanings consistent with the corresponding inquiries and research fields corresponding to these terms and expressions, unless a specific meaning is otherwise set forth herein. Relational terms such as first and second, etc. can be used only to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between these entities or actions. The terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a list of elements includes not only those elements, but may also include other elements that are not explicitly listed or that are inherent to such processes, methods, articles, or devices. Without further constraints, an element beginning with "a" or "an" does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0106] The Abstract of the present disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It should be understood that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, it can be seen that various features are grouped together in various examples for the purpose of simplifying the present disclosure. The method of the present disclosure should not be interpreted as reflecting the following intention: the claims require more features than those expressly recited in each claim. On the contrary, as reflected in the appended claims, the inventive subject matter lies in less than all the features of a single disclosed example. Therefore, the appended claims are hereby incorporated into the Detailed Description, with each claim independently serving as a separately claimed subject matter.

Claims

1. A data processing system comprising: processor; as well as A machine-readable medium storing executable instructions that, when executed, cause the processor to perform operations comprising: receiving textual content from a first application on a first client device associated with a first user for analysis by a writing assistant, the writing assistant configured to provide suggestions for improving the textual content; analyzing the textual content using one or more natural language processing (NLP) models to identify one or more candidate suggestions for improving the textual content; determining that a first candidate suggestion of the one or more candidate suggestions is associated with a first suggestion category, the first suggestion category being associated with first model performance information indicating performance of a first NLP model of the one or more NLP models that generated the first candidate suggestion, the first suggestion category being further associated with first user acceptance information indicating a user acceptance rate of previous candidate suggestions of the first suggestion category; determining that the first model performance information and the first user acceptance information associated with the first suggestion category satisfy autocorrection threshold criteria; In response to determining that the model performance threshold and the user acceptance information associated with the first suggestion category satisfy the auto-correction threshold criterion, generating auto-corrected textual content by updating the textual content based on the first candidate suggestion; and The automatically corrected text content is provided to the first client device.

2. The data processing system according to claim 1, wherein: The model performance information includes an accuracy rate that indicates a confidence that an error or potential improvement in the text content has been correctly detected.

3. The data processing system according to claim 2, wherein: The model performance information includes a suggestion accuracy indicating a confidence that the candidate suggestion is a suitable suggestion for resolving the error or the potential improvement to the textual content.

4. The data processing system according to claim 1, wherein: The machine-readable medium includes instructions configured to cause the processor to: determining that a second candidate suggestion of the one or more candidate suggestions is associated with a second suggestion category, the second suggestion category being associated with second model performance information indicating performance of a second NLP model of the one or more NLP models that generated the second candidate suggestion, the second suggestion category being further associated with second user acceptance information indicating a user acceptance rate of previous candidate suggestions of the second suggestion category; as well as It is determined that the second model performance information and the second user acceptance information associated with the first suggestion category do not satisfy the autocorrection threshold criteria.

5. The data processing system according to claim 4, wherein: The machine-readable medium includes instructions configured to cause the processor to: sending the second candidate suggestion to the first client device; and The first client device is caused to display the automatically corrected text content and the second candidate suggestion in the first application on the first client device.

6. The data processing system according to claim 4, wherein: Determining whether the second model performance threshold associated with the second suggestion category and the second user acceptance information satisfy the automatic correction threshold criteria is performed using an automatic correction model, which is configured to receive a suggestion category identifier of a corresponding suggestion category as input and output an indication of whether automatic correction of the text content is allowed for the corresponding suggestion category.

7. The data processing system according to claim 6, wherein: The machine-readable medium includes instructions configured to cause the processor to: receiving feedback from the first client device including user acceptance information associated with the second candidate suggestion; aggregating the user acceptance information included in the feedback with historical user acceptance information to obtain aggregated user acceptance information; obtaining updated second model performance information; analyzing the aggregated user acceptance information and the updated second model performance information to determine whether the aggregated user acceptance information and the updated second model performance information meet the auto-correction threshold criteria; as well as The auto-correction model is updated based on a determination of whether the aggregated user acceptance information and the updated second model performance information satisfy the auto-correction threshold criteria.

8. The data processing system according to claim 7, wherein: The auto-correction model is a rule-based model.

9. The data processing system according to claim 7, wherein: The autocorrection model is a machine learning model.

10. A data processing system comprising: processor; as well as A machine-readable medium storing executable instructions that, when executed, cause the processor to perform operations comprising: receiving textual content from a first application on a first client device associated with a first user for analysis by a writing assistant, the writing assistant configured to provide suggestions for improving the textual content; analyzing the textual content using one or more natural language processing (NLP) models to identify one or more candidate suggestions for improving the textual content; causing the first application of the first client device to display the one or more candidate suggestions; receiving feedback from the first client device including user acceptance information associated with the one or more candidate suggestions; aggregating the user acceptance information included in the feedback with historical user acceptance information to obtain aggregated user acceptance information; obtaining model performance information associated with the one or more NLP models; analyzing the aggregated user acceptance information and the model performance information to determine whether the aggregated user acceptance information and the model performance information meet auto-correction threshold criteria; and An auto-correction model is trained based on a determination of whether the aggregated user acceptance information and the model performance information meet the auto-correction threshold criteria.

11. The data processing system according to claim 10, wherein: The model performance information includes an accuracy rate indicating a confidence that the error or potential improvement of the text content has been correctly detected, and wherein the model performance information includes a suggestion accuracy indicating a confidence that the candidate suggestion is a suitable suggestion for resolving the error or the potential improvement of the text content.

12. The data processing system according to claim 11, wherein: The auto-correction model is selected from a rule-based model and a machine learning model.

13. A method implemented in a data processing system for providing suggestions for improving text content, the method comprising: receiving textual content from a first application on a first client device associated with a first user for analysis by a writing assistant, the writing assistant configured to provide suggestions for improving the textual content; analyzing the textual content using one or more natural language processing (NLP) models to identify one or more candidate suggestions for improving the textual content; determining that a first candidate suggestion of the one or more candidate suggestions is associated with a first suggestion category, the first suggestion category being associated with first model performance information indicating performance of a first NLP model of the one or more NLP models that generated the first candidate suggestion, the first suggestion category being further associated with first user acceptance information indicating a user acceptance rate of previous candidate suggestions of the first suggestion category; determining that the first model performance information and the first user acceptance information associated with the first suggestion category satisfy autocorrection threshold criteria; generating auto-corrected textual content by updating the textual content based on the first candidate suggestion in response to determining that the model performance threshold and the user acceptance information associated with the first suggestion category satisfy the auto-correction threshold criterion; as well as The automatically corrected text content is provided to the first client device.

14. The method according to claim 13, wherein The model performance information includes an accuracy rate that indicates a confidence that an error or potential improvement in the text content has been correctly detected.

15. The method according to claim 14, wherein The model performance information includes a suggestion accuracy indicating a confidence that the candidate suggestion is a suitable suggestion for resolving the error or the potential improvement to the textual content.

16. The method according to claim 13, further comprising: determining that a second candidate suggestion of the one or more candidate suggestions is associated with a second suggestion category, the second suggestion category being associated with second model performance information indicating performance of a second NLP model of the one or more NLP models that generated the second candidate suggestion, the second suggestion category being further associated with second user acceptance information indicating a user acceptance rate of previous candidate suggestions of the second suggestion category; as well as It is determined that the second model performance information and the second user acceptance information associated with the first suggestion category do not meet autocorrection threshold criteria.

17. The method according to claim 16, further comprising: sending the second candidate suggestion to the first client device; as well as The first client device is caused to display the automatically corrected text content and the second candidate suggestion in the first application on the first client device.

18. The method according to claim 16, wherein Determining that the second model performance threshold and the second user acceptance information associated with the second suggestion category satisfy the automatic correction threshold criteria also includes using an automatic correction model, which is configured to receive a suggestion category identifier of a corresponding suggestion category as input and output an indication of whether automatic correction of the text content is allowed for the corresponding suggestion category.

19. The method according to claim 18, further comprising: receiving feedback from the first client device including user acceptance information associated with the second candidate suggestion; aggregating the user acceptance information included in the feedback with historical user acceptance information to obtain aggregated user acceptance information; obtaining updated second model performance information; analyzing the aggregated user acceptance information and the updated second model performance information to determine whether the aggregated user acceptance information and the updated second model performance information meet the auto-correction threshold criteria; as well as The auto-correction model is updated based on a determination of whether the aggregated user acceptance information and the updated second model performance information satisfy the auto-correction threshold criteria.

20. The method according to claim 18, wherein The auto-correction model is selected from a rule-based model and a machine learning model.

Images