Online text recognition model training method, online text recognition method and device

Through the federated learning system, local training of online text recognition models and aggregation weight calculation of feature distribution parameters are solved on the client, and the problems of privacy and security and low recognition accuracy in the prior art are solved, and efficient model training and recognition effects are achieved.

CN118396085BActive Publication Date: 2024-11-19SUN YAT SEN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410833543.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-26
Publication Date
2024-11-19
Estimated Expiration
2044-06-26

AI Technical Summary

Technical Problem

On the premise of ensuring user privacy and security, the existing online text recognition model has low training efficiency and recognition accuracy, resulting in low online text recognition accuracy.

Method used

Using the federated learning system, the server builds the initial recognition model and iteratively trains iteratively, sends model parameters to the client for local training, the client uploads feature distribution parameters to determine the aggregation weight, and the server updates the global model parameters to avoid data leakage and pays attention to high quality Data, accelerate model convergence.

Benefits of technology

On the premise of ensuring user privacy and security, the training efficiency and recognition accuracy of the online text recognition model are improved, and the generalization ability and accuracy of the model are enhanced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118396085B_ABST
    Figure CN118396085B_ABST
Patent Text Reader

Abstract

The online text recognition model training method, online text recognition method and device provided by the present application, the training method includes: the server sends the first model parameters of the initial recognition model after the last generation of training to each client during each iterative training; each client uses local data to iteratively train the received first model parameters, and uploads the obtained second model parameters to the server. In this way, there is no need to upload the local data of each client to the server for training, but only upload the trained second model parameters to the server to avoid data leakage. Then, the quality of the data owned by each client can be quantified through the feature distribution parameters, and then the aggregation weight is determined based on the feature distribution parameters, so that the model pays more attention to high-quality data, accelerates the convergence process of the model, and also improves the accuracy of the model. Thereby improving the training efficiency and recognition accuracy of the model while ensuring the privacy and security of the user.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to an online text recognition model training method, an online text recognition method and a device. Background Art

[0002] Online text recognition, also known as dynamic handwriting recognition or real-time handwriting recognition, focuses on capturing and analyzing the dynamic information of handwriting during the writing process to achieve text recognition. Unlike static text recognition, it not only focuses on the final form of the text, but also on the dynamic characteristics of the stroke order, speed, pressure, etc. during the writing process. Online text recognition can provide a more natural and intuitive human-computer interaction method, such as smart notebooks and electronic whiteboards. These devices can convert the user's handwriting input into electronic text in real time, greatly improving the user experience and work efficiency.

[0003] When we consider centrally training a machine learning model by uploading the writing data of different writers to the server, we must be aware of the potential risk of privacy leakage. First of all, writing data often contains personal writing habits, styles, and may even include sensitive information, such as personal signatures, note contents, etc., which are part of personal privacy. Once this data is uploaded to the server, there is a possibility of unauthorized access or abuse. During the centralized training process, the server needs to collect and store a large number of writing samples in order to build an effective writing recognition model. However, if the server's security measures are not perfect or the data transmission process is not fully encrypted, malicious attackers may intercept this data, thereby infringing on the user's personal privacy.

[0004] At present, models that may involve privacy leakage are generally trained through federated learning to minimize privacy leakage. However, in the process of training online text recognition models through federated learning, due to the large amount of data processed by online text recognition, model convergence requires more time and computing resources. Therefore, under the premise of ensuring the privacy security of users, the training efficiency and recognition accuracy of the online text recognition model are low, which affects the accuracy of online text recognition. Summary of the invention

[0005] The purpose of this application is to solve at least one of the above-mentioned technical defects, especially the technical defects of low training efficiency and recognition accuracy of online text recognition models in the prior art under the premise of ensuring the privacy and security of users.

[0006] In a first aspect, the present application provides an online text recognition model training method, which is applied to a federated learning system, wherein the federated learning system includes a server and multiple clients; the method includes:

[0007] The server constructs an initial recognition model and performs iterative training on the initial recognition model. During each iterative training, the first model parameter of the initial recognition model after the last iterative training is sent to each client.

[0008] Each client uses local data to iteratively train the received first model parameters, and uploads the second model parameters obtained after training to the server;

[0009] The server receives the second model parameters uploaded by each client to determine the feature distribution parameters in the second model parameters of each client, determines the aggregation weight of each client in this iteration according to each feature distribution parameter, and aggregates the second model parameters sent by each client according to each aggregation weight to form the global model parameters of this iteration, and, after updating the first model parameters of the initial recognition model according to the global model parameters, continues to iteratively train the initial recognition model until the preset training conditions are met, and sends the trained initial recognition model as an online text recognition model to each client; wherein the feature distribution parameters are used to measure the data quality of the corresponding local data.

[0010] In one embodiment, the step of determining the aggregation weight of each client in this iteration according to each characteristic distribution parameter includes:

[0011] The server averages and sums each feature distribution parameter to determine a global feature distribution parameter, and determines the aggregation weight corresponding to each client based on each feature distribution parameter and the global feature distribution parameter; wherein the feature distribution parameter is the mean of the learnable parameters of the normalized layer of each layer in the corresponding second model parameter.

[0012] In one embodiment, the server determines the aggregation weight corresponding to each client according to the following expression based on each feature distribution parameter and the global feature distribution parameter:

[0013]

[0014] In the formula, Indicates The aggregate weight corresponding to each client is Indicates The aggregation weight vector corresponding to each client is Indicates The first one in the aggregation weight vector corresponding to the client elements, Indicates The feature dimension of the aggregation weight vector corresponding to the client, where The expression is as follows:

[0015]

[0016] In the formula, , All are The characteristic distribution parameters of the client, , are all global characteristic distribution parameters.

[0017] In one embodiment, the step of aggregating the second model parameters sent by each client according to each aggregation weight to form the global model parameters of this iteration includes:

[0018] The server determines a first target value, and determines a target ratio corresponding to each client according to the first target value; wherein the first target value is the sum of the aggregate weights of the respective clients, and the target ratio is the ratio between the aggregate weight of the corresponding client and the first target value;

[0019] The server determines the product of the target ratio of each client and the second model parameter of the client as the second target value of the client, and sums the second target value of each client to obtain the global model parameter.

[0020] In one embodiment, the step of constructing an initial recognition model includes:

[0021] The server obtains a handwriting point data set, determines a preset target universal model, and iteratively trains the target universal model according to the handwriting point data set, so as to determine the target universal model obtained after the training is completed as an initial recognition model;

[0022] Among them, the target general model includes a dimensionality increase mapping layer, a sequence feature encoding module and a prediction layer; the dimensionality increase mapping layer is used to extract the dimensionality increase features of the handwriting points in the handwriting point data set; the sequence feature encoding module is used to determine the handwriting point encoding feature sequence according to the handwriting point dimensionality increase features; the prediction layer is used to determine the text recognition probability vector according to the handwriting point encoding feature sequence, and update the learnable parameters of the target general model.

[0023] In one embodiment, the step of each client using local data to iteratively train the received first model parameters includes:

[0024] In each iterative training, each client samples from its corresponding local data to obtain a handwriting point sequence and a character recognition label, and determines a recognition loss function based on the handwriting point sequence and character recognition label sampled respectively; each client backpropagates its own recognition loss function to update the current first model parameters until the preset conditions are met, and the latest first model parameters are determined as the second model parameters.

[0025] In a second aspect, the present application provides an online text recognition method, which is applied to any client in a federated learning system, wherein the federated learning system includes a server and multiple clients; the method includes:

[0026] Obtain the handwriting point data to be recognized input by the user, and obtain the online text recognition model sent by the server;

[0027] The handwriting point data is input into the online text recognition model to obtain text data corresponding to the handwriting point data to complete the online text recognition.

[0028] In a third aspect, the present application provides an online text recognition model training device, which is applied to a federated learning system, wherein the federated learning system includes a server and multiple clients; the device includes:

[0029] An iterative training module, used for the server to construct an initial recognition model and iteratively train the initial recognition model. During each iterative training, the first model parameter of the initial recognition model after the last iterative training is sent to each client.

[0030] A local training module, used for each client to iteratively train the received first model parameters using local data, and upload the second model parameters obtained after training to the server;

[0031] A parameter aggregation module is used for the server to receive the second model parameters uploaded by each client to determine the feature distribution parameters in the second model parameters of each client, determine the aggregation weight of each client in this iteration according to each feature distribution parameter, and form the global model parameters of this iteration after aggregating the second model parameters sent by each client according to each aggregation weight, and, after updating the first model parameters of the initial recognition model according to the global model parameters, continue to iteratively train the initial recognition model until the preset training conditions are met, and send the trained initial recognition model to each client as an online text recognition model; wherein the feature distribution parameters are used to measure the data quality of the corresponding local data.

[0032] In a fourth aspect, the present application provides an online text recognition device, which is applied to any client in a federated learning system, wherein the federated learning system includes a server and multiple clients; the device includes:

[0033] The data acquisition module is used to obtain the handwriting point data to be recognized input by the user, and to obtain the online text recognition model sent by the server;

[0034] The text recognition module is used to input the handwriting point data into the online text recognition model to obtain text data corresponding to the handwriting point data to complete online text recognition.

[0035] In a fifth aspect, the present application provides a computer device, comprising: one or more processors, and a memory;

[0036] The memory stores computer-readable instructions, and when the one or more processors execute the computer-readable instructions, the steps of the method described in any one of the above embodiments are performed.

[0037] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0038] The online text recognition model training method, online text recognition method and device provided by the present application, the model training method includes: the server performs generation training on the initial recognition model, and in each iterative training, sends the first model parameter of the initial recognition model after the last generation training to each client; each client uses local data to iteratively train the received first model parameter, and uploads the second model parameter obtained after training to the server. In this way, there is no need to upload the local data of each client to the server for training, but to train locally on each client, and then upload the trained second model parameter to the server to avoid data leakage as much as possible. Then, the server receives the second model parameter uploaded by each client to determine the feature distribution parameter in the second model parameter of each client, determines the aggregation weight of each client at this iteration according to each feature distribution parameter, and aggregates the second model parameter sent by each client according to each aggregation weight to form the global model parameter at this iteration, so as to update the first model parameter of the initial recognition model. The feature distribution parameter is used to measure the data quality of the corresponding local data. In this way, the quality of the data owned by each client can be quantified through the feature distribution parameters, and then the aggregation weight can be determined based on the feature distribution parameters, so that the initial recognition model trained pays more attention to high-quality data, accelerates the convergence process of the model, and also improves the accuracy of the initial recognition model. In this way, the training efficiency and recognition accuracy of the recognition model can be improved while ensuring the privacy and security of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0040] Figure 1 A flowchart of an online text recognition model training method provided in an embodiment of the present application;

[0041] Figure 2 A flowchart of an online text recognition method provided in an embodiment of the present application;

[0042] Figure 3 A schematic diagram of the structure of an online text recognition model training device provided in an embodiment of the present application;

[0043] Figure 4 A schematic diagram of the structure of an online text recognition device provided in an embodiment of the present application;

[0044] Figure 5 An internal structure diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0046] In one embodiment, the present application provides an online text recognition model training method, and the following embodiment is described by applying this method to a federated learning system.

[0047] Specifically, the federated learning system is a system composed of multiple devices participating in federated learning and a server, where the server can be understood as a central server. In the process of federated learning, multiple devices participating in federated learning are allowed to jointly train a global model without concentrating the data set on one server to solve the problem of data security. It is understandable that the device participating in federated learning is referred to as a client in the following embodiments, which can be any computer device with computing and storage capabilities, such as mobile terminals, smart home devices, medical devices, vehicle-mounted devices, etc.

[0048] like Figure 1As shown, the present application provides an online text recognition model training method, which is applied to a federated learning system, wherein the federated learning system includes a server and multiple clients; the method includes:

[0049] Step S101: The server constructs an initial recognition model and performs iterative training on the initial recognition model.

[0050] Among them, the initial recognition model is the initial global model constructed by the server based on the public data set.

[0051] In this step, the server in the federated learning system must first build an initial recognition model, and then iteratively train the initial recognition model. When the training is completed, an online text recognition model can be obtained.

[0052] Step S102: During each iterative training, the server sends the first model parameter of the initial recognition model after the last iterative training to each client.

[0053] The first model parameter refers to the model parameter of the corresponding initial recognition model.

[0054] In this step, in each iteration of the training of the initial recognition model, the server sends the first model parameter of the initial recognition model after the last iteration to each client. It can be understood that when the initial recognition model is iterated for the first time, the initial recognition model after the last iteration refers to the initial recognition model constructed by the server.

[0055] Step S103: Each client uses local data to iteratively train the received first model parameters, and uploads the trained second model parameters to the server.

[0056] The local data consists of a handwriting point sequence and a character recognition label. The local data is represented as follows:

[0057]

[0058] In the formula, Indicates The local data of each client, Indicates Client's A sequence of handwriting points, Indicates Client's Character recognition tags, is a one-hot vector. Client's The character is Class time, ,otherwise, , express No. elements.

[0059] In this step, each client uses its own local data to iteratively train the received first model parameters, and uploads the second model parameters obtained by each client training to the server, so that the server aggregates the second model parameters uploaded by each client. The second model parameters are the model parameters obtained by the corresponding client through iterative training of the first model parameters.

[0060] For example, assuming that the clients participating in federated learning are client A and client B, the server sends the first model parameter C to client A and client B respectively. After client A receives the first model parameter C, it uses the local data of client A to iteratively train the first model parameter C to obtain the second model parameter D corresponding to client A. At the same time, when client B receives the first model parameter C, it uses the local data of client B to iteratively train the first model parameter C to obtain the second model parameter E corresponding to client B. Client A uploads the second model parameter D to the server, and client B uploads the second model parameter E to the server. At this time, the server can receive the second model parameter D sent by client A and the second model parameter E sent by client B in this iteration.

[0061] Step S104: the server receives the second model parameters uploaded by each client to determine the feature distribution parameters in the second model parameters of each client, and determines the aggregation weight of each client in this iteration according to each feature distribution parameter.

[0062] The feature distribution parameter is used to measure the data quality of the corresponding local data. The aggregation weight refers to the weight of the corresponding second model parameter during aggregation. The aggregation weight can be used to measure the contribution of the client corresponding to the second model parameter to the global model in this iteration.

[0063] In this step, when the server receives the second model parameters uploaded by each client, it can determine the feature distribution parameters corresponding to each client based on the second model parameters of each client, and then use the feature distribution parameters corresponding to each client to measure the data quality of the local data of the corresponding client, and then determine the aggregation weight of each client in this iteration.

[0064] Step S105: the server aggregates the second model parameters sent by each client according to each aggregation weight to form the global model parameters of this iteration, and updates the first model parameters of the initial recognition model according to the global model parameters.

[0065] The global model parameters refer to the model parameters of the global model trained in the server.

[0066] In this step, the server can aggregate the second model parameters sent by each client according to the aggregation weight of each client to form the global model parameters of this iteration. In addition, the server will also update the first model parameters of the initial recognition model according to the global model parameters. Therefore, in the next iterative training, the first model parameters of the initial recognition model used refer to the global model parameters of the previous iteration.

[0067] Furthermore, the step of updating the first model parameters of the initial recognition model according to the global model parameters includes: updating the first model parameters of the initial recognition model to the global model parameters of this iteration.

[0068] Step S106: The server determines whether the preset training conditions are met.

[0069] Among them, the preset training condition refers to the condition for the termination of iteration. When the preset training condition is met, the server can determine the initial recognition model finally obtained by training as the online text recognition model and send it to each client, so that the client can apply the online text recognition model in the online text recognition method. When the preset training condition is not met, the initial recognition model continues to be iteratively trained. It can be understood that the preset training condition can be set to the number of iterations reaching a preset threshold, or it can be set to the absolute value of the parameter difference between the initial recognition model obtained in the previous iteration and the initial recognition model obtained in this iteration is less than a preset threshold. This application does not impose specific restrictions on this.

[0070] Step S107: When the server determines that the preset training condition is not met, it continues to iteratively train the initial recognition model.

[0071] Step S108: When the server determines that the preset training conditions are met, the server sends the trained initial recognition model as an online text recognition model to each client.

[0072] The online text recognition model training method, online text recognition method and device provided by the present application, the model training method includes: the server performs generation training on the initial recognition model, and in each iterative training, sends the first model parameter of the initial recognition model after the last generation training to each client; each client uses local data to iteratively train the received first model parameter, and uploads the second model parameter obtained after training to the server. In this way, there is no need to upload the local data of each client to the server for training, but to train locally on each client, and then upload the trained second model parameter to the server to avoid data leakage as much as possible. Then, the server receives the second model parameter uploaded by each client to determine the feature distribution parameter in the second model parameter of each client, determines the aggregation weight of each client at this iteration according to each feature distribution parameter, and aggregates the second model parameter sent by each client according to each aggregation weight to form the global model parameter at this iteration, so as to update the first model parameter of the initial recognition model. The feature distribution parameter is used to measure the data quality of the corresponding local data. In this way, the quality of the data owned by each client can be quantified through the feature distribution parameters, and then the aggregation weight can be determined based on the feature distribution parameters, so that the initial recognition model trained pays more attention to high-quality data, accelerates the convergence process of the model, and also improves the accuracy of the initial recognition model. In this way, the training efficiency and recognition accuracy of the recognition model can be improved while ensuring the privacy and security of users.

[0073] In one embodiment, the step of determining the aggregation weight of each client in this iteration according to each characteristic distribution parameter includes:

[0074] The server averages and sums the feature distribution parameters to determine the global feature distribution parameters, and determines the aggregation weight corresponding to each client based on the feature distribution parameters and the global feature distribution parameters.

[0075] The characteristic distribution parameter is the mean of the learnable parameters of each normalized layer in the corresponding second model parameter. The average sum refers to the arithmetic average sum, that is, summing a group of data and dividing the summed result by the number of data in the group.

[0076] In this embodiment, the server may average the corresponding parameters in each feature distribution parameter to form a global feature distribution parameter, and then calculate the aggregation weight corresponding to each client according to each feature distribution parameter and the determined global feature distribution parameter.

[0077] In an example, the process in which the server averages and sums the corresponding parameters in each feature distribution parameter to form a global feature distribution parameter can be expressed as follows:

[0078]

[0079]

[0080] In the formula, , is the global characteristic distribution parameter, , For the The characteristic distribution parameters of each client. Indicates the number of clients.

[0081] It is understandable that, since the feature distribution parameters can be used to measure the data quality of the corresponding local data, the global feature distribution parameters are calculated based on this, and then the aggregation weight corresponding to each client can be determined according to each feature distribution parameter and the global feature distribution parameter, so as to determine the contribution of each client to the global model parameter aggregation. In this way, the data quality of each client can be considered when aggregating the global model parameters, so that the global model pays more attention to high-quality data. On the one hand, it can speed up the convergence of the model and improve the efficiency of model training. On the other hand, it can also enable the global model to better learn the real and useful features in the data, improve the generalization ability of the model, so as to improve the accuracy of the model, thereby improving the accuracy of online text recognition.

[0082] In one embodiment, the server determines the aggregation weight corresponding to each client according to the following expression based on each feature distribution parameter and the global feature distribution parameter:

[0083]

[0084] In the formula, Indicates The aggregate weight corresponding to each client is Indicates The aggregation weight vector corresponding to each client is Indicates The first one in the aggregation weight vector corresponding to the client elements, Indicates The feature dimension of the aggregation weight vector corresponding to the client, where The expression is as follows:

[0085]

[0086] In the formula, , All are The characteristic distribution parameters of the client, , are all global characteristic distribution parameters.

[0087] Specifically, since the feature dimensions of the aggregation weight vector corresponding to each client are consistent, The value of is the same for different clients.

[0088] It is understandable that the aggregation weight corresponding to each client can be determined based on each feature distribution parameter and the global feature distribution parameter, so as to determine the contribution of each client to the global model parameter aggregation. In this way, the data quality of each client can be considered when aggregating the global model parameters, so that the global model pays more attention to high-quality data. On the one hand, it can accelerate model convergence and improve the efficiency of model training. On the other hand, it can also enable the global model to better learn the real and useful features in the data, improve the generalization ability of the model, so as to improve the accuracy of the model, thereby improving the accuracy of online text recognition.

[0089] In one embodiment, the step of aggregating the second model parameters sent by each client according to each aggregation weight to form the global model parameters of this iteration includes:

[0090] The server determines a first target value, and determines a target ratio corresponding to each client according to the first target value; wherein the first target value is the sum of the aggregate weights of each client, and the target ratio is the ratio between the aggregate weight of the corresponding client and the first target value;

[0091] The server determines the product of the target ratio of each client and the second model parameter of the client as the second target value of the client, and sums the second target value of each client to obtain the global model parameter.

[0092] In an example, the process of aggregating the second model parameters sent by each client according to each aggregation weight to form the global model parameters in this iteration can be expressed by the following expression:

[0093]

[0094] In the formula, represents the global model parameters, Indicates The aggregate weight corresponding to each client is Indicates The second model parameter sent by the client, Indicates The aggregate weight corresponding to each client is Indicates the number of clients. Specifically, refers to the first target value in this embodiment, In this embodiment, The target ratio corresponding to each client is In this embodiment, The second target value corresponding to the client.

[0095] It is understandable that the second model parameters sent by each client are aggregated according to the aggregation weight of each client to form the global model parameters for this iteration. In this way, the contribution of each client, that is, the aggregation weight, will be considered when forming the global model parameters for this iteration, thereby assigning a higher weight to high-quality data and a lower weight to low-quality data. On the one hand, it can speed up model convergence and improve training efficiency. On the other hand, it can improve the recognition accuracy of the model while ensuring the privacy and security of the user, thereby improving the recognition accuracy of online text recognition.

[0096] In one embodiment, the step of constructing an initial recognition model includes:

[0097] The server obtains a handwriting point data set, determines a preset target universal model, and iteratively trains the target universal model according to the handwriting point data set, so as to determine the target universal model obtained when the training is completed as an initial recognition model;

[0098] Among them, the target general model includes a dimensionality-increasing mapping layer, a sequence feature encoding module and a prediction layer; the dimensionality-increasing mapping layer is used to extract the dimensionality-increasing features of the handwriting points in the handwriting point data set; the sequence feature encoding module is used to determine the handwriting point encoding feature sequence according to the dimensionality-increasing features of the handwriting points; the prediction layer is used to determine the text recognition probability vector according to the handwriting point encoding feature sequence, and update the learnable parameters of the target general model.

[0099] Specifically, the handwriting point dataset refers to a dataset composed of historical handwriting point data, and the handwriting point dimension-raising feature refers to the feature representation form that maps the handwriting data (handwriting point dataset) from a low-dimensional space to a high-dimensional space. The handwriting point encoding feature sequence is a feature vector sequence that can be obtained by converting the handwriting point dimension-raising feature. The text recognition probability vector reflects the probability of each possible character or word being the input text.

[0100] In this embodiment, the target general model obtained in the previous iteration is trained and updated in each iteration training. When the iteration is finished, the latest target general model is determined as the initial recognition model.

[0101] In one example, the process of building an initial recognition model can be expressed as follows:

[0102] The data in the handwriting point data set obtained by the server can be expressed as:

[0103]

[0104] In the formula, Represents a data record in the handwriting point dataset. express The number of trajectory points in express A trajectory point in express Middle The horizontal coordinate of the trajectory point, express Middle The vertical coordinate of the trajectory point.

[0105] Input the handwriting point data set into the dimension-raising mapping layer to obtain the handwriting point dimension-raising features, which are expressed as:

[0106]

[0107] In the formula, Indicates the dimensionality-increasing features of handwriting points, is the activation function, and is a learnable parameter. , express Length, represents the hidden layer dimension.

[0108] The handwriting point dimension-raising features are input into the sequence feature encoding module to obtain the handwriting point encoding feature sequence, wherein the sequence feature encoding module is composed of z sequence feature encoding layers stacked, the first The expression of the layer sequence feature encoding layer is:

[0109]

[0110]

[0111] In the formula, Indicates Layer sequence characteristics, , Indicates the number of layers, Indicates Layer sequence features also refer to the dimensionality-increasing features of handwriting points. is a general sequence encoding model. Representation layer normalization layer, Indicates The sequence features after layer normalization, is the handwriting point coding feature sequence output by the sequence feature coding module. , , , express or Length, represents the hidden layer dimension.

[0112] Among them, the layer normalization layer The expression is:

[0113]

[0114] In the formula, express No. elements, express No. elements, express The statistical mean of express The statistical standard deviation of and Indicates The learnable parameters of the layer normalization layer. Among them, , , , , , , represents the hidden layer dimension.

[0115] In addition, the feature distribution parameters can be determined according to the above layer normalization layer, and the expression is as follows:

[0116]

[0117]

[0118] In the formula, and represents the characteristic distribution parameter in the second model parameter, Indicates the number of layers, and Indicates The learnable parameters of the layer normalization layer. Among them, , , , , represents the hidden layer dimension.

[0119] The process of inputting the handwriting point coding feature sequence into the prediction layer to obtain the character recognition probability vector can be expressed as follows:

[0120]

[0121]

[0122] In the formula, is the text recognition probability vector, is the adaptive weight, , are all learnable parameters. , , , , , is the number of predefined text recognition categories, represents the hidden layer dimension, Indicates length.

[0123] Finally, the learnable parameters of the target general model are initialized, and then the update of the target general model in one iterative training is completed. Then, the target general model is iteratively trained until the preset training termination condition of the target general model is reached, and the latest target general model is determined as the initial recognition model.

[0124] It is understandable that the server can obtain a preliminary recognition model (initial recognition model) by training the target general model through the handwriting point dataset, and then complete the construction of the initial recognition model. This can help the initial recognition model converge at a faster speed in subsequent training, so that the initial recognition model can achieve better performance in the same number of iterations.

[0125] In one embodiment, the step of each client using local data to iteratively train the received first model parameters includes:

[0126] In each iterative training, each client samples from its corresponding local data to obtain a handwriting point sequence and a character recognition label, and determines a recognition loss function based on the handwriting point sequence and character recognition label sampled respectively; each client backpropagates its own recognition loss function to update the current first model parameters until the preset conditions are met, and the latest first model parameters are determined as the second model parameters.

[0127] The preset condition refers to the termination condition for each client to iteratively train the first model parameter. The preset condition can be set to the number of iterations reaching a preset threshold, or it can be set to the absolute value of the difference between the first model parameter obtained in the previous iteration and the first model parameter obtained in this iteration is less than the preset threshold. This application does not impose specific restrictions on this. The handwriting point sequence includes a sequence of multiple handwriting points, and the character recognition label is used to construct a recognition loss function.

[0128] Specifically, no. Local data of each client It consists of paired handwriting point sequences and character recognition labels, and is expressed as:

[0129]

[0130] in, Indicates The local data of each client, Indicates Client's A sequence of handwriting points, Indicates Client's character recognition label, which is a one-hot vector. Client's The character is Class time, ,otherwise, , express No. elements.

[0131] For example, in each training iteration, Paired handwriting point sequences and character recognition labels are sampled from , and the recognition loss function is calculated. The expression is:

[0132]

[0133]

[0134] in, represents the recognition loss function, Indicates the use of The first model parameter received by the client For Client's Sequence of handwriting points The predicted output text recognition probability vector, is the activation function, express No. elements, Indicates Client's character recognition label, which is a one-hot vector. Client's The character is Class time, ,otherwise, , express No. elements, , is the number of predefined text recognition categories.

[0135] Loss function for character recognition Back propagation is used to train the The first model parameter of the client When the training is completed, the latest first model parameters of each client are obtained, and the first model parameters are determined as the second model parameters.

[0136] It is understandable that when each client uses local data to update the parameters of the first model, it can fine-tune the parameters of the first model in a personalized manner according to the characteristics and distribution of its own data. This enables each client to obtain a model that is more suitable for its own data, thereby improving the performance and applicability of the final global model. Moreover, since each client only uses local data to update parameters instead of sending data to the server for training, it will not be exposed to other participants or the server. Therefore, the privacy of users can be better protected.

[0137] In one embodiment, when the model parameters are transmitted between the server and the client, the model parameters may be encrypted for transmission, thereby further improving the privacy security of the user and preventing data leakage.

[0138] like Figure 2 As shown, the present application provides an online text recognition method, which is applied to any client in a federated learning system, and the federated learning system includes a server and multiple clients; the method includes:

[0139] Step S201: Obtain the handwriting point data to be recognized input by the user, and obtain the online text recognition model sent by the server.

[0140] The online text recognition model is generated by the server using the online text recognition model training method described in any one of the above embodiments.

[0141] In this step, when the user enables online text recognition, the client will sample the handwriting points of the user's handwritten text to obtain handwriting point data, and recognize the handwriting point data through the online text recognition model to obtain text data. It can be understood that the effect of online text recognition can be achieved by periodically sampling the handwritten text and setting a shorter period.

[0142] Step S202: input the handwriting point data into the online text recognition model to obtain text data corresponding to the handwriting point data to complete the online text recognition.

[0143] The text data refers to the recognition data of the user's handwritten text.

[0144] It is understandable that in the training process of the online text recognition model, the quality of the local data owned by each client is quantified through the feature distribution parameters, and then the aggregation weight is determined based on the feature distribution parameters, so that the trained initial recognition model pays more attention to high-quality data, accelerates the convergence process of the model, and also improves the accuracy of the initial recognition model. Then, the training efficiency and recognition accuracy of the recognition model are improved while protecting the privacy and security of the user. Therefore, when such an online text recognition model is applied to the online text recognition method, the online text recognition method can also have higher security and recognition accuracy.

[0145] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0146] The online text recognition model training device provided in an embodiment of the present application is described below. The online text recognition model training device described below and the online text recognition model training method described above can be referenced to each other.

[0147] like Figure 3 As shown, the present application provides an online text recognition model training device 300, which is applied to a federated learning system. The federated learning system includes a server and multiple clients; the device includes:

[0148] Iterative training module 301 is used for the server to construct an initial recognition model and iteratively train the initial recognition model. During each iterative training, the first model parameter of the initial recognition model after the previous iterative training is sent to each client.

[0149] A local training module 302 is used for each client to iteratively train the received first model parameters using local data, and upload the second model parameters obtained after training to the server;

[0150] The parameter aggregation module 303 is used for receiving the second model parameters uploaded by each client on the server side to determine the feature distribution parameters in the second model parameters of each client, determine the aggregation weight of each client in this iteration according to each feature distribution parameter, and form the global model parameters of this iteration after aggregating the second model parameters sent by each client according to each aggregation weight, and, after updating the first model parameters of the initial recognition model according to the global model parameters, continue to iteratively train the initial recognition model until the preset training conditions are met, and send the trained initial recognition model as an online text recognition model to each client; wherein the feature distribution parameters are used to measure the data quality of the corresponding local data.

[0151] In one embodiment, the parameter aggregation module includes:

[0152] The aggregation weight determination submodule is used for the server to average the sum of each feature distribution parameter to determine the global feature distribution parameter, and determine the aggregation weight corresponding to each client according to each feature distribution parameter and the global feature distribution parameter; wherein the feature distribution parameter is the mean of the learnable parameters of the normalized layer of each layer in the corresponding second model parameter.

[0153] In one embodiment, the parameter aggregation module includes:

[0154] The target value determination submodule is used for the server to determine the first target value, and determine the target ratio corresponding to each client according to the first target value; wherein the first target value is the sum of the aggregate weights of each client, and the target ratio is the ratio between the aggregate weight of the corresponding client and the first target value;

[0155] The parameter aggregation submodule is used for the server to determine the product of the target ratio of each client and the second model parameter of the client as the second target value of the client, and sum the second target value of each client to obtain the global model parameter.

[0156] In one embodiment, the iterative training module includes:

[0157] The model building submodule is used for the server to obtain the handwriting point data set and determine the preset target universal model, and iteratively train the target universal model according to the handwriting point data set, so as to determine the target universal model obtained after the training is completed as the initial recognition model;

[0158] Among them, the target general model includes a dimensionality-increasing mapping layer, a sequence feature encoding module and a prediction layer; the dimensionality-increasing mapping layer is used to extract the dimensionality-increasing features of the handwriting points in the handwriting point data set; the sequence feature encoding module is used to determine the handwriting point encoding feature sequence according to the dimensionality-increasing features of the handwriting points; the prediction layer is used to determine the text recognition probability vector according to the handwriting point encoding feature sequence, and update the learnable parameters of the target general model.

[0159] In one embodiment, the local training module includes:

[0160] The local training submodule is used for each client to sample the handwriting point sequence and character recognition label from its corresponding local data in each iterative training, and determine the recognition loss function according to the handwriting point sequence and character recognition label sampled respectively; each client back-propagates its own recognition loss function to update the current first model parameters until the preset conditions are met, and the latest first model parameters are determined as the second model parameters.

[0161] The online text recognition device provided in the embodiment of the present application is described below. The online text recognition device described below and the online text recognition method described above can be referenced to each other.

[0162] like Figure 4 As shown, the present application provides an online text recognition device 400, which is applied to any client in a federated learning system. The federated learning system includes a server and multiple clients; the device includes:

[0163] The data acquisition module 401 is used to acquire the handwriting point data to be recognized input by the user, and to acquire the online text recognition model sent by the server;

[0164] The text recognition module 402 is used to input the handwriting point data into the online text recognition model to obtain text data corresponding to the handwriting point data to complete the online text recognition.

[0165] The division of each module in the above device is only for illustration. In other embodiments, the device can be divided into different modules as needed to complete all or part of the functions of the above device. Each module in the above device can be implemented in whole or in part by software, hardware and a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0166] In one embodiment, the present application further provides a computer device having computer-readable instructions stored therein. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the method described in any one of the above embodiments.

[0167] Indicatively, if Figure 5 As shown, Figure 5 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. The computer device 500 may be provided as a server. Figure 5 , the computer device 500 includes a processing component 502, which further includes one or more processors, and a memory resource represented by a memory 501, for storing instructions that can be executed by the processing component 502, such as an application. The application stored in the memory 501 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 502 is configured to execute instructions to perform the method of any of the above embodiments.

[0168] The computer device 500 may further include a power supply component 503 configured to perform power management of the computer device 500, a wired or wireless network interface 504 configured to connect the computer device 500 to a network, and an input / output (I / O) interface 505. The computer device 500 may operate based on an operating system stored in the memory 501, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, Free BSD TM, or the like.

[0169] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0170] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish an entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only include those elements, but also include other elements that are not clearly listed, or also include elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements. Herein, the singular "one", "one" and "described / the" may also include plural forms, unless the context clearly indicates another way. It should also be understood that the terms "include / comprise" or "have" etc. specify the existence of stated features, wholes, steps, operations, components, parts or combinations thereof, but do not exclude the possibility of the existence or addition of one or more other features, wholes, steps, operations, components, parts or combinations thereof. At the same time, the term "and / or" used in this specification includes any and all combinations of the relevant listed items.

[0171] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0172] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An online text recognition model training method, characterized in that: Applied to a federated learning system, the federated learning system includes a server and multiple clients; the method includes: The server constructs an initial recognition model and performs iterative training on the initial recognition model. During each iterative training, the first model parameter of the initial recognition model after the last iterative training is sent to each client. Each client uses local data to iteratively train the received first model parameters, and uploads the second model parameters obtained after training to the server; The server receives the second model parameters uploaded by each client to determine the feature distribution parameters in the second model parameters of each client, determines the aggregation weight of each client in this iteration according to each feature distribution parameter, and forms the global model parameters of this iteration after aggregating the second model parameters sent by each client according to each aggregation weight, and, after updating the first model parameters of the initial recognition model according to the global model parameters, continues to iteratively train the initial recognition model until the preset training conditions are met, and sends the trained initial recognition model as an online text recognition model to each client; wherein the feature distribution parameters are used to measure the data quality of the corresponding local data; The step of determining the aggregation weight of each client in this iteration according to each characteristic distribution parameter includes: The server averages and sums each feature distribution parameter to determine a global feature distribution parameter, and determines the aggregation weight corresponding to each client according to each feature distribution parameter and the global feature distribution parameter; wherein the feature distribution parameter is the mean of the learnable parameters of each normalized layer in the corresponding second model parameter; The process in which the server averages and sums the corresponding parameters in each feature distribution parameter to form the global feature distribution parameter is expressed as formula (1) and formula (2): (1) (2) In the formula, , is the global characteristic distribution parameter, , For the The characteristic distribution parameters of each client; Indicates the number of clients; According to formula (3), the aggregation weight corresponding to each client is determined: (3) In the formula, Indicates The aggregate weight corresponding to each client is Indicates The aggregation weight vector corresponding to each client is Indicates The first one in the aggregation weight vector corresponding to the client elements, Indicates The feature dimension of the aggregation weight vector corresponding to the client, where The expression of is formula (4): (4) The process of aggregating the second model parameters sent by each client according to each aggregation weight to form the global model parameters of this iteration is expressed as formula (5): (5) In the formula, represents the global model parameters, Indicates The second model parameter sent by the client, Indicates The aggregation weight corresponding to each client; The step of constructing the initial recognition model comprises: The server obtains a handwriting point data set, determines a preset target universal model, and iteratively trains the target universal model according to the handwriting point data set, so as to determine the target universal model obtained after the training is completed as an initial recognition model; The target general model includes a dimension-raising mapping layer, a sequence feature encoding module and a prediction layer; the dimension-raising mapping layer is used to extract the dimension-raising features of the handwriting points in the handwriting point data set; the sequence feature encoding module is used to determine the handwriting point encoding feature sequence according to the dimension-raising features of the handwriting points; the prediction layer is used to determine the character recognition probability vector according to the handwriting point encoding feature sequence, and update the learnable parameters of the target general model; The handwriting point dataset is input into the dimension-raising mapping layer to obtain the handwriting point dimension-raising features, which are expressed as formula (6): (6) In the formula, Represents a data record in the handwriting point dataset. Indicates the dimensionality-increasing features of handwriting points, is the activation function, and is a learnable parameter; The handwriting point dimension-raising features are input into the sequence feature encoding module to obtain the handwriting point encoding feature sequence, wherein the sequence feature encoding module is composed of z sequence feature encoding layers stacked, the first The expressions of the layer sequence feature coding layer are as follows: (7) (8) In the formula, Indicates Layer sequence characteristics, , Indicates the number of layers, when Indicates Layer sequence features, that is, the dimension-increasing features of handwriting points, is a general sequence encoding model. Representation layer normalization layer, Indicates Sequence features after layer normalization; Determine the characteristic distribution parameters in the second model parameters of each client, expressed as formula (9) and formula (10): (9) (10) In the formula, and represents the characteristic distribution parameter in the second model parameter, and Indicates The learnable parameters of the layer normalization layer; The handwriting point coding feature sequence is input into the prediction layer to obtain the character recognition probability vector, which can be expressed as formula (11) and formula (12): (11) (12) In the formula, is the text recognition probability vector, is the adaptive weight, represents the transpose of the adaptive weights, , are all learnable parameters, It is the handwriting point coding feature sequence output by the sequence feature coding module.

2. The online text recognition model training method according to claim 1, characterized in that: The step of aggregating the second model parameters sent by each client according to each aggregation weight to form the global model parameters of this iteration includes: The server determines a first target value, and determines a target ratio corresponding to each client according to the first target value; wherein the first target value is the sum of the aggregate weights of the respective clients, and the target ratio is the ratio between the aggregate weight of the corresponding client and the first target value; The server determines the product of the target ratio of each client and the second model parameter of the client as the second target value of the client, and sums the second target value of each client to obtain the global model parameter.

3. The online text recognition model training method according to claim 1, characterized in that: The step of each client using local data to iteratively train the received first model parameters includes: In each iterative training, each client samples from its corresponding local data to obtain a handwriting point sequence and a character recognition label, and determines a recognition loss function based on the handwriting point sequence and character recognition label sampled respectively; each client backpropagates its own recognition loss function to update the current first model parameters until the preset conditions are met, and the latest first model parameters are determined as the second model parameters.

4. An online text recognition method, characterized in that: Applied to any client in a federated learning system, the federated learning system includes a server and multiple clients; the method includes: Acquire the handwriting point data to be recognized input by the user, and acquire the online text recognition model sent by the server; the online text recognition model is generated by the online text recognition model training method according to any one of claims 1 to 3; The handwriting point data is input into the online text recognition model to obtain text data corresponding to the handwriting point data to complete the online text recognition.

5. An online text recognition model training device, characterized in that: Applied to a federated learning system, the federated learning system comprising a server and multiple clients; The device comprises: An iterative training module, used for the server to construct an initial recognition model and iteratively train the initial recognition model. During each iterative training, the first model parameter of the initial recognition model after the last iterative training is sent to each client. A local training module, used for each client to iteratively train the received first model parameters using local data, and upload the second model parameters obtained after training to the server; A parameter aggregation module, used for the server to receive the second model parameters uploaded by each client to determine the feature distribution parameters in the second model parameters of each client, determine the aggregation weight of each client in this iteration according to each feature distribution parameter, and form the global model parameters of this iteration after aggregating the second model parameters sent by each client according to each aggregation weight, and, after updating the first model parameters of the initial recognition model according to the global model parameters, continue to iteratively train the initial recognition model until the preset training conditions are met, and send the trained initial recognition model as an online text recognition model to each client; wherein the feature distribution parameters are used to measure the data quality of the corresponding local data; The parameter aggregation module includes: The aggregation weight determination submodule is used for the server to average and sum each feature distribution parameter to determine the global feature distribution parameter, and determine the aggregation weight corresponding to each client according to each feature distribution parameter and the global feature distribution parameter; wherein the feature distribution parameter is the mean of the learnable parameters of each layer normalization layer in the corresponding second model parameter; The aggregation weight determination submodule includes: the process in which the server averages and sums the corresponding parameters in each feature distribution parameter to form a global feature distribution parameter is expressed as formula (1) and formula (2): (1) (2) In the formula, , is the global characteristic distribution parameter, , For the The characteristic distribution parameters of each client; Indicates the number of clients; The aggregation weight determination submodule includes: determining the aggregation weight corresponding to each client according to formula (3): (3) In the formula, Indicates The aggregate weight corresponding to each client is Indicates The aggregation weight vector corresponding to each client is Indicates The first one in the aggregation weight vector corresponding to the client elements, Indicates The feature dimension of the aggregation weight vector corresponding to the client, where The expression of is formula (4): (4) The parameter aggregation module includes: The process of aggregating the second model parameters sent by each client according to each aggregation weight to form the global model parameters of this iteration is expressed as formula (5): (5) In the formula, represents the global model parameters, Indicates The second model parameter sent by the client, Indicates The aggregation weight corresponding to each client; The iterative training module includes: The server obtains a handwriting point data set, determines a preset target universal model, and iteratively trains the target universal model according to the handwriting point data set, so as to determine the target universal model obtained after the training is completed as an initial recognition model; The target general model includes a dimension-raising mapping layer, a sequence feature encoding module and a prediction layer; the dimension-raising mapping layer is used to extract the dimension-raising features of the handwriting points in the handwriting point data set; the sequence feature encoding module is used to determine the handwriting point encoding feature sequence according to the dimension-raising features of the handwriting points; the prediction layer is used to determine the character recognition probability vector according to the handwriting point encoding feature sequence, and update the learnable parameters of the target general model; The handwriting point dataset is input into the dimension-raising mapping layer to obtain the handwriting point dimension-raising features, which are expressed as formula (6): (6) In the formula, Represents a data record in the handwriting point dataset. Indicates the dimensionality-increasing features of handwriting points, is the activation function, and is a learnable parameter; The handwriting point dimension-raising features are input into the sequence feature encoding module to obtain the handwriting point encoding feature sequence, wherein the sequence feature encoding module is composed of z sequence feature encoding layers stacked, the first The expressions of the layer sequence feature coding layer are as follows: (7) (8) In the formula, Indicates Layer sequence characteristics, , Indicates the number of layers, when Indicates Layer sequence features, that is, the dimension-increasing features of handwriting points, is a general sequence encoding model. Representation layer normalization layer, Indicates Sequence features after layer normalization; Determine the characteristic distribution parameters in the second model parameters of each client, expressed as formula (9) and formula (10): (9) (10) In the formula, and represents the characteristic distribution parameter in the second model parameter, and Indicates The learnable parameters of the layer normalization layer; The handwriting point coding feature sequence is input into the prediction layer to obtain the character recognition probability vector, which can be expressed as formula (11) and formula (12): (11) (12) In the formula, is the text recognition probability vector, is the adaptive weight, represents the transpose of the adaptive weights, , are all learnable parameters, It is the handwriting point coding feature sequence output by the sequence feature coding module.

6. An online text recognition device, characterized in that: Applied to any client in a federated learning system, the federated learning system includes a server and multiple clients; the device includes: A data acquisition module, used to acquire the handwriting point data to be recognized input by the user, and to acquire the online text recognition model sent by the server; the online text recognition model is generated by the online text recognition model training method according to any one of claims 1 to 3; The text recognition module is used to input the handwriting point data into the online text recognition model to obtain text data corresponding to the handwriting point data to complete online text recognition.

7. A computer device, characterized in that: include: one or more processors, and memory; The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the method according to any one of claims 1 to 4 are performed.