Image recognition model training method, system and cluster

Abstract

An image recognition model training method can be applied to the field of cloud computing. The method comprises the following steps: inputting a first image data set locally stored by a user into a coding module in a local first training device of the user to train the coding module to obtain a trained coding module; a second training device at the cloud end obtains the trained coding module from the first training device; and inputting the labeled second image data set stored in the cloud into an image recognition model comprising the recognition module and the trained coding module, so as to train the recognition module and obtain a trained recognition module. According to the method, the image recognition model can be trained by using the image data of the user under the condition of avoiding privacy leakage of the user.

Description

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on June 8, 2023, with application number 202310680382.6 and application name “A Method for Optimizing Text Recognition”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of cloud computing technology, and in particular to an image recognition model training method, system and cluster. Background Art

[0003] The application and development of deep learning and other artificial intelligence (AI) technologies in the field of image recognition has improved the efficiency of image recognition and reduced manpower. Among them, a common application of AI technology in the field of image recognition is to use AI technology to train image recognition models to achieve automatic recognition of target objects in images.

[0004] The training of image recognition models requires image data as a training set. In some scenarios, the owner of the image data and the training party of the image recognition model are not the same party, and the image data may contain sensitive information of the owner. Giving the image data to the training party in order to train the image recognition model may lead to the leakage of user privacy information. Summary of the invention

[0005] The embodiments of the present application propose an image recognition model training method, system, device and cluster, which can use the user's image data to train the image recognition model while avoiding the leakage of the user's privacy.

[0006] In a first aspect, a method for training an image recognition model is provided, wherein the image recognition model comprises: an encoding module and a recognition module, wherein the encoding module is used to extract features of a target object from an image to obtain an encoding vector of the target object, and the recognition module is used to recognize the target object based on the encoding vector of the target object; the method comprises: in a first training device locally stored by the user, inputting a first image data set stored locally by the user into the encoding module to train the encoding module to obtain a trained encoding module; in a second training device in the cloud, obtaining the trained encoding module from the first training device; and, inputting a labeled second image data set stored in the cloud into an image recognition model comprising a recognition module and the trained encoding module to train the recognition module to obtain a trained recognition module.

[0007] The method uses the user's image data set to train the encoding module in the image recognition model locally. In the cloud, the trained encoding module is combined with the labeled image data set to train the recognition module in the image recognition model, thereby completing the training of the image recognition model. Through this method, the image recognition model can be trained without leaving the user's image data set locally, thereby avoiding the leakage of the user's privacy information.

[0008] Moreover, compared with the image data set, the data volume of the encoding module is small. This method sends the encoding module to the cloud instead of sending the image data set to the cloud, which saves data transmission costs while avoiding user privacy leakage.

[0009] In addition, the labeled image dataset is usually an asset in the cloud. In this method, there is no need to send the labeled image dataset in the cloud to other parties, thus avoiding the loss of cloud assets.

[0010] In one possible implementation, training the recognition model includes: extracting features of the target object from images in the second image data set based on a trained encoding module to obtain a first encoding vector of the target object; inputting the first encoding vector into the recognition module to identify the target object to obtain a first recognition result; and updating parameters of the recognition module based on the first recognition result and labels of the images in the second image data set.

[0011] In this method, in the cloud, the trained encoding module is used to extract the features of the target object from the labeled image dataset to obtain the encoding vector. The recognition module can recognize the target object based on the encoding vector. Then, the loss can be calculated based on the recognition result and label of the recognition module, so that the loss can be used to update the parameters of the recognition module to achieve the training of the recognition module.

[0012] In a possible implementation, the encoding module corresponds to a decoding module, and the encoding module is trained, including: extracting features of the target object from an image in a first image data set based on the encoding module to obtain a second encoding vector of the target object; inputting the second encoding vector into the decoding module to generate a first image; displaying the first image and the image in the first image data set; and completing the training of the encoding module when a training termination operation issued by a user is received.

[0013] In this implementation, during the training process of the encoding module, the decoding module corresponding to the encoding module generates an image based on the encoding vector extracted by the encoding module. By displaying the generated image and the image as the training set of the encoding module, the user can see the training effect of the encoding module with the human eye, and then control the training of the encoding module. In other words, the training of the encoding module can be controlled without the user having professional model training knowledge.

[0014] In one possible implementation, the encoding module corresponds to a decoding module, and the encoding module is trained, including: extracting features of the target object from an image in a first image data set based on the encoding module to obtain a second encoding vector of the target object; inputting the second encoding vector into the decoding module to generate a second image; and updating parameters of the encoding module based on the image in the first image data set and the second image.

[0015] In this implementation, during the training process of the encoding module, the decoding module corresponding to the encoding module generates an image based on the encoding vector extracted by the encoding module. By calculating the similarity between the generated image and the image as the training set of the encoding module, the training effect of the encoding module can be obtained, and then it can be decided whether to continue the training or terminate the training.

[0016] In one possible implementation, the method also includes: obtaining a trained recognition module from a second training device at a user's local verification device; and extracting features of the target object from images in the first image data set based on the trained encoding module to obtain a third encoding vector of the target object; inputting the third encoding vector into the trained encoding module to identify the target object to obtain a second recognition result; and instructing the first training device to train the encoding module again if the second recognition result is erroneous.

[0017] In this implementation, the verification device can verify the effect of the image recognition model locally on the user, and use the user's image data to verify the image recognition model while avoiding leakage of the user's privacy information. In addition, when the recognition result is wrong, the verification device triggers the retraining of the encoding module, and then triggers the retraining of the recognition module, thereby achieving the retraining of the entire image recognition model. This process does not require human participation, which improves the automation of the image recognition model.

[0018] In a possible implementation, inputting a first image data set stored locally by a user into an encoding module includes: identifying a local area where a target object is located in an image in the first image data set; and inputting the local area into the encoding module.

[0019] In this implementation, the local area where the target object is located can be used as a training set to train the encoding module. Compared with using the entire original image as a training set to train the encoding module, this implementation can reduce the computational complexity in the training process and save computing resources.

[0020] In a possible implementation, both the images in the first image data set and the images in the second image data set contain text, and the encoding module is trained, including: training the encoding module to extract the features of the text from the images in the first image data set; training the recognition module, including: extracting the features of the text from the images in the second image data set based on the trained encoding module to obtain a first encoding vector; and inputting the first encoding vector into the recognition module to recognize the text in the images in the second image data set to obtain a first recognition result. Exemplarily, the area where the text in the image is located has interference information such as a watermark or a seal, or the text is handwritten.

[0021] In this implementation, the method provided in the embodiment of the present application can be used to train a text recognition model. The training of the text recognition model requires a large number of images containing text, and these images often contain more private information. Through the method provided in the embodiment of the present application, an image recognition model that meets the user's needs can be trained while avoiding the leakage of the user's private information.

[0022] In a second aspect, an image recognition model training system is provided, characterized in that the image recognition model includes: a coding module and a recognition module, the coding module is used to extract the characteristics of the target object from the image to obtain the coding vector of the target object, and the recognition module is used to recognize the target object based on the coding vector of the target object; the system includes: a first training device locally at the user, used to input a first image data set stored locally at the user into the coding module to train the coding module and obtain a trained coding module; a second training device in the cloud is used to obtain the trained coding module from the first training device; and, inputting the labeled second image data set stored in the cloud into the image recognition model including the recognition module and the trained coding module to train the recognition module and obtain a trained recognition module.

[0023] In one possible implementation, the second training device is used to: extract features of the target object from images in the second image data set based on the trained encoding module to obtain a first encoding vector of the target object; input the first encoding vector into the recognition module to identify the target object to obtain a first recognition result; and update the parameters of the recognition module based on the first recognition result and the labels of the images in the second image data set.

[0024] In one possible implementation, the encoding module corresponds to a decoding module, and the first training device is used to: extract features of the target object from an image in a first image data set based on the encoding module to obtain a second encoding vector of the target object; input the second encoding vector into the decoding module to generate a first image; display the first image and the images in the first image data set; and complete the training of the encoding module when a training termination operation issued by the user is received.

[0025] In one possible implementation, the encoding module corresponds to a decoding module, and the first training device is used to: extract features of the target object from an image in a first image data set based on the encoding module to obtain a second encoding vector of the target object; input the second encoding vector into the decoding module to generate a second image; and update the parameters of the encoding module based on the image in the first image data set and the second image.

[0026] In one possible implementation, the system also includes: a verification device local to the user, used to obtain a trained recognition module from a second training device; and, based on the trained encoding module, extracting features of the target object from the image in the first image data set to obtain a third encoding vector of the target object; inputting the third encoding vector into the trained encoding module to identify the target object to obtain a second recognition result; and instructing the first training device to train the encoding module again if the second recognition result is erroneous.

[0027] In a possible implementation, the first training device is further used to: identify a local area where the target object is located in the image in the first image data set; and input the local area into the encoding module.

[0028] In one possible implementation, both the images in the first image data set and the images in the second image data set contain text, and the first training device is used to: train the encoding module's ability to extract features of text from the images in the first image data set; the second training device is used to: extract features of text from the images in the second image data set based on the trained encoding module to obtain a first encoding vector; and, input the first encoding vector into a recognition module to recognize the text in the images in the second image data set to obtain a first recognition result.

[0029] In a third aspect, a method for training an image recognition model is provided, which is applied to a training device in the cloud. The image recognition model includes: a coding module and a recognition module. The coding module is used to extract features of a target object from an image to obtain a coding vector of the target object. The recognition module is used to recognize the target object based on the coding vector of the target object. The method includes: obtaining a trained coding module from a user locally; the trained coding module is trained locally at the user using a first image data set stored locally at the user; and inputting a labeled second image data set stored in the cloud into an image recognition model including a recognition module and a trained coding module to train the recognition module to obtain a trained recognition module.

[0030] In one possible implementation, training the recognition model includes: extracting features of the target object from images in the second image data set based on a trained encoding module to obtain a first encoding vector of the target object; inputting the first encoding vector into the recognition module to identify the target object to obtain a first recognition result; and updating parameters of the recognition module based on the first recognition result and labels of the images in the second image data set.

[0031] In one possible implementation, both the images in the first image data set and the images in the second image data set contain text; training the recognition module includes: extracting features of the text from the images in the second image data set based on the trained encoding module to obtain a first encoding vector; and inputting the first encoding vector into the recognition module to recognize the text in the images in the second image data set to obtain a first recognition result.

[0032] In a fourth aspect, an image recognition model training device is provided, and the image recognition model includes: a coding module and a recognition module, the coding module is used to extract the features of the target object from the image and obtain the coding vector of the target object, and the recognition module is used to recognize the target object based on the coding vector of the target object; the device is located in the cloud, and the device includes: an acquisition module, used to obtain a trained coding module from the user locally; the trained coding module is trained locally by using a first image data set stored locally by the user; an input module, used to input a labeled second image data set stored in the cloud into the image recognition model including the recognition module and the trained coding module, so as to train the recognition module and obtain a trained recognition module.

[0033] In one possible implementation, the device also includes an update module; wherein the input module is used to: extract features of the target object from images in the second image data set based on the trained encoding module to obtain a first encoding vector of the target object; and input the first encoding vector into the recognition module to identify the target object to obtain a first recognition result; the update module is used to: update the parameters of the recognition module based on the first recognition result and the labels of the images in the second image data set.

[0034] In one possible implementation, both the images in the first image data set and the images in the second image data set contain text, and the input module is used to: extract features of the text from the images in the second image data set based on the trained encoding module to obtain a first encoding vector; input the first encoding vector into the recognition module to recognize the text in the images in the second image data set to obtain a first recognition result.

[0035] In a fifth aspect, a computing device cluster is provided, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster executes the method provided in the third aspect.

[0036] In a sixth aspect, a computer-readable storage medium is provided, comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method provided in the third aspect.

[0037] In a seventh aspect, a computer program product comprising instructions is provided. When the instructions are executed by a computer device cluster, the computer device cluster executes the method provided in the third aspect.

[0038] The beneficial effects of the second to seventh aspects can be referred to the above introduction to the beneficial effects of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of a check image provided in an embodiment of the present application;

[0040] Figure 2 A schematic diagram of a bank electronic receipt image provided in an embodiment of the present application;

[0041] Figure 3 A schematic diagram of the structure of an image recognition model provided in an embodiment of the present application;

[0042] Figure 4 A schematic diagram of an architecture system provided in an embodiment of the present application;

[0043] Figure 5 A flowchart of an image recognition model training method provided in an embodiment of the present application;

[0044] Figure 6 A schematic diagram of a coding module training scheme provided in an embodiment of the present application;

[0045] Figure 7 A schematic diagram of a recognition module training scheme provided in an embodiment of the present application;

[0046] Figure 8 A schematic diagram of an image recognition model verification provided in an embodiment of the present application;

[0047] Fig. 9 A schematic diagram of a coding module training scheme provided in an embodiment of the present application;

[0048] Fig. 10A A schematic diagram of an image generated by a decoding module provided in an embodiment of the present application;

[0049] Fig. 10B A schematic diagram of an image generated by a decoding module provided in an embodiment of the present application;

[0050] Fig.11 A schematic diagram of interaction between a user's local computer and the cloud provided in an embodiment of the present application;

[0051] Fig.12 A schematic block diagram of an image recognition model training system provided in an embodiment of the present application;

[0052] Fig.13 A flowchart of an image recognition model training method provided in an embodiment of the present application;

[0053] Fig.14 A schematic diagram of the structure of an image recognition model training device provided in an embodiment of the present application;

[0054] Fig.15 A schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0055] Fig.16 A schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0056] Fig.17 A schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The following will describe the solution provided by the embodiment of the present application in conjunction with the accompanying drawings. In the embodiment of the present application, "multiple" refers to two or more than two.

[0058] To facilitate understanding of the solutions of the embodiments of the present application, some technical terms that may be involved in the embodiments of the present application are first introduced before the solutions of the embodiments of the present application are introduced.

[0059] Generative model (GM): A model is established by setting conditions, and then the results are obtained by using the established model. Specifically, the generative model consists of an encoder and a decoder. The encoder is a module that can extract the essential laws and probability distribution of data based on a deep neural network and trained with a large-scale data set. The decoder is used to generate new data using the essential laws and probability distribution of the data extracted by the encoder. The essential laws and probability distribution of the extracted data can be called extracted features.

[0060] Data privacy protection (DPP) refers to the method of protecting sensitive data of users (such as enterprises or individuals). Generally speaking, data privacy protection requires that user data does not leave the user's local area to ensure privacy security.

[0061] Optical character recognition (OCR) refers to the process of analyzing and identifying image files of text materials to obtain layout information and text. The layout information is also called the text image area, which refers to the position of the text in the image. OCR generally includes two processes: text detection and text recognition. Text detection refers to the process of detecting the text image area in the image, and text recognition refers to the process of extracting text in the text image area.

[0062] Computer vision (CV): is the science of how to make machines "see". Specifically, computer vision refers to the technology of using cameras and computers to replace human eyes to identify, track and measure targets in images. In addition, in computer vision, images can be further processed and processed by computers to become images more suitable for human observation or transmission to instruments for detection. Common computer vision technologies include OCR, image classification, object detection, object segmentation, target tracking and other technologies.

[0063] Deep learning: A type of machine learning technology based on deep neural network algorithms. Its main feature is the use of multiple nonlinear transformations to process and analyze data. Deep learning is mainly used in perception, decision-making and other scenarios in the field of artificial intelligence, such as image recognition, speech recognition, natural language translation, computer games, etc.

[0064] In some scenarios, due to the particularity of user images, it is necessary to train an image recognition model specifically for the user's images, that is, to train specifically for identifying target objects from the user's images.

[0065] For example, in the task of recognizing text from an image, if the text in the image has interference information such as watermarks or seals, or the text is handwritten, it is difficult for a conventional text recognition model to recognize the text from the image. Therefore, a text recognition model needs to be specially trained for such images. The text recognition model here refers to a model for recognizing text from an image, so the text recognition model belongs to the image recognition model.

[0066] For another example, for the task of identifying a target object from an image, if the position of the target object in the image carries interference information such as a watermark, or the target object is not a common object, conventional object recognition models will find it difficult to identify the target object from the image. Therefore, it is also necessary to specially train image recognition models for such images or target objects.

[0067] Image recognition model training is a highly specialized and computationally demanding job. Many users do not have the conditions or capabilities to train image recognition models, so specialized organizations are needed to train image recognition models for users. In other words, the owner of the image and the model trainer are often not the same party.

[0068] In one solution, when the owner of the image and the model training party are not the same party, the owner of the image sends the image to the model training party. The model training party annotates the image and uses the annotated image to train the image recognition model. This solution may have the following problems.

[0069] Privacy information leakage. Specifically, images may contain sensitive information. Sensitive information can also be called private information, which refers to information involving the privacy of an individual or organization. For example, a user wants to obtain an image recognition model that can recognize text from a check image. Figure 1 As shown in Figure 1, check images include sensitive information such as the payee's name, account, amount, and purpose. For another example, a user wants to obtain an image recognition model that can recognize text from a bank electronic receipt image. Figure 2 As shown in the figure, the bank electronic receipt image includes sensitive information of the payee, sensitive information of the payee, amount, purpose, etc. If the image containing sensitive information is sent to the model training party, private information may be leaked.

[0070] Large amount of data and high data transmission cost. Specifically, image recognition models require a large amount of training data, which requires a large number of images to be transmitted to the model training party, resulting in high data transmission costs.

[0071] Labeling user images is time-consuming and labor-intensive, and the labeling cost is high.

[0072] In addition, in this solution, if the trained image recognition model does not perform well, it is necessary to manually trigger the retraining of the image recognition model, and the degree of automation of model training is low.

[0073] The embodiment of the present application provides an image recognition model and a training method for the model. The image recognition model is composed of a coding module and a recognition module. The coding module is used to extract the features of the target object from the image, and the recognition module is used to identify the target object based on the features extracted by the coding module. The method trains the coding module locally using the user's image data set. Then, in the cloud, the recognition module is trained using the labeled image data set in combination with the trained coding module to complete the training of the image recognition model. Through this training method, the training of the image recognition model can be achieved without the user's image data set leaving the user's local area, thereby avoiding the leakage of the user's privacy information. In addition, there is no need to transmit the image data set between different parties, saving the data transmission cost.

[0074] Next, the image recognition model and training method provided in the embodiments of the present application are introduced.

[0075] Figure 1 FIG. 1 shows an image recognition model 100 provided in an embodiment of the present application. Figure 1 As shown, the image recognition model 100 includes an encoding module 110 and a recognition module 120 .

[0076] The input of the encoding module 110 is an image. The encoding module 110 can extract the features of the target object from the input image, obtain and output the encoding vector of the target object. The encoding vector of the target object refers to the features extracted by the encoding module 110 from the image.

[0077] The encoding vector of the target object output by the encoding module 110 is the input of the recognition module 120. The recognition module 120 can recognize the target object based on the input encoding vector, and obtain and output the recognition result of the target object.

[0078] The image recognition model 100 may adopt a neural network structure. The encoding module 110 is composed of one or more neural network layers, and the recognition module 120 may also be composed of one or more neural network layers. Each neural network layer has one or more parameters, and the data input to the neural network layer is transformed (e.g., nonlinear transformation) by the one or more parameters. The transformed data can be output to the next layer or output as the final result.

[0079] In some embodiments, the encoding module 110 may adopt an encoder structure in a transformer. Specifically, the encoding module 110 is composed of multiple encoding layers (encoders). In the encoding layer, the features of the target object in the image are extracted using a self-attention mechanism, etc., to obtain an encoding vector of the target object. In other embodiments, the encoding module 110 may also adopt other neural network structures, such as a recurrent neural network (RNN), a convolutional neural network (CNN), etc.

[0080] In some embodiments, the recognition module 120 includes a feature conversion layer and a classification layer. Specifically, the process by which the encoding module 110 extracts the features of the target object from the input image can be understood as the process of converting the high-dimensional information of the image (i.e., the original information of the image) into the low-dimensional information of the image. Compared with high-dimensional information, although low-dimensional information retains the key features of the target object, it lacks details. In order to improve the recognition accuracy, it is necessary to convert the low-dimensional information extracted by the encoding module 110 (i.e., the encoding vector of the target object) into high-dimensional information. In other words, it is necessary to fill in the details based on the information represented by the encoding vector. This task is performed by the feature conversion layer. In one example, the feature conversion layer can be an RNN. In another example, the feature conversion layer can be a CNN.

[0081] The classification layer is used to classify the target object based on the data output by the feature conversion layer to identify the target object. In one example, when the target object is text, the classification layer is trained based on the connectionist temporal classification (CTC) algorithm. In other words, the classification layer recognizes text based on the CTC algorithm. In one example, when the target object is an object in an image, the classification layer is trained based on the cross entropy algorithm or the softmax algorithm. In other words, the classification layer recognizes objects based on the cross entropy algorithm or the softmax algorithm.

[0082] The above examples introduce the image recognition model 100 provided in the embodiment of the present application. Next, a system architecture for training the image recognition model 100 is introduced.

[0083] First, a system architecture provided in an embodiment of the present application is introduced. The system architecture can be used to implement the training method provided in an embodiment of the present application to obtain an image recognition model 100.

[0084] like Figure 4 As shown, the system architecture includes a training device 200 located locally at the user and a training device 300 located in the cloud.

[0085] The training device 200 is used to train the encoding module 110 to extract the features of the target object from the image using the image data set A1 stored locally by the user. The image data set A1 includes multiple images, and the images contain the target object. The training device 200 can input the image data set A1 into the encoding module 110, so that the encoding module 110 uses the image data set A1 as a training set to train the ability to extract the features of the target object.

[0086] In some embodiments, Figure 4 As shown, the training device 200 includes a decoding module 210 corresponding to the encoding module 110. The decoding module 210 is used to generate an image containing the target object using the encoding vector of the target object extracted by the encoding module 110. Exemplarily, the encoding module 110 and the decoding module 210 can form a generative model. That is, the encoding module 110 can be implemented as an encoder in the generative model, and the decoding module 210 can be implemented as a decoder in the generative model.

[0087] By comparing the similarity between the image generated by the decoding module 210 and the image in the image data set A1, it can be determined whether the encoding module 110 has the ability to extract the features of the target object from the image in the image data set A1.

[0088] In one example of this embodiment, Figure 4 As shown, the training device 200 includes a display module 220. The display module 220 is used to display the images generated by the decoding module 210 and the images in the image data set A1, so that the user can see the images generated by the decoding module 210 and the images in the image data set A1, thereby realizing the similarity between the images generated by the decoding module 210 and the images in the image data set A1 by the human eye, and then when the similarity between the images generated by the decoding module 220 and the images in the image data set A1 is high, the training of the encoding module 110 can be terminated to obtain a trained encoding module. Exemplarily, the training device 200 includes an effect confirmation module 230. The effect confirmation module 230 can receive a training termination operation issued by the user, and in response to the operation, terminate the training of the encoding module 110, that is, complete the training of the encoding module 110, and obtain a trained encoding module 110.

[0089] In another example of this embodiment, the training device includes a similarity calculation module (not shown). The similarity calculation module can calculate the similarity between the image generated by the decoding module 210 and the image in the image data set A1, for example, pixel similarity. Then, based on the calculated similarity, it is determined whether to terminate the training of the encoding module 110.

[0090] In some embodiments, Figure 4As shown, a preprocessing module 240 is also provided locally in the user. The preprocessing module 240 is used to preprocess the image data set A1 before inputting the image data set A1 into the encoding module 110. Specifically, the preprocessing module 240 includes a detection submodule 241 and a slicing submodule 242. Among them, the detection submodule 241 is used to detect the position of the target object in the image, that is, to detect in which local area of ​​the image the target object is located. The slicing submodule 242 is used to cut off the local area where the target object is located in the image, and input the cut local area into the encoding module 110. In this way, the encoding module 110 only needs to extract the features of the target object from the local area. Compared with the entire image, the local area has a smaller range and fewer pixels. Therefore, compared with extracting the features of the target object from the entire image, less computing resources are required to extract the features of the target object from the local area.

[0091] In an example of this embodiment, the target object is text, and the detection module can be a pre-trained deep bidirectional neural network (DBNET). DBNET is a deep learning model for text detection, which can detect the text area in the image and output its position and size information. Thus, the area of ​​the text in the image and the size of the area can be obtained.

[0092] In some embodiments, Figure 4 As shown, the user also locally deploys a verification device 400. The verification device 400 verifies the effect of the trained image recognition model 100, and triggers retraining of the encoding module 110 and the recognition module 120 when the effect is not good.

[0093] Continue reading Figure 4 , the training device 300 in the cloud can obtain the trained encoding module 110 from the training device 200. Among them, the trained encoding module 110 and the untrained recognition module 120 constitute an untrained image recognition model 100. The training device 300 can use the image data set A2 stored in the cloud to train the untrained image recognition model 100. Among them, the training of the image recognition model 100 by the training device 300 is specifically to train the recognition module 120, that is, during the training process of the image recognition model 100, the parameters of the recognition module 120 are updated without updating the parameters of the encoding module 110. The image data set A2 is annotated data. Specifically, the images in the image data set A2 include the target object, and the images have the label of the target object. The image data set A2 can be used to implement supervised training of the image recognition model 110, so that the image recognition model 110 learns the ability to recognize the target object.

[0094] The above generally introduces the functions of each device and module in the system architecture provided by the embodiments of the present application. The functions of these devices and modules will also be described in detail in the method embodiments below.

[0095] Each device in the above system architecture can be implemented as any device, equipment, cluster or platform with data processing function. In some embodiments, each device in the system architecture can be implemented by hardware, for example, the training device 200 or the training device 300 can be a server. In some embodiments, each device in the system architecture can be implemented by software, for example, the training device 200 or the training device 300 can be a virtual machine (VM) or a container.

[0096] The above describes the image recognition model and system architecture provided by the embodiment of the present application. Next, in combination with the image recognition model and system architecture described above, the image recognition model training method provided by the embodiment of the present application is introduced.

[0097] See also Figure 5 The user's local training device 200 can execute step 501 to input the image data set A1 into the encoding module 110 to train the encoding module 110.

[0098] The image data set A1 is data stored locally by the user. When executing step 501, the training device 200 obtains the image data set A1 from the user's local storage and inputs the image data set A1 into the encoding module 110 deployed locally by the user. Thus, the image data set A1 can be input into the encoding module 110 without going through an external network such as the Internet, thereby avoiding the leakage of user privacy data.

[0099] The image data set A1 includes multiple images such as image A11. Image A11 contains the target object of the image recognition model 100. The target object can be text or an object (such as a person, a vehicle, a plant, etc.). Figure 1 Taking the check image shown as an example, the target object can be the text in the check image, such as "One hundred thousand yuan", "200307094100857110", "28184557" and so on.

[0100] In some embodiments, Figure 5 As shown, step 501 may include step 5011 and step 5012. Through step 5011, the training device 200 directly inputs the image data set A1 to the encoding module 110. In step 5012, the encoding module 110 extracts the features of the target object from the image in the image data set A1 (for example, image A11) to obtain the encoding vector B1.

[0101] In some embodiments, Figure 6 As shown, the training device 200 includes a preprocessing module 240. The preprocessing module 240 can detect the area where the target object is located in the image (for example, image A11). The slicing submodule 242 can cut off the area where the target object is located to obtain a slice. Among them, the area where the target object is located is a local area of ​​image A11. In other words, the obtained slice is a local area of ​​image A11. The slice can be input into the encoding module 110. The encoding module 110 can extract the features of the target object from the slice to obtain the encoding vector B1. Compared with extracting the features of the target object from the entire image, extracting the features of the target object from the slice can reduce the amount of calculation and save computing resources.

[0102] See also Figure 5 or Figure 6 In step 5013, the encoding module 110 may input the encoding vector B1 to the decoding module 210. The decoding module 210 generates the image C1 based on the encoding vector B1.

[0103] In one example, in step 5014, the decoding module 210 may input the image C1 to the display module 220. In step 5014, the display model 220 may display the image C1. The display model 220 may also display the image A11 or a slice. Wherein, when the image input to the encoding module 110 is the image A11, the display model 220 displays the image A11. When the image input to the encoding module 110 is a slice, the display model 220 displays the slice. In one example, the display model 220 displays the image A11 or the slice while displaying the image C1. In this way, even users who have no knowledge of model training can learn the training effect of the encoding module 110 by observing the image C1, the image A11 or the slice. When the difference between the image C1 and the image A11 or the slice is large, or when the target object in the image C1 and the target object in the image A11 or the slice are large, the user does not make a training termination operation, so that the encoding module 110 and the decoding module 210 continue to perform iterative training. Until the user observes that the difference between image C1 and image A11 or slice is small, or in other words, the difference between the target object in image C1 and the target object in image A11 or slice is small, the user can make a training termination operation. The effect confirmation module 230 can receive the training termination operation, and in response to the training termination operation, terminate the training of the encoding module 110, and obtain the trained encoding module 110.

[0104] In this example, by displaying the image generated based on the coding vector B1 (i.e., image C1) and the original image (i.e., image A11 or slice), the training of the coding module 110 is visualized, so that the user can know when to terminate the training of the coding module 110 and obtain the trained coding module 110. Moreover, in this way, the problem of difficulty in converging the training of the coding module 110 can be avoided by the user.

[0105] In one example, the training device 200 can calculate the similarity between the image generated based on the coding vector B1 (i.e., image C1) and the original image (i.e., image A11 or slice). In one example, the similarity between image C1 and the original image can specifically be the pixel similarity between image C1 and the original image, such as the Euclidean distance between pixels. Based on the similarity between image C1 and the original image, the parameters of the encoding module 110 and the decoding module 210 are updated. When the similarity between the similarity between image C1 and the original image is greater than a preset threshold, the training can be terminated, that is, the training of the encoding module 110 is completed, and the trained encoding module 110 is obtained.

[0106] In this example, the encoding module 110 is trained by comparing the pixel degree between the image generated by the encoding vector B1 and the original image, without user participation, thus saving user operations.

[0107] Through the above method, the training of the encoding module 110 can be completed locally by the user.

[0108] Continue reading Figure 5 , the training device 300 located in the cloud can obtain the trained encoding module 110 from the training device 200 through step 502. Specifically, the user's local training device 200 can send the trained encoding module 110 to the training device 300 in the cloud through the network. The trained encoding module 110 itself is composed of network structure information and parameters. Compared with the image data set used as a training set (for example, image data set A1), the amount of data of the trained encoding module 110 is small. Generally speaking, the amount of data of the trained encoding module 110 is less than 1GB. In addition, the trained encoding module 110 does not include the user's image data set and does not involve privacy leakage issues. Therefore, sending the trained encoding module to the cloud not only avoids privacy leakage, but also reduces the transmission bandwidth and reduces the transmission cost.

[0109] An untrained recognition module 120 is deployed in the cloud. Figure 7As shown, after acquiring the trained encoding module 110, the training device 300 combines the recognition module 120 and the trained encoding module 110 into the image recognition model 100. Then, in step 503, the image data set A2 is input into the recognition model 100 including the recognition module 120 and the trained encoding module 110 to train the recognition module 120.

[0110] The image dataset A2 is a labeled image dataset owned by the cloud. The labeled image dataset consists of multiple images such as image A21. Figure 7 As shown, image A21 contains the target object and has a target object label. Using the labeled image data set, the recognition module 120 in the image recognition model 100 can be supervised trained so that the recognition module 120 learns the ability to recognize the target object.

[0111] In some embodiments, the target object in the image of the image data set A2 and the target object in the image of the image data set A1 have the same or similar interference information or characteristics. For example, the area where the target object is located has a watermark or a seal. For another example, the target objects are all handwritten text. This can ensure the consistency of the image data set used for the recognition module and the user's image data set, thereby ensuring the recognition effect of the trained image recognition model on the user's image data set.

[0112] In some embodiments, the image data set A2 may be synthesized data based on the sample images mentioned by the user. Specifically, the user may provide one or more sample images to the cloud, and the sample images may be desensitized. The sample images display interference information or characteristics of the target object. The cloud may generate an image containing the target object based on the interference information or characteristics of the target object displayed in the sample images, and the interference information or characteristics of the target object in the generated image may be the same or similar to the interference information or characteristics of the target object in the sample images.

[0113] In some embodiments, the image dataset A2 may be data accumulated in the cloud.

[0114] In some embodiments, step 503 includes step 5031, step 5032, step 5033, and step 5034. In step 5031, the image data set A2 is input into the encoded encoding module 110, and the trained encoding module 110 extracts the features of the target object from the image in the image data set A2 (e.g., image A21) to obtain the encoding vector B2. The trained encoding module 110 can input the encoding vector into the recognition module 120 through step 5032.

[0115] In step 5033, the recognition module 120 recognizes the target object based on the coding vector B2 and obtains a recognition result. Specifically, as described above, the recognition module 120 includes a feature conversion layer and a classification layer. In the feature conversion layer, feature conversion is performed on the coding vector B2, for example, the low-dimensional coding vector B2 is converted into high-dimensional information. In the classification layer, classification is performed based on the converted features to obtain a recognition result.

[0116] Then, in step 5034, the parameters of the recognition module 120 are updated based on the recognition result obtained in step 5033 and the label of the target object. Specifically, the parameters of the recognition module 120 are updated in a direction that reduces the difference between the recognition result obtained in step 5033 and the label of the target object.

[0117] In this way, after multiple iterations, the training of the recognition module 120 can be completed to obtain a trained recognition module 120 .

[0118] In addition, the labeled data is usually an asset of the cloud. If the labeled data is sent to other parties, it may lead to the loss of cloud assets. In the embodiment of the present application, the cloud uses the labeled image data set to train the recognition module 120 in the image recognition model 100, thereby completing the supervised training of the image recognition model 100 without losing cloud assets.

[0119] The trained recognition module 120 and the trained encoding module 110 constitute a trained image recognition model 100. The trained image recognition model 100 can be deployed locally to the user, so that the image recognition model 100 can be used to recognize target objects from images in the user's image dataset (e.g., image dataset A1).

[0120] In some embodiments, as described above, the trained encoding module 110 is obtained by training at the user's local location, so the user's local location has the trained encoding module 110. The training device 300 can send the trained recognition module 120 to the user's local location. At the user's local location, the trained recognition module 120 and the trained encoding module 110 are combined into the image recognition model 100.

[0121] In some embodiments, the cloud can send the image recognition model 100 including the trained encoding module 110 and the trained recognition module 120 to the user locally.

[0122] In some embodiments, see Figure 8 The user also has a verification device 400 deployed locally. The verification device 400 is used to verify the effect of the image recognition model 100 in recognizing the target object from the user's image, and trigger the retraining of the encoding module 110 when the effect is not good. The details are as follows.

[0123] The verification device 400 may obtain the trained encoding module 110 from the training device 200 or the training device 300, and obtain the trained recognition module 120 from the training device 300. The verification device 400 inputs the image data set A1 into the trained encoding module 110, so as to extract the features of the target object from the image in the image data set A1 using the trained encoding module 110, and obtain the encoding vector B3. Then, the encoding vector B3 is input into the trained recognition module 120. The recognition module 120 recognizes the target object based on the encoding vector B3 and obtains a recognition result.

[0124] If the recognition result is wrong, the verification device 400 triggers retraining of the encoding module 110. Exemplarily, the user can determine whether the recognition result is wrong. If the user determines that the recognition result is wrong, an operation for indicating that the recognition result is wrong can be issued. In response to the operation, the verification device 400 can trigger retraining of the encoding module 110, for example, triggering the preprocessing module 240 to start preprocessing the images in the image data set A1. The preprocessing result of the preprocessing module 240 is input to the encoding module 110 to trigger the training device 200 to train the encoding module 110.

[0125] The training device 200 may send the retrained encoding module 110 to the training device 300 to trigger the training device 300 to retrain the recognition module 120. The retraining of the encoding module 110 may be implemented by referring to the above introduction to the training of the encoding module 110, and the retraining of the recognition module 120 may be implemented by referring to the above introduction to the training of the recognition module 120, which will not be described in detail here.

[0126] The verification device 400 allows the user to verify the effect of the image recognition model 100, thereby realizing the verification of the image recognition model 100 without leaking the user's privacy information. In addition, the verification device 400 triggers the retraining of the encoding module 110, and then triggers the retraining of the recognition module, thereby realizing the retraining of the entire image recognition model 100. This process does not require human participation and is automatically executed by the verification device 400, the preprocessing module 240, the training device 200 and the training device 300.

[0127] Through the above scheme, an image recognition model with recognition effect that meets the requirements can be trained.

[0128] Next, in conjunction with the text recognition task, the image recognition model training method provided in the embodiment of the present application is introduced by way of example.

[0129] Bill documents such as checks and electronic receipts include personal privacy information such as personal name, address, amount, etc. When training for such bill documents, privacy information protection is the primary consideration. In view of this, in the embodiment of the present application, the encoding module 110 in the image recognition model 100 is trained locally using the image data set of such bill documents. The details are as follows.

[0130] See also Fig. 9 Taking a check as an example, the image of the check can be input into the preprocessing module 240. The preprocessing module 240 cuts off the local area where the text in the check image is located to obtain a text slice. The text slice is then input into the encoding module 110. The encoding module 110 extracts the features of the text from the text slice to obtain a coding vector B4. The decoding module 210 generates an image C2 based on the coding vector B4. The display module 220 displays the image C2 and the text slice to the user.

[0131] If the image C2 is as shown in FIG10 , there are many noise points, and the human eye cannot clearly see the text in the image C2, it means that the generative model composed of the encoding module 110 and the decoding module 210 has not fully converged. In this case, the parameters of the encoding module 110 and the decoding module 210 are continuously updated.

[0132] As shown in image C2 Fig.11 As shown, it is consistent or almost consistent with the text slice, and the text in image C2 is clearly visible. This indicates that the generative model composed of the encoding module 110 and the decoding module 210 has converged. At this time, the encoding module 110 is a trained encoding module 110.

[0133] See also Fig.11 , the trained encoding module 110 can be transmitted from the user's local to the cloud. Generally speaking, the user's image data set is usually at the level of tens of TB, while the data volume of the encoding module 110 is less than 1GB. Therefore, transmitting the encoding module 110 to the cloud instead of transmitting the user's image data set to the cloud can avoid privacy information leakage while reducing transmission costs.

[0134] In the cloud, the trained encoder 110 and the labeled image data set can be combined to train the image recognition model 100. For details, please refer to the above introduction, which will not be repeated here.

[0135] Continue reading Fig.11 , the trained image recognition model 100 can be sent from the cloud to the user's local area. The verification device 400 deployed at the user's local area can verify the effect of the image recognition model 100 in identifying the target object from the user's image. And when the effect is not good, it triggers the retraining of the encoding module 110, and then triggers the retraining of the image recognition model 100.

[0136] The above text uses the text recognition task as an example to introduce the training method provided by the embodiment of the present application. The training method can also be applied to other tasks that require identifying target objects from images, such as classification, detection, segmentation, target tracking and other tasks in the field of computer vision.

[0137] The training method provided in the embodiment of the present application uses the user's image data set locally to train the encoding module's ability to recognize target objects from the user's image data set. This achieves the training of the encoding module without leaving the user's image data set locally, thereby avoiding leakage of user privacy.

[0138] Moreover, in the training method provided in the embodiment of the present application, the encoding module is transmitted from the user's local to the cloud, and there is no need to transmit the user's image data set to the cloud, which reduces the data transmission cost while avoiding the leakage of user privacy. Due to this advantage, even if there is no need to protect privacy information, the training method provided in the embodiment of the present application can also be applied to scenarios where the data volume of the training set is large and inconvenient to transmit. For example, the scenario of identifying the target object from the drawing. Generally speaking, the data volume of the image data set of the drawing is huge and the transmission cost is very high. Using the solution provided in the embodiment of the present application, there is no need to transmit the image data set of the drawing, and the training of the encoding module can be implemented on the storage side of the image data set of the drawing.

[0139] Furthermore, in the training method provided in the embodiment of the present application, the annotated image dataset in the cloud is used to implement supervised training of the image recognition model, without the need to annotate the user's image dataset, thus saving manpower and time costs. Due to this advantage, the training method provided in the embodiment of the present application can be applied to multilingual recognition tasks. Specifically, for users, it is more difficult to find annotators for some languages. The training method provided in the embodiment of the present application does not require annotating the user's image dataset, and therefore, the user does not need to look for annotators.

[0140] In addition, the training method provided in the embodiment of the present application enables the user to verify the effect of the image recognition model by deploying a verification device locally on the user's side, thereby avoiding the risk of privacy information leakage caused by image recognition model verification.

[0141] Based on the above description, the present application embodiment provides an image recognition model training system 1200, wherein the image recognition model includes: an encoding module and a recognition module, wherein the encoding module is used to extract the features of the target object from the image to obtain the encoding vector of the target object, and the recognition module is used to recognize the target object based on the encoding vector of the target object. Fig.12 As shown, the system 1200 includes:

[0142] A first training device 1210 locally stored in the user is used to input the first image data set stored locally in the user into the encoding module to train the encoding module and obtain a trained encoding module;

[0143] The second training device 1220 in the cloud is used to obtain the trained encoding module from the first training device; and input the labeled second image data set stored in the cloud into the image recognition model including the recognition module and the trained encoding module to train the recognition module to obtain a trained recognition module.

[0144] In some embodiments, the second training device 1220 is used to: extract features of the target object from images in the second image data set based on the trained encoding module to obtain a first encoding vector of the target object; input the first encoding vector into the recognition module to identify the target object to obtain a first recognition result; and update the parameters of the recognition module based on the first recognition result and the labels of the images in the second image data set.

[0145] In some embodiments, the encoding module corresponds to a decoding module, and the first training device 1210 is used to: extract the features of the target object from the image in the first image data set based on the encoding module to obtain a second encoding vector of the target object; input the second encoding vector into the decoding module to generate a first image; display the first image and the images in the first image data set; and complete the training of the encoding module when a training termination operation issued by the user is received.

[0146] In some embodiments, the encoding module corresponds to a decoding module, and the first training device 1210 is used to: extract the features of the target object from the image in the first image data set based on the encoding module to obtain a second encoding vector of the target object; input the second encoding vector into the decoding module to generate a second image; and update the parameters of the encoding module based on the image in the first image data set and the second image.

[0147] In some embodiments, the system 1200 also includes: a verification device 1230 local to the user, used to obtain the trained recognition module from the second training device; and, based on the trained encoding module, extracting features of the target object from the image in the first image data set to obtain a third encoding vector of the target object; inputting the third encoding vector into the trained encoding module to identify the target object to obtain a second recognition result; in case the second recognition result is erroneous, instructing the first training device to train the encoding module again.

[0148] In some embodiments, the first training device 1210 is further used to: identify a local area where the target object is located in the image in the first image data set; and input the local area into the encoding module.

[0149] In some embodiments, the images in the first image data set and the images in the second image data set both contain text, and the first training device 1210 is used to: train the encoding module to extract features of text from images in the first image data set; the second training device 1220 is used to: extract features of text from images in the second image data set based on the trained encoding module to obtain a first encoding vector; and, input the first encoding vector into the recognition module to recognize text in the images in the second image data set to obtain a first recognition result.

[0150] Among them, the function of the first training device 1210 can also be implemented by referring to the above introduction of the training device 200, the function of the second training device 1220 can also be implemented by referring to the above introduction of the training device 300, and the function of the verification device 1230 can also be implemented by referring to the above introduction of the verification device 400.

[0151] The first training device 1210, the second training device 1220, and the verification device 1230 can all be implemented by software or hardware. As an example, the implementation of the first training device 1210 is described below. Similarly, the implementation of the second training device 1220 and the verification device 1230 can refer to the implementation of the first training device 1210.

[0152] As an example of a software functional unit, the first training device 1210 may include code running on a computing instance. The computing instance may be at least one of a physical host (computing device), a virtual machine, a container, and other computing devices. Furthermore, the computing device may be one or more. For example, the first training device 1210 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application may be distributed in the same region or in different regions. The multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple data centers with close geographical locations. Typically, a region may include multiple AZs.

[0153] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0154] As an example of a hardware functional unit, the first training device 1210 may include at least one computing device, such as a server, etc. Alternatively, the first training device 1210 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0155] The multiple computing devices included in the first training device 1210 can be distributed in the same region or in different regions. The multiple computing devices included in the first training device 1210 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the first training device 1210 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0156] Based on the above description, the embodiment of the present application provides an image recognition model training method. The method can be applied to a training device in the cloud, such as the training device 300 or the first training device 1220 described above. The image recognition model includes: a coding module and a recognition module, the coding module is used to extract the features of the target object from the image to obtain the coding vector of the target object, and the recognition module is used to identify the target object based on the coding vector of the target object. Fig.13 As shown, the method includes the following steps.

[0157] Step 1301, obtain a trained encoding module from the user's local computer; the trained encoding module is obtained by training the user's local computer using the first image data set stored in the user's local computer. Figure 5 The introduction and implementation of step 501 and step 502.

[0158] Step 1302: Input the annotated second image data set stored in the cloud into the image recognition model including the recognition module and the trained encoding module to train the recognition module to obtain a trained recognition module. Figure 5 The introduction of step 503 is implemented.

[0159] In some embodiments, the training of the recognition model includes: extracting the features of the target object from the images in the second image data set based on the trained encoding module to obtain a first encoding vector of the target object; inputting the first encoding vector into the recognition module to identify the target object and obtain a first recognition result; and updating the parameters of the recognition module based on the first recognition result and the labels of the images in the second image data set. For details, please refer to the above Figure 5 The introduction and implementation of steps 5031 to 5034.

[0160] In some embodiments, the images in the first image data set and the images in the second image data set both contain text; the training of the recognition module includes: extracting features of text from the images in the second image data set based on the trained encoding module to obtain a first encoding vector; and inputting the first encoding vector into the recognition module to recognize the text in the images in the second image data set to obtain a first recognition result. Figure 9-11 The illustrated embodiment is described in detail.

[0161] The embodiment of the present application also provides an image recognition model training device 1400. The image recognition model includes: a coding module and a recognition module, the coding module is used to extract the features of the target object from the image to obtain the coding vector of the target object, and the recognition module is used to identify the target object based on the coding vector of the target object. The device 1400 is located in the cloud, and the device 1400 includes:

[0162] The acquisition module 1410 is used to acquire a trained encoding module from a user locally; the trained encoding module is obtained by training at the user locally using a first image data set stored at the user locally;

[0163] The input module 1420 is used to input the annotated second image data set stored in the cloud into the image recognition model including the recognition module and the trained encoding module to train the recognition module and obtain a trained recognition module.

[0164] Among them, the acquisition module 1410 and the input module 1420 can be implemented by software or by hardware. Exemplarily, the implementation of the acquisition module 1410 is described below by taking the acquisition module 1410 as an example. Similarly, the implementation of the input module 1420 can refer to the implementation of the acquisition module 1410.

[0165] As an example of a software functional unit, the acquisition module 1410 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the acquisition module 1410 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same AZ or in different AZs, and each AZ includes a data center or multiple data centers with similar geographical locations. Among them, usually a region can include multiple AZs.

[0166] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same VPC or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0167] As an example of a hardware functional unit, the acquisition module 1410 may include at least one computing device, such as a server, etc. Alternatively, the acquisition module 1410 may also be a device implemented using an ASIC, or a programmable logic device PLD, etc. The PLD may be implemented using a CPLD, an FPGA, a GAL, or any combination thereof.

[0168] The multiple computing devices included in the acquisition module 1410 can be distributed in the same region or in different regions. The multiple computing devices included in the acquisition module 1410 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the acquisition module 1410 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0169] It should be noted that, in other embodiments, the acquisition module 1410 may be used to execute Fig.13 In any step of the method shown, the input module 1420 can be used to perform Fig.13 The steps implemented by the acquisition module 1410 and the input module 1420 can be specified as needed, and the acquisition module 1410 and the input module 1420 can be respectively implemented Fig.13 The different steps in the method shown are used to implement the overall functions of the apparatus 1400 .

[0170] The present application also provides a computing device 1500. Fig.15 As shown, computing device 1500 includes: bus 1502, processor 1504, memory 1506 and communication interface 1508. Processor 1504, memory 1506 and communication interface 1508 communicate through bus 1502. Computing device 1500 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in computing device 1500.

[0171] The bus 1502 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.15 The bus 1502 may include a path for transmitting information between various components of the computing device 1500 (eg, the memory 1506, the processor 1504, and the communication interface 1508).

[0172] The processor 1504 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0173] The memory 1506 may include a volatile memory, such as a random access memory (RAM). The memory 1506 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0174] The memory 1506 stores executable program codes, and the processor 1504 executes the executable program codes to respectively implement the functions of the acquisition module 1410 and the input module 1420, thereby implementing Fig.13 That is, the memory 1506 stores the method for executing Fig.13 Instructions for the method shown.

[0175] The communication interface 1508 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 1500 and other devices or communication networks.

[0176] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0177] like Fig.16 As shown, the computing device cluster includes at least one computing device 1500. The memory 1506 in one or more computing devices 1500 in the computing device cluster may store the same Fig.13 Instructions for the method shown.

[0178] In some possible implementations, the memory 1506 of one or more computing devices 1500 in the computing device cluster may also store a program for executing Fig.13In other words, the combination of one or more computing devices 1500 can jointly execute instructions for performing Fig.13 Instructions for the method shown.

[0179] It should be noted that the memory 1506 in different computing devices 1500 in the computing device cluster may store different instructions, which are respectively used to execute part of the functions of the apparatus 1400. That is, the instructions stored in the memory 1506 in different computing devices 1500 may implement the functions of one or more modules in the acquisition module 1410 and the input module 1420.

[0180] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.17 A possible implementation is shown. Fig.17 As shown, two computing devices 1500A and 1500B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 1506 in the computing device 1500A stores instructions for executing the functions of the acquisition module 1410. At the same time, the memory 1506 in the computing device 1500B stores instructions for executing the functions of the input module 1420.

[0181] It should be understood that Fig.17 The functionality of the computing device 1500A shown in FIG. 1 may also be implemented by multiple computing devices 1500. Similarly, the functionality of the computing device 1500B may also be implemented by multiple computing devices 1500.

[0182] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Fig.16 and Fig.17 The difference is that the memory 1506 in one or more computing devices 1500 in the computing device cluster may store the same memory for executing Fig.13 Instructions for the method shown.

[0183] In some possible implementations, the memory 1506 of one or more computing devices 1500 in the computing device cluster may also store a program for executing Fig.13 In other words, the combination of one or more computing devices 1500 can jointly execute instructions for performing Fig.13 Instructions for the method shown.

[0184] The present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes Fig.13 The method shown.

[0185] The present application also provides a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be stored by a computing device or a host migration device such as a data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute Fig.13 The method shown.

[0186] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for training an image recognition model, characterized in that: The image recognition model comprises: an encoding module and a recognition module, wherein the encoding module is used to extract features of a target object from an image to obtain a coding vector of the target object, and the recognition module is used to recognize the target object based on the coding vector of the target object; the method comprises: In a first training device locally stored by a user, inputting a first image data set locally stored by the user into the encoding module to train the encoding module and obtain a trained encoding module; A second training device in the cloud obtains the trained encoding module from the first training device; and The annotated second image data set stored in the cloud is input into an image recognition model including the recognition module and the trained encoding module to train the recognition module to obtain a trained recognition module.

2. The method according to claim 1, characterized in that The training of the recognition model comprises: Extracting features of the target object from images in the second image data set based on the trained encoding module to obtain a first encoding vector of the target object; Inputting the first encoding vector into the recognition module to recognize the target object and obtain a first recognition result; Based on the first recognition result and labels of images in the second image data set, parameters of the recognition module are updated.

3. The method according to claim 1 or 2, characterized in that: The encoding module corresponds to a decoding module, and the training of the encoding module includes: Extracting features of the target object from the image in the first image data set based on the encoding module to obtain a second encoding vector of the target object; Inputting the second encoding vector into the decoding module to generate a first image; displaying the first image and images in the first image dataset; When a training termination operation issued by the user is received, the training of the encoding module is completed.

4. The method according to claim 1 or 2, characterized in that: The encoding module corresponds to a decoding module, and the training of the encoding module includes: Extracting features of the target object from the image in the first image data set based on the encoding module to obtain a second encoding vector of the target object; Inputting the second encoding vector into the decoding module to generate a second image; Based on the images in the first image data set and the second image, parameters of the encoding module are updated.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: The user's local verification device obtains the trained recognition module from the second training device; and Extracting features of the target object from the images in the first image data set based on the trained encoding module to obtain a third encoding vector of the target object; Inputting the third encoding vector into the trained encoding module to identify the target object and obtain a second recognition result; In case the second recognition result is erroneous, the first training device is instructed to train the encoding module again.

6. The method according to any one of claims 1 to 5, characterized in that The step of inputting the first image data set stored locally by the user into the encoding module comprises: In the image in the first image data set, identifying a local area where the target object is located; The local area is input into the encoding module.

7. The method according to any one of claims 1 to 6, characterized in that The images in the first image data set and the images in the second image data set both contain text, The training of the encoding module includes: training the encoding module to be able to extract features of text from images in the first image data set; The training of the recognition module includes: extracting features of text from images in the second image data set based on the trained encoding module to obtain a first encoding vector; and The first encoding vector is input into the recognition module to recognize text in the image in the second image data set to obtain a first recognition result.

8. An image recognition model training system, characterized in that: The image recognition model includes: a coding module and a recognition module, wherein the coding module is used to extract the features of the target object from the image to obtain the coding vector of the target object, and the recognition module is used to recognize the target object based on the coding vector of the target object; the system includes: A first training device at a user's location is used to input a first image data set stored at the user's location into the encoding module to train the encoding module and obtain a trained encoding module; a second training device in the cloud, used to obtain the trained encoding module from the first training device; and The annotated second image data set stored in the cloud is input into an image recognition model including the recognition module and the trained encoding module to train the recognition module to obtain a trained recognition module.

9. The system according to claim 8, characterized in that The second training device is used for: Extracting features of the target object from images in the second image data set based on the trained encoding module to obtain a first encoding vector of the target object; Inputting the first encoding vector into the recognition module to recognize the target object and obtain a first recognition result; Based on the first recognition result and labels of images in the second image data set, parameters of the recognition module are updated.

10. The system according to claim 8 or 9, characterized in that The encoding module corresponds to a decoding module, and the first training device is used for: Extracting features of the target object from the image in the first image data set based on the encoding module to obtain a second encoding vector of the target object; Inputting the second encoding vector into the decoding module to generate a first image; displaying the first image and images in the first image dataset; When a training termination operation issued by the user is received, the training of the encoding module is completed.

11. The system according to claim 8 or 9, characterized in that: The encoding module corresponds to a decoding module, and the first training device is used for: Extracting features of the target object from the image in the first image data set based on the encoding module to obtain a second encoding vector of the target object; Inputting the second encoding vector into the decoding module to generate a second image; Based on the images in the first image data set and the second image, parameters of the encoding module are updated.

12. The system according to any one of claims 8 to 11, characterized in that: The system further comprises: A verification device at the user's local location is used to obtain the trained recognition module from the second training device; and Extracting features of the target object from the images in the first image data set based on the trained encoding module to obtain a third encoding vector of the target object; Inputting the third encoding vector into the trained encoding module to identify the target object and obtain a second recognition result; In case the second recognition result is erroneous, the first training device is instructed to train the encoding module again.

13. The system according to any one of claims 8 to 12, characterized in that: The first training device is also used for: In the image in the first image data set, identifying a local area where the target object is located; The local area is input into the encoding module.

14. The system according to any one of claims 8 to 13, characterized in that: The images in the first image data set and the images in the second image data set both contain text, The first training device is used to: train the encoding module to extract features of text from images in the first image data set; The second training device is used to: extract features of text from images in the second image data set based on the trained encoding module to obtain a first encoding vector; as well as, The first encoding vector is input into the recognition module to recognize text in the image in the second image data set to obtain a first recognition result.

15. A method for training an image recognition model, characterized in that: A training device applied to a cloud, the image recognition model comprises: an encoding module and a recognition module, the encoding module is used to extract the features of a target object from an image to obtain a coding vector of the target object, and the recognition module is used to recognize the target object based on the coding vector of the target object; the method comprises: Acquire a trained encoding module from a local user; the trained encoding module is obtained by training at the local user using a first image data set stored at the local user; The annotated second image data set stored in the cloud is input into an image recognition model including the recognition module and the trained encoding module to train the recognition module to obtain a trained recognition module.

16. The method according to claim 15, characterized in that The training of the recognition model comprises: Extracting features of the target object from images in the second image data set based on the trained encoding module to obtain a first encoding vector of the target object; Inputting the first encoding vector into the recognition module to recognize the target object and obtain a first recognition result; Based on the first recognition result and labels of images in the second image data set, parameters of the recognition module are updated.

17. The method according to claim 15 or 16, characterized in that The images in the first image data set and the images in the second image data set both contain text; The training of the recognition module includes: extracting features of text from images in the second image data set based on the trained encoding module to obtain a first encoding vector; and The first encoding vector is input into the recognition module to recognize text in the image in the second image data set to obtain a first recognition result.

18. An image recognition model training device, characterized in that: The image recognition model includes: a coding module and a recognition module, the coding module is used to extract the characteristics of the target object from the image to obtain the coding vector of the target object, and the recognition module is used to identify the target object based on the coding vector of the target object; the device is located in the cloud, and the device includes: An acquisition module, used for acquiring a trained encoding module from a user locally; the trained encoding module is obtained by training at the user locally using a first image data set stored at the user locally; An input module is used to input the annotated second image data set stored in the cloud into an image recognition model including the recognition module and the trained encoding module to train the recognition module to obtain a trained recognition module.

19. The device according to claim 18, characterized in that The device also includes an update module; wherein, The input module is used to: extract the features of the target object from the image in the second image data set based on the trained encoding module to obtain a first encoding vector of the target object; and input the first encoding vector into the recognition module to recognize the target object and obtain a first recognition result; The updating module is used to update the parameters of the recognition module based on the first recognition result and the labels of the images in the second image data set.

20. The device according to claim 18 or 19, characterized in that The images in the first image data set and the images in the second image data set both contain text, and the input module is used to: Extracting features of text from images in the second image data set based on the trained encoding module to obtain a first encoding vector; The first encoding vector is input into the recognition module to recognize text in the image in the second image data set to obtain a first recognition result.

21. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 15 to 17.

22. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, which, when executed by a computing device cluster, perform the method as claimed in any one of claims 15 to 17.

23. A computer program product comprising instructions, characterized in that When the instructions are executed by a computer device cluster, the computer device cluster is caused to perform the method according to any one of claims 15 to 17.

Images