Image-text recognition method and data processing method for image-text recognition model

Abstract

In an embodiment of the present invention, an image-text recognition method and an image-text recognition model data processing method are proposed. The image-text recognition method includes the steps of receiving an image-text recognition request input by a user, inputting a recognition target image into an image-text recognition model and processing it through the image-text recognition model to obtain Chinese text in the recognition target image, the image-text recognition model being obtained by training a pre-trained model based on Chinese image-text pairs of a plurality of different scenes, the pre-trained model being obtained by pre-training based on a plurality of pre-training images and description text corresponding to each pre-training image, and feeding back the Chinese text to a user. Since the pre-trained model is obtained by pre-training based on a plurality of pre-training images and description text corresponding to each pre-training image, it has image description ability. Furthermore, by training the pre-trained model using Chinese image-text pairs of a plurality of different scenes, the obtained image-text recognition model is able to recognize Chinese text in images, thereby improving the efficiency and accuracy of image-text recognition.

Description

Technical Field

[0001] The present application claims priority from the Chinese patent application filed with the China National Intellectual Property Administration on October 26, 2022, with the application number 202211318775.4 and the title of the invention "Image text recognition method and data processing method for image text recognition model", and the entire content of said Chinese patent application is incorporated into the present application by reference. Embodiments of the present invention relate to the field of computer technology, and in particular to an image text recognition method. One or more embodiments of the present invention further relate to a data processing method for an image text recognition model, a data processing apparatus for an image text recognition model, an image text recognition apparatus, a computing device, a computer-readable storage medium, and a computer program.

Background Art

[0002] With the development of Internet technology, Optical Character Recognition (OCR) technology has been widely applied in various scenarios, for example, recognition of text content in documents, recognition of text content in invoices, and the like. Effectively improving OCR accuracy can help users improve item effects and improve the process efficiency of related items.

[0003] Currently, OCR technology is roughly divided into a character detection part and a character recognition part. Character detection is relatively mature, but for character recognition, compared with traditional document recognition, in some complex scenes, due to factors such as image resolution and diversity of character forms, the accuracy of character recognition still needs to be improved. Therefore, an efficient and accurate image text recognition method is urgently needed as soon as possible.

Summary of the Invention

[0004] In embodiments of the present invention, an image-text recognition method is proposed. In one or more embodiments of the present invention, a data processing method for an image-text recognition model, a data processing device for an image-text recognition model, an image-text recognition device, a computing device, a computer-readable storage medium, and a computer program are proposed to solve one or more of the above-mentioned technical problems.

[0005] According to a first aspect of an embodiment of the present invention, an image text recognition method applicable to a terminal device is proposed, which includes the steps of: receiving an image text recognition request from a user that includes an image to be recognized; inputting the image to be recognized into an image text recognition model, and obtaining Chinese text within the image to be recognized after processing by the image text recognition model, wherein the image text recognition model is obtained by training a pre-trained model based on a plurality of Chinese image text pairs in multiple different scenes, wherein the pre-trained model is obtained by pre-training based on a plurality of pre-training images and descriptive text corresponding to each pre-training image; and providing feedback of the Chinese text to the user.

[0006] According to a second aspect of an embodiment of the present invention, a data processing method for an image-text recognition model applied to a cloud-side device is proposed, which includes the steps of: obtaining a sample set containing Chinese image-text pairs in multiple different scenes, where each Chinese image-text pair contains a sample image and sample Chinese text; inputting the sample images into a pre-trained model to obtain predicted Chinese text for the sample images, where the pre-trained model is obtained by pre-training based on multiple pre-training images and descriptive text corresponding to each pre-training image; training the pre-trained model based on the predicted Chinese text and sample Chinese text, and if a third preset stop condition is reached, obtaining the model parameters of the image-text recognition model obtained during training; and transmitting the model parameters of the image-text recognition model obtained during training to a terminal-side device.

[0007] According to a third aspect of the embodiments of the present invention, an image-text recognition device applicable to a terminal device is proposed, which includes: a receiving module configured to receive an image-text recognition request containing an image to be recognized input from a user; a first acquisition module configured to input the image to be recognized into an image-text recognition model and, after processing by the image-text recognition model, acquire the Chinese text within the image to be recognized; the first acquisition module is obtained by training a pre-trained model based on a plurality of different Chinese image-text pairs in multiple scenes, and the pre-trained model is obtained by pre-training based on a plurality of pre-training images and descriptive text corresponding to each pre-training image; and a feedback module configured to provide feedback of the Chinese text to the user.

[0008] According to a fourth aspect of an embodiment of the present invention, a data processing device for an image-text recognition model applied to a cloud-side device is proposed, which includes: an acquisition module configured to acquire a sample set containing Chinese image-text pairs in multiple different scenes, where each Chinese image-text pair includes a sample image and sample Chinese text; a second acquisition module configured to input the sample images into a pre-trained model to acquire predicted Chinese text for the sample images, where the pre-trained model is obtained by pre-training based on multiple pre-training images and descriptive text corresponding to each pre-training image; a training module configured to train the pre-trained model based on predicted Chinese text and sample Chinese text, and to acquire model parameters of the image-text recognition model obtained during training when a third preset stop condition is reached; and a transmission module configured to transmit the model parameters of the image-text recognition model obtained during training to a terminal-side device.

[0009] According to a fifth aspect of an embodiment of the present invention, a computing device is proposed that includes memory and a processor, wherein the memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions, and when the computer executable instructions are executed by the processor, the steps of the method according to the first or second embodiment described above are realized.

[0010] According to a sixth aspect of the embodiments of the present invention, a computer-readable storage medium is proposed in which computer-executable instructions are stored, and when the instructions are executed by a processor, the steps of the method according to the first or second embodiment described above are realized.

[0011] According to a seventh aspect of the embodiments of the present invention, a computer program is proposed, and when the computer program is executed on a computer, the steps of the method according to the first or second embodiment described above are performed on the computer.

[0012] The above overview is for illustrative purposes only and is not intended to limit the scope in any way. Further aspects, embodiments, and features of the application will be readily apparent by referring to the accompanying drawings and the detailed description below, in addition to the exemplary aspects, embodiments, and features described above. [Brief explanation of the drawing]

[0013] In the attached drawings, the same reference numerals in multiple attached drawings indicate the same or similar parts or elements unless otherwise defined. These attached drawings are not necessarily drawn proportionally. These attached drawings relate only to some embodiments of the disclosure herein and should not be considered limitations to the scope of the application. [Fig. 1] This is a diagram showing the configuration of a data processing system for an image-text recognition model according to one embodiment of the present invention. [Fig. 2] This is a diagram illustrating the configuration of a data processing system for another image-text recognition model according to one embodiment of the present invention. [Fig. 3] This is a flowchart of an image-text recognition method according to one embodiment of the present invention. [Fig. 4] This is a schematic diagram of an image-text recognition interface according to one embodiment of the present invention. [Fig. 5] This is a flowchart illustrating a data processing method for an image-text recognition model applied to a cloud-side device according to one embodiment of the present invention. [Fig. 6] This is a flowchart of an image-text recognition method according to one embodiment of the present invention. [Fig. 7] This is a flowchart of the data processing process for an image-text recognition model according to one embodiment of the present invention. [Fig. 8] This is a flowchart of the processing process using an image-text recognition method according to one embodiment of the present invention. [Fig. 9] This is a schematic diagram of the configuration of an image-text recognition device according to one embodiment of the present invention. [Fig. 10]This is a schematic diagram of the configuration of a data processing device for an image-text recognition model according to one embodiment of the present invention. [Fig. 11] This is a block diagram of the configuration of a computing device according to one embodiment of the present invention. [Modes for carrying out the invention]

[0014] The following description details many specific aspects in order to fully understand the present invention. However, the present invention can be carried out in many other embodiments different from those described herein, and those skilled in the art can make similar disseminations without violating the principles of the present invention, and the present invention is not limited to the specific embodiments disclosed below.

[0015] The terms used in one or more embodiments of the present invention are merely for the purpose of describing a particular embodiment and are not intended to limit one or more embodiments of the present invention. The singular forms “one,” “the said,” and “the said” used in one or more embodiments of the present invention and in the appended claims shall also include the plural form unless the context explicitly defines otherwise. Furthermore, it should be understood that the term “and / or” used in one or more embodiments of the present invention shall include any or all possible combinations relating to one or more related listed items.

[0016] In one or more embodiments of the present invention, various pieces of information may be described using terms such as "first," "second," etc., but it should be understood that this information is not limited to these terms. These terms are merely used to distinguish information from information of the same category. For example, as long as it does not deviate from the scope of one or more embodiments of the present invention, "first" may be referred to as "second," and similarly, "second" may be referred to as "first." This is determined by linguistic context, and for example, the term "when" as used herein may be interpreted as "when," "if," or "in response to a decision."

[0017] First, the noun terms mentioned in one or more embodiments of the present invention will be interpreted.

[0018] Character recognition: it is intended for recognizing character information in an image containing characters to accurately evaluate model effects, and in the embodiments of the present invention, character recognition is also referred to as image text recognition.

[0019] Ultra-large-scale multimodal pre-training: based on training with data of large-scale weakly supervised image texts, a model is trained on large-scale data, thereby obtaining model parameters with outstanding effects, and fine-tuning can be performed on the model in the form of initialization on a pre-trained model with outstanding effects.

[0020] Multi-task training: since data of multiple tasks are unified into a consistent form, the model is trained by using data of multiple tasks at the same time.

[0021] With the development of Internet technology, optical character recognition (OCR, Optical Character Recognition) technology has been widely applied in various scenarios, for example, recognition of text content in documents, recognition of text content in invoices, recognition of handwritten character content, recognition of street view character content, etc. Effective improvement of OCR accuracy can contribute to improvement of item effects by users and improvement of process efficiency of related items.

[0022] Currently, OCR technology is roughly divided into a character detection part and a character recognition part. Character detection is relatively mature, but compared with traditional document recognition for character recognition, in complex scenes such as street views, drawings on networks, and even handwritten character recognition, due to factors such as image resolution and diversity of character forms, the accuracy of character recognition still needs to be improved. Therefore, an efficient and accurate image text recognition method is urgently needed as soon as possible.

[0023] To solve the above-mentioned problems, an embodiment of the present invention proposes an image-text recognition method applicable to a terminal device. Specifically, it receives an image-text recognition request from a user that includes an image to be recognized, inputs the image to be recognized into an image-text recognition model, and after processing by the image-text recognition model, obtains the Chinese text within the image to be recognized. The image-text recognition model is obtained by training a pre-trained model based on multiple pairs of Chinese image-texts in different scenes. The pre-trained model is obtained by pre-training based on multiple pre-training images and corresponding descriptive texts for each pre-training image, and provides the Chinese text as feedback to the user. Because the pre-trained model is obtained by pre-training based on multiple pre-training images and corresponding descriptive texts for each pre-training image, it has image description capabilities. Furthermore, by training the pre-trained model using multiple pairs of Chinese image-texts in different scenes, the resulting image-text recognition model is given the ability to recognize Chinese text within images, thereby improving the efficiency and accuracy of image-text recognition.

[0024] The present invention proposes an image-text recognition method, and further proposes a data processing method for an image-text recognition model, a data processing device for an image-text recognition model, an image-text recognition device, a computing device, and a computer-readable storage medium. The following examples will be described in detail one by one.

[0025] Referring to Figure 1, a configuration diagram of a data processing system for an image-text recognition model according to one embodiment of the present invention is shown. The data processing system for the image-text recognition model includes a cloud-side device and a terminal-side device. The terminal-side device is configured to construct a sample set containing Chinese image-text pairs in multiple different scenes and transmit it to the cloud-side device, where each Chinese image-text pair includes a sample image and sample Chinese text. The cloud-side device is configured to input the sample images into a pre-trained model to obtain predicted Chinese text for the sample images. The pre-trained model is obtained by pre-training based on multiple pre-training images and the corresponding descriptive text for each pre-training image. Furthermore, the pre-trained model is trained based on the predicted Chinese text and sample Chinese text, and when a third preset stop condition is reached, the model parameters of the image-text recognition model obtained during training are acquired.

[0026] In one possible embodiment, the cloud-side device is further configured to send the model parameters of the image-text recognition model obtained during training to the terminal-side device.

[0027] In the technical solution according to an embodiment of the present invention, the terminal device is configured to construct a sample set containing Chinese image-text pairs in multiple different scenes and transmit it to the cloud device, where each Chinese image-text pair includes a sample image and sample Chinese text. The cloud device is configured to input the sample images into a pre-trained model to obtain predicted Chinese text for the sample images. The pre-trained model is obtained by pre-training based on multiple pre-training images and corresponding descriptive text for each pre-training image. The cloud device is further configured to train the pre-trained model based on the predicted Chinese text and sample Chinese text, and when a third preset stop condition is reached, it obtains the model parameters of the image-text recognition model obtained during training. The cloud device transmits the model parameters of the image-text recognition model obtained during training to the terminal device. The pre-trained model is obtained by pre-training based on multiple pre-training images and the corresponding descriptive text for each pre-training image, thus possessing image description capabilities. Furthermore, training the pre-trained model using Chinese image-text pairs in multiple different scenes improves the efficiency and accuracy of model training, giving the image-text recognition model the ability to recognize Chinese text within images. Terminal devices can then directly perform image-text recognition based on model parameters transmitted from cloud-based devices, further improving the efficiency and accuracy of image-text recognition.

[0028] Referring to Figure 2, a configuration diagram of a data processing system for another image-text recognition model according to one embodiment of the present invention is shown. Such a system includes a cloud-side device and a plurality of terminal-side devices. The plurality of terminal-side devices can communicate with each other via the cloud-side device, and in the image-text recognition model data processing scene, the cloud-side device is configured to provide the image-text recognition model data processing service to the plurality of terminal-side devices, and each of the plurality of terminal-side devices can communicate in real time via the cloud-side device as either a transmitting terminal or a receiving terminal.

[0029] Users can interact with cloud-side devices via their terminal devices to receive data sent from other terminal devices and send data to other terminal devices. In the data processing scene for an image-text recognition model, the user releases a data stream to the cloud-side device via their terminal device, the cloud-side device processes the image-text recognition model based on this data stream, and can push the processed image-text recognition model to other terminal devices with established communication.

[0030] Here, a network connection is established between the terminal device and the cloud device. The network is the medium that provides the communication link between the terminal device and the cloud device. The network can include various connection types, such as wired or wireless links, or fiber optic cables. Data transmitted from the terminal device may need to undergo processing such as encoding, transcoding, or compression before being released to the cloud device.

[0031] Examples of terminal devices include web applications such as browsers, apps (Applications), or H5 (HyperText Markup Language 5) applications, lightweight applications (also called applets, which are lightweight applications), or cloud applications. Terminal devices can be obtained based on a software development kit (SDK) that provides the relevant service on the cloud device, such as a Real Time Communication (RTC) SDK development kit. Terminal devices may be deployed on electronic devices and operate depending on the device or some apps within the device. Electronic devices may have a display to support information viewing, and may be personal mobile terminals such as mobile phones, tablets, or personal computers. Electronic devices can usually be deployed with various types of applications, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant communication tools, mailbox devices, and social platform software.

[0032] The cloud-side equipment can include servers that provide various services, such as servers that provide communication services to multiple terminal-side devices, servers for background training that provide support to models used by terminal-side devices, and servers that process data sent from terminal-side devices. The cloud-side equipment may be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server may be a server in a distributed system, or a server with blockchain technology incorporated. The server may be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or a smart cloud computing server or smart cloud host equipped with artificial intelligence technology.

[0033] In addition, while the data processing method for the image-text recognition model according to the embodiment of the present invention is executed by a cloud-side device, in other embodiments of the present invention, the terminal-side device has the same functions as the cloud-side device, and therefore can also execute the data processing method for the image-text recognition model according to the embodiment of the present invention. In other embodiments, the data processing method for the image-text recognition model according to the embodiment of the present invention may be executed jointly by the cloud-side device and the terminal-side device.

[0034] Referring to Figure 3, a flowchart of an image-text recognition method according to one embodiment of the present invention is shown, which includes the following steps.

[0035] Step 302: Receive an image text recognition request from the user that includes the image to be recognized.

[0036] In one or more embodiments of the present invention, the terminal device receives an image-text recognition request input by the user and processes the image to be recognized included in the image-text recognition request to generate Chinese text within the image to be recognized.

[0037] In one possible embodiment, the image to be recognized refers to an image waiting to be subjected to image-text recognition. Examples of images to be recognized include, but are not limited to, Street View images, network drawing images, handwritten images, and document images. They can be specifically selected according to the actual situation, and the embodiments of this invention do not impose any limitations on this.

[0038] In actual applications, the terminal device may either directly obtain the image to be recognized from the image library or receive an image-text recognition request from the user.

[0039] Furthermore, there are various methods by which the terminal device receives image-text recognition requests input by the user, and these can be specifically selected according to the actual situation. The embodiments of this invention do not limit this in any way.

[0040] In one possible embodiment of the present invention, the terminal device can receive an image-text recognition request that includes an image to be recognized, which has been voluntarily entered by the user.

[0041] In another possible embodiment of the present invention, after acquiring an image-text recognition model, the terminal device may send an image acquisition command to the user to indicate that the training of the image-text recognition model is complete and prompt the user to input an image-text recognition request.

[0042] Step 304: The image to be recognized is input into the image-text recognition model, and after processing by the image-text recognition model, the Chinese text within the image to be recognized is obtained. This image-text recognition model is obtained by training a pre-trained model based on multiple pairs of Chinese image-texts in multiple different scenes, and this pre-trained model is obtained by pre-training based on multiple pre-training images and the corresponding descriptive text for each pre-training image.

[0043] In one or more embodiments of the present invention, after receiving an image-text recognition request input by a user, the terminal device can, in one possible embodiment, input the image to be recognized into an image-text recognition model, and, after processing by the image-text recognition model, obtain the Chinese text within the image to be recognized.

[0044] Furthermore, in order to improve the accuracy of the image-text recognition model's recognition of the target image, the terminal device adds pre-set suggestion text when inputting the target image into the image-text recognition model. That is, both the target image and the pre-set suggestion text are input into the image-text recognition model, and after processing by the image-text recognition model, the Chinese text within the target image can be obtained.

[0045] In one possible embodiment, the pre-set presentation text is pre-set presentation text for image text recognition, such as "What are the characters in the image?" or "What characters are contained in the image?". The pre-set presentation text can be specifically selected according to the actual situation, and the embodiments of the present invention do not limit this in any way.

[0046] Step 306: Provide feedback to the user in Chinese.

[0047] In one or more embodiments of the present invention, the terminal device receives an image-text recognition request input from the user, inputs the image to be recognized into an image-text recognition model, processes it with the image-text recognition model to obtain the Chinese text within the image, and then provides the Chinese text back to the user.

[0048] Furthermore, the method of providing feedback to the user in Chinese may involve displaying only the Chinese text, or displaying the image to be recognized and the Chinese text simultaneously. Specifically, the method can be selected according to the actual situation, and the embodiments of this invention do not limit this in any way.

[0049] In the technical solution according to an embodiment of the present invention, an image text recognition request containing an image to be recognized is received from the user, the image to be recognized is input to an image text recognition model, and after processing by the image text recognition model, the Chinese text within the image to be recognized is obtained. The image text recognition model is obtained by training a pre-trained model based on multiple pairs of Chinese image texts in multiple different scenes, and the pre-trained model is obtained by pre-training based on multiple pre-training images and descriptive texts corresponding to each pre-training image, and the Chinese text is fed back to the user. Because the pre-trained model is obtained by pre-training based on multiple pre-training images and descriptive texts corresponding to each pre-training image, it has image description capabilities. Furthermore, by training the pre-trained model using pairs of Chinese image texts in multiple different scenes, the resulting image text recognition model is given the ability to recognize Chinese text within images, thereby improving the efficiency and accuracy of image text recognition.

[0050] In one possible embodiment, the method for training the image-text recognition model described above includes the steps of: obtaining a sample set containing Chinese image-text pairs in multiple different scenes, where each Chinese image-text pair contains a sample image and sample Chinese text; inputting the sample images into a pre-trained model to obtain predicted Chinese text for the sample images; and obtaining an image-text recognition model by training the pre-trained model based on the predicted Chinese text and the sample Chinese text.

[0051] In one or more embodiments of the present invention, a model can be trained using a large number of Chinese image-text pairs to impart image-text recognition capabilities to the image-text recognition model obtained during training. A sample set can be obtained when training of the model begins. In one possible embodiment, the model can be trained using Chinese image-text pairs from multiple different scenes to further enhance the universality of the image-text recognition model obtained during training.

[0052] In one possible embodiment, images containing text can vary, for example, a Street View image containing a store name, or a network drawing containing network characters. Therefore, a Chinese image-text pair in multiple different scenes may include at least two selected from Street View Chinese image-text pairs, network drawing Chinese image-text pairs, document Chinese image-text pairs, and handwritten Chinese image-text pairs. Sample Chinese text refers to the text content corresponding to the Chinese characters contained in the sample image.

[0053] In actual applications, there are various methods for obtaining a sample set, which can be specifically selected depending on the actual situation, and the embodiments of this invention do not limit this in any way. In one possible embodiment, a sample set is constructed by manually inputting a large number of Chinese image-text pairs from different scenes. In another possible embodiment of this invention, a sample set is constructed by reading a large number of Chinese image-text pairs from different scenes from another data acquisition device or database.

[0054] In one possible embodiment, since the multiple sample images in the sample set have varying shapes and unbalanced aspect ratios, if the sample images are directly converted to a matching shape such as a square, the text in the sample images will be severely distorted, affecting the accuracy of image-text recognition. Therefore, by processing the sample images while maintaining their aspect ratio based on the size data of the sample images, it is possible to obtain processed sample images with matching shapes. In other words, after the step of obtaining the sample set described above, the method further includes the steps of obtaining size data of the sample images, determining the aspect ratio of the sample images based on the size data of the sample images, and obtaining processed sample images by processing the sample images based on the aspect ratio of the sample images.

[0055] In one possible embodiment, the size data of the sample image may be specific size values ​​such as an image length of 8 cm and a width of 6 cm, or it may be size data divided by a predetermined size value. For example, if the length of the sample image is 8 cm and the width is 6 cm, and the predetermined size value is 2 cm, then the length of the sample image will be 4 and the width will be 3. Of course, the size data of the sample image also includes pixel values.

[0056] In actual applications, there are various methods for obtaining sample image size data, and these can be selected depending on the specific situation; the embodiments of this invention do not limit this in any way. In one possible embodiment, sample image size data can be received from the user. In another possible embodiment of this invention, sample image size data can also be obtained by measuring the sample image with a size measuring tool.

[0057] Furthermore, after obtaining the size data of the sample image, the aspect ratio of the sample image can be determined based on that size data. For example, if the length of the sample image is 8 cm and the width is 6 cm, the aspect ratio of the sample image will be 8 cm:6 cm = 4:3. If the length of the sample image is 4 and the width is 3, the aspect ratio of the sample image will be 4:3.

[0058] In one possible embodiment, after determining the aspect ratio of the sample image, the sample image is processed based on the aspect ratio of the sample image. There are various methods for obtaining the processed sample image, which can be specifically selected depending on the actual situation, and the embodiments of this invention do not limit this in any way.

[0059] In the technical method according to the embodiment of the present invention, size data of a sample image is acquired, the aspect ratio of the sample image is determined based on the size data of the sample image, and the processed sample image is obtained by processing the sample image based on the aspect ratio of the sample image. Since the size and shape of the sample image are unified on the premise that no deformation of characters in the sample image occurs, the robustness and accuracy of the model can be improved.

[0060] In one possible embodiment, the sample image is converted to a predetermined resolution while maintaining the original aspect ratio of the sample image, its length is randomly determined relative to its width based on the aspect ratio, and the remaining portion is filled in based on edge pixel values ​​to obtain a processed sample image. In other words, the step of obtaining a processed sample image by processing the sample image based on the aspect ratio of the sample image as described above includes the steps of: converting the length of the sample image to a predetermined resolution to determine the converted length of the sample image; adjusting the width of the sample image based on the aspect ratio and converted length to determine the converted width of the sample image; and obtaining a processed sample image by filling in the pixel values ​​of the sample image based on the converted length and converted width of the sample image.

[0061] In the embodiment of the present invention, after determining the converted length and converted width of a sample image, a preset length and width size can be obtained. Furthermore, a processed sample image can be obtained by embedding the pixel values ​​of the sample image according to the converted length, converted width, and preset length and width size.

[0062] Furthermore, when embedding pixel values ​​in a sample image, it is possible to randomly select pixel values. In one possible embodiment, in order to make the processed image more realistic, the edge pixel values ​​of the sample image are recognized, and then the pixel values ​​of the sample image are embedded according to the edge pixel values, thereby eliminating the obvious boundary of the embedding in the embedded image.

[0063] In one embodiment, assuming the sample image has a length of 128, a width of 96, and a preset resolution of 68, the aspect ratio of the sample image can be determined to be 128:96 = 4:3 based on its length and width. The sample image length of 128 is converted to the preset resolution of 68, and the converted length of the sample image is determined to be 68. Based on the aspect ratio of 4:3 and the converted length of 68, the converted width of the sample image can be determined to be 51. Once the converted length and width are determined, the obtained preset length-width size becomes 68*68. Then, by comparing the converted length, converted width, and preset length-width size, it is determined that the converted width needs to be adjusted from 51 to 68. Therefore, by embedding the pixel values ​​of the sample image, a sample image with a processed length-width size of 68*68 can be obtained.

[0064] In the technical method according to the embodiment of the present invention, the length of the sample image is converted to a preset resolution, the converted length of the sample image is determined, the width of the sample image is adjusted based on the aspect ratio and converted length, the converted width of the sample image is determined, and the pixel values ​​of the sample image are embedded based on the converted length and width of the sample image to obtain a processed sample image. Since the size and shape of the sample image are unified on the premise that no deformation of characters in the sample image occurs, the robustness and accuracy of the model can be improved.

[0065] In one or more embodiments of the present invention, after obtaining a sample set, in one possible embodiment, the sample images can be input into a pre-trained model to obtain predicted Chinese text for the sample images, the pre-trained model being obtained by pre-training based on a plurality of pre-training images and descriptive text corresponding to each pre-training image.

[0066] In actual applications, there are various methods for inputting sample images into a pre-trained model to obtain predicted Chinese text for the sample images. These methods can be selected according to the specific situation, and the embodiments of this invention do not limit this in any way.

[0067] In one possible embodiment, the pre-trained model includes a feature extraction layer, an encoder, and a decoder. A sample image is input directly into the pre-trained model, the feature extraction layer in the pre-trained model extracts image features from the sample image, and then the image features are input into the decoder in the pre-trained model. By decoding with the decoder, the predicted Chinese text of the sample image can be obtained.

[0068] In another possible embodiment of the present invention, the pre-trained model includes an encoder and a decoder. Image features of a sample image can be extracted, and then the image features can be input into the pre-trained model and processed by the encoder and decoder to obtain the predicted Chinese text of the sample image. In other words, the step of inputting the above-described sample image into the pre-trained model to obtain the predicted Chinese text of the sample image includes the steps of: extracting image features of a first sample image, the first sample image being any one sample image selected from a plurality of Chinese image-text pairs; and inputting the image features into the pre-trained model to obtain a first predicted Chinese text of the first sample image.

[0069] In one possible embodiment, when the terminal device inputs a sample image into the pre-trained model, it may also input pre-set presentation text. In other words, the terminal device can input both the sample image and the pre-set presentation text into the pre-trained model to obtain the Chinese text of the sample image.

[0070] In one possible embodiment, a sample image and pre-set presentation text are input directly into a pre-trained model. The feature extraction layer within the pre-trained model extracts image features from the sample image and text features from the pre-set presentation text. After concatenating the image and text features, the data is input into a decoder within the pre-trained model. Decoding by the decoder yields the predicted Chinese text for the sample image.

[0071] In another possible embodiment of the present invention, the pre-trained model includes an encoder and a decoder. After extracting image features from a sample image and text features from a pre-set presentation text, the image and text features are input into the pre-trained model and processed by the encoder and decoder to obtain predicted Chinese text for the sample image.

[0072] In the technical method according to the embodiment of the present invention, image features of a first sample image are extracted, and this first sample image is one of several sample images selected from multiple Chinese image-text pairs. The image features are input into a pre-trained model to obtain the first predicted Chinese text for the first sample image. This reduces the amount of data processing required for the pre-trained model and improves the acquisition efficiency of the image-text recognition model.

[0073] In embodiments of the present invention, there are various methods for extracting text features from a pre-set presentation text, and these can be specifically selected according to the actual situation, with no limitations in the embodiments of the present invention. In one possible embodiment of the present invention, text features can be extracted by methods such as one-hot encoding and term frequency-inverse document frequency (TF-IDF). In another possible embodiment of the present invention, text features can also be extracted by models such as long short-term memory recurrent neural networks (LSTM) and BERT.

[0074] There are various methods for extracting image features from sample images, and these can be selected according to the actual situation. The embodiments of this invention do not limit this in any way.

[0075] In one possible embodiment of the present invention, image features can be extracted from an image using methods such as the discrete Fourier transform, geometric parameter method, shape-invariant moment method, and two-dimensional histogram projection method.

[0076] In one embodiment, the two-dimensional histogram projection method involves obtaining the cumulative distribution of black pixels in each matrix in the horizontal and vertical directions of the sample image, projecting the sample image in both the horizontal and vertical directions, then statistically normalizing the values ​​of black pixels in each matrix direction, and finally obtaining a feature vector.

[0077] In another possible embodiment of the present invention, image features of a sample image can be extracted using a pre-configured feature extraction model. In other words, the step of extracting image features of a first sample image as described above includes inputting the first sample image into a pre-configured feature extraction model and determining the image features of the first sample image by processing it with the pre-configured feature extraction model.

[0078] In one possible embodiment, a pre-configured feature extraction model refers to a model with feature extraction capabilities, such as, but not limited to, convolutional neural networks (CNNs), residual neural networks (ResNets), and feature extraction models (ViTs, Vision Transformers).

[0079] In one embodiment, using the VIT model as an example, the first sample image is divided into individual patches, for example, into nine patches. The size of each patch can be specified, for example, 16x16. Each patch is input to an embedding layer (Linear Projection of Flattened Patches), and after passing through this layer, a series of vectors (tokens) are obtained, with each of the nine patches having a corresponding vector. Then, a classification vector* is added before all the vectors, and the dimensions of this classification vector match those of the other nine vectors. Position information 0-9 also needs to be added. All the vectors (patches) are input to a Transformer Encoder, and the output of the encoder is stacked L times. The output of the classification token is then input to a Multilayer Perceptron Head (MLP Head) to obtain the final image features.

[0080] In the technical method according to the embodiment of the present invention, a first sample image is input into a pre-configured feature extraction model, and the image features of the first sample image are determined by processing with the pre-configured feature extraction model. By extracting image features of the sample image using the pre-configured feature extraction model, the efficiency of feature extraction and the accuracy of image features can be improved.

[0081] In one possible embodiment, the step of obtaining an image text recognition model by training the above-described pre-trained model based on predicted Chinese text and sample Chinese text includes the steps of: comparing the predicted Chinese text and sample Chinese text to calculate a text loss value; adjusting the model parameters of the pre-trained model based on the text loss value; and returning to the step of inputting a sample image into the pre-trained model to obtain predicted Chinese text for the sample image; and obtaining an image text recognition model if a first preset stop condition is reached.

[0082] In one possible embodiment of the present invention, the first training stop condition includes the text loss value being less than or equal to a first preset threshold. A sample image can be input into a pre-trained model to obtain the predicted Chinese text of the sample image, and after calculating the text loss value by comparing the predicted Chinese text with the sample Chinese text, the text loss value can be compared with the first preset threshold.

[0083] In one possible embodiment, if the text loss value is greater than a first preset threshold, it indicates that the difference between the predicted Chinese text and the sample Chinese text is large, i.e., the predictive ability of the pre-trained model is poor. At this time, the model parameters of the pre-trained model are adjusted based on the text loss value, and the process returns to the step of inputting a sample image into the pre-trained model to obtain the predicted Chinese text of the sample image. On the other hand, if the text loss value is less than or equal to the first preset threshold, it indicates that the difference between the predicted Chinese text and the sample Chinese text is small, and the first training stop condition has been reached, and a trained image-text recognition model is obtained.

[0084] In another possible embodiment of the present invention, in addition to comparing the text loss value with a first preset threshold, the number of iterations can also be combined to determine whether training on the current pre-trained model is complete.

[0085] In one possible embodiment, if the text loss value is greater than a first preset threshold, the model parameters of the pre-trained model are adjusted, and the process returns to the step of inputting a sample image into the pre-trained model to obtain predicted Chinese text for the sample image, training the pre-trained model continues, and when a first preset number of iterations is reached, the iterations are stopped and the trained image-text recognition model is obtained, the first preset number of iterations can be specifically selected according to the actual situation, and is not limited in any way in the embodiments of the present invention.

[0086] In actual applications, various functions can be used to calculate text loss values, such as cross-entropy loss functions, L1-norm loss functions, maximum loss functions, mean squared error (MSE) loss functions, and logarithmic loss functions. These can be specifically selected depending on the actual situation, and the embodiments of this invention do not limit this in any way.

[0087] In the technical method according to an embodiment of the present invention, a text loss value is calculated by comparing the predicted Chinese text with the sample Chinese text, the model parameters of the pre-trained model are adjusted based on the text loss value, and the process returns to the step of inputting a sample image into the pre-trained model to obtain the predicted Chinese text of the sample image. When a first preset stop condition is reached, an image text recognition model is obtained. By continuously adjusting the model parameters of the pre-trained model, the image text recognition model finally obtained can be made more accurate.

[0088] In one possible embodiment, after acquiring an image-text recognition model, the model can be directly evaluated on multiple downstream tasks. In another possible embodiment, the image-text recognition model can be fine-tuned based on Chinese image-text pairs in a specified scene to improve its performance in that scene. In other words, after the step of acquiring an image-text recognition model by training the aforementioned pre-trained model with predicted Chinese text and sample Chinese text, the process further includes the steps of receiving a training command from a user that includes a specified scene, acquiring multiple Chinese image-text pairs in the specified scene, and acquiring an image-text recognition model for the specified scene by training the image-text recognition model using multiple Chinese image-text pairs in the specified scene.

[0089] In actual applications, there are various methods for receiving training commands sent by the user, and these can be specifically selected depending on the actual situation. The embodiments of this invention do not limit this in any way.

[0090] In one possible embodiment, after acquiring an image-text recognition model, if the user needs to perform secondary training on the model, a training command including a specified scene can be sent directly. The specified scene may include, but is not limited to, Street View, network drawings, handwritten text, or documents.

[0091] In another possible embodiment of the present invention, after acquiring an image-text recognition model, a query command containing multiple scenes can be sent to the user so that the user can decide whether or not to perform secondary training of the image-text recognition model based on specified scenes. If the user decides to perform secondary training of the image-text recognition model, they can click to select the specified scenes. In one possible embodiment, the method by which the user clicks to select a scene may be any one of the following: click, double-click, touch, mouseover, slide, long press, voice control, or shake, and may be specifically selected depending on the actual situation, and the embodiments of the present invention are not limited thereto.

[0092] The user who sends the training command and the user who inputs the image-text recognition request may be the same or different. Upon receiving a training command from a user, multiple Chinese image-text pairs in a specified scene can be acquired. The method for acquiring Chinese image-text pairs in a specified scene may be to screen for Chinese image-text pairs in a specified scene from a sample set, or it may be to acquire Chinese image-text pairs of a specified type again. The method for acquiring multiple Chinese image-text pairs in a specified scene can be specifically selected according to the actual situation, and there are no limitations on this in the embodiments of the present invention.

[0093] In one possible embodiment, the method for obtaining an image-text recognition model in a specified scene by training the image-text recognition model using multiple pairs of Chinese image-text in a specified scene is the same as the method for obtaining an image-text recognition model by training a pre-trained model, and therefore will not be described further in the embodiments of the present invention.

[0094] In the technical proposal according to an embodiment of the present invention, a training command containing a specified scene is received from the user, multiple Chinese image-text pairs in the specified scene are acquired, and an image-text recognition model is trained using the multiple Chinese image-text pairs in the specified scene, thereby acquiring an image-text recognition model for the specified scene. This makes the effect of the image-text recognition model in the specified scene better and improves the accuracy of image-text recognition in the specified scene.

[0095] In one possible embodiment, a method for training a pre-trained model includes the steps of: obtaining a training set comprising a plurality of pre-training image-text pairs, each pre-training image-text pair comprising a pre-training image and a descriptive text corresponding to the pre-training image; extracting a first pre-training image-text pair from the plurality of pre-training image-text pairs, the first pre-training image-text pair being any one selected from the plurality of pre-training image-text pairs; inputting the first pre-training image in the first pre-training image-text pair into an initial model to obtain a first predictive text corresponding to the first pre-training image; comparing the first predictive text with the first descriptive text in the first pre-training image-text pair to calculate a pre-training loss value; adjusting the model parameters of the initial model based on the pre-training loss value, returning to the step of extracting the first pre-training image-text pair from the plurality of pre-training image-text pairs, and obtaining a pre-trained model if a second preset stop condition is reached.

[0096] There are various methods for obtaining training sets, and these can be selected according to the actual situation; the embodiments of this invention do not limit this in any way. In one possible embodiment of this invention, a large number of pre-training image-text pairs are manually entered, Training set This constitutes a training set. In another possible embodiment of the present invention, a large number of pre-training image-text pairs are read from another data acquisition device or database to constitute a training set. In one embodiment, pre-training image-text pairs include, but are not limited to, product images and their corresponding titles, landscape images and their corresponding introductory texts.

[0097] In one possible embodiment of the present invention, the second training stop condition includes the pre-training loss value being less than or equal to a second preset threshold. The first pre-training image in the first pre-training image-text pair can be input to the initial model to obtain the first predicted text corresponding to the first pre-training image, and the pre-training loss value can be calculated by comparing the first predicted text with the first descriptive text in the first pre-training image-text pair, and then the pre-training loss value can be compared with a second preset threshold.

[0098] In one possible embodiment, if the pre-training loss value is greater than a second preset threshold, it indicates that the difference between the first predicted text and the first descriptive text in the first pre-training image-text pair is large, i.e., the initial model's recognition ability is poor. At this point, the model parameters of the initial model are adjusted based on the pre-training loss value, and the process returns to the step of extracting the first pre-training image-text pair from multiple pre-training image-text pairs. On the other hand, if the pre-training loss value is less than or equal to the second preset threshold, it indicates that the difference between the first predicted text and the first descriptive text in the first pre-training image-text pair is small, indicating that the second training stop condition has been met, and a pre-trained model with completed training is obtained.

[0099] In another possible embodiment of the present invention, in addition to comparing the magnitude of the pre-training loss value with a second preset threshold, the number of iterations can also be combined to determine whether training on the current initial model is complete.

[0100] In one possible embodiment, if the pre-training loss is greater than a second preset threshold, the model parameters of the initial model are adjusted, and the process returns to the step of extracting a first pre-training image-text pair from a plurality of pre-training image-text pairs, training continues on the initial model, and when a second preset number of iterations is reached, the iterations are stopped, and the trained pre-trained model is obtained, the second preset number of iterations can be specifically selected according to the actual situation, and is not limited in any way in the embodiments of the present invention.

[0101] In actual applications, various functions can be used to calculate the pre-training loss value, such as the cross-entropy loss function, L1-norm loss function, maximum loss function, mean squared error loss function, and logarithmic loss function. These can be specifically selected according to the actual situation, and the embodiments of this invention do not limit this in any way.

[0102] In the technical method according to an embodiment of the present invention, the first pre-training image in the first pre-training image-text pair is input to the initial model, the first predicted text corresponding to the first pre-training image is obtained, the first predicted text is compared with the first descriptive text in the first pre-training image-text pair to calculate the pre-training loss value, the model parameters of the initial model are adjusted based on the pre-training loss value, and the process returns to the step of extracting the first pre-training image-text pair from multiple pre-training image-text pairs, and if a second preset stop condition is reached, the pre-trained model is obtained. By constantly adjusting the model parameters of the initial model, the pre-trained model finally obtained can be made more accurate.

[0103] In one possible embodiment, in order to better position the text in the image and perform image text recognition, the sample image may have sample position information of sample Chinese text in the sample image, and the step of inputting the above-mentioned sample image and pre-set presentation text into a pre-trained model to obtain the predicted Chinese text of the sample image includes the step of inputting the sample image and pre-set presentation text into a pre-trained model to obtain the predicted Chinese text and predicted position information of the sample image. Furthermore, the step of obtaining an image text recognition model by training the pre-trained model based on the predicted Chinese text and sample Chinese text includes the step of obtaining an image text recognition model by training the pre-trained model based on the predicted Chinese text, predicted position information, sample Chinese text, and sample position information.

[0104] In actual applications, the step of obtaining an image text recognition model by training a pre-trained model based on predicted Chinese text, predicted location information, sample Chinese text, and sample location information includes: calculating a text loss value by comparing the predicted Chinese text and the sample Chinese text; calculating a location loss value by comparing the predicted location information and the sample location information; adjusting the model parameters of the pre-trained model based on the text loss value and the location loss value; inputting a sample image and pre-set presentation text into the pre-trained model and returning to the step of obtaining the predicted Chinese text and predicted location information of the sample image; and obtaining the image text recognition model when a first pre-set stop condition is reached. Note that the specific implementation method of this embodiment is the same as the embodiment shown in Figure 3, so it will not be described further in this embodiment of the present invention.

[0105] Furthermore, when determining the Chinese text in the recognition target image, position information of the Chinese text in the recognition target image may be further determined, the position information may be expressed in Chinese text and fed back to the user together with the Chinese text. Here, the position information, such as the center, the upper left corner, the upper right corner, etc., may be specific coordinates in the recognition target image. If it is specific coordinates, it may include the specific coordinates of the upper left corner, lower left corner, upper right corner, and lower right corner of the Chinese text.

[0106] Referring to FIG. 4, a schematic diagram of an image text recognition interface according to an embodiment of the present invention is shown.

[0107] The image text recognition interface includes an image upload box, a "confirm" controller, a "cancel" controller, and a recognition result display box. When a user uploads a recognition target image to the image upload box, for example, a street view image including the store name "炸鷄店" (fried chicken shop), and clicks and selects the "confirm" controller, a terminal device inputs the recognition target image and preset prompt text "what are the characters in the image" into an image text recognition model, obtains a recognition result after processing by the image text recognition model, and displays "炸鷄店 (position information: center of the recognition target image)" as the recognition result in the recognition result display box.

[0108] In the technical solution according to the embodiment of the present invention, a sample set including Chinese image-text pairs in a plurality of different scenes is acquired, the Chinese image-text pairs include a sample image and sample Chinese text, the sample image has sample position information of the sample Chinese text in the sample image, the sample image and a preset prompt text are input into a pre-training model to acquire predicted Chinese text and predicted position information of the sample image, and the pre-training model is trained based on the predicted Chinese text, the predicted position information, the sample Chinese text and the sample position information, thereby acquiring an image text recognition model. Accordingly, the image text recognition model can predict the position of the sample Chinese text in the sample image, and the accuracy of the image text recognition model can be improved.

[0109] In a possible embodiment, after the terminal device feeds back the Chinese text in the to-be-recognized image to the user, if the user determines that the Chinese text fed back by the terminal device is incorrect, the user can input a revised text corresponding to the Chinese text. For example, the revised text corresponding to the Chinese text "我恨開心" is "我很開心". After the terminal device receives the revised text, it can train the image text recognition model based on the revised text and the to-be-recognized image, thereby acquiring an updated image text recognition model. In other words, after the step of feeding back the above-mentioned Chinese text to the user, the method further includes: receiving the revised text corresponding to the Chinese text input by the user; and training the image text recognition model based on the revised text and the to-be-recognized image to acquire an updated image text recognition model.

[0110] It should be noted that the specific implementation method for acquiring the updated image text recognition model by training the image text recognition model based on the revised text and the to-be-recognized image is the same as the training method for the image text recognition model according to the image text recognition method shown in the above-mentioned FIG. 3, and thus no further description is given in the present invention.

[0111] In the technical proposal according to an embodiment of the present invention, a revised text corresponding to Chinese text input by the user is received, and an updated image-text recognition model is obtained by training the image-text recognition model based on the revised text and the image to be recognized. This enables optimization and adjustment of the image-text recognition model, thereby improving the accuracy of the image-text recognition model.

[0112] Referring to Figure 5, a flowchart of a data processing method for an image-text recognition model applied to a cloud-side device according to one embodiment of the present invention is shown. This method includes the following steps.

[0113] Step 502: The cloud-side device obtains the sample set.

[0114] Here, the sample set includes Chinese image-text pairs in multiple different scenes, and each Chinese image-text pair includes a sample image and sample Chinese text.

[0115] Step 504: The cloud-side device inputs sample images into a pre-trained model to obtain predicted Chinese text for the sample images.

[0116] Here, the pre-trained model is obtained by performing pre-training based on multiple pre-training images and descriptive text corresponding to each pre-training image.

[0117] Step 506: The cloud-side device trains the pre-trained model based on predicted Chinese text and sample Chinese text. When the third pre-set stop condition is reached, it retrieves the model parameters of the image-text recognition model obtained during training.

[0118] Step 508: The cloud-side device sends the model parameters of the image-text recognition model obtained during training to the terminal-side device.

[0119] In actual applications, there are various ways in which the cloud-side device can acquire sample sets, and these can be specifically selected depending on the actual situation, and the embodiments of this invention do not limit this in any way. In one possible embodiment of this invention, the cloud-side device can acquire a sample set from a cloud database that contains Chinese image-text pairs in multiple different scenes. In another possible embodiment of this invention, the cloud-side device can also receive a sample set that has been built and uploaded by the terminal-side device.

[0120] Furthermore, the specific embodiments for realizing steps 502, 504, and 506 are the same as the training method for the image text recognition model related to the image text recognition method shown in Figure 3 above, and therefore will not be described further in this invention. The third preset stop condition includes a third preset threshold and a third preset number of repetitions, and the third preset threshold and the third preset number of repetitions can be specifically selected according to the actual situation, and there are no limitations on this in the embodiments of this invention.

[0121] In one possible embodiment, the cloud-side device acquires the model parameters of the image-text recognition model during training, then transmits the model parameters of the image-text recognition model to multiple terminal-side devices that have established a connection to the cloud-side device, allowing the terminal-side devices to perform image-text recognition using the image-text recognition model.

[0122] In the technical solution according to an embodiment of the present invention, the cloud-side device acquires a sample set containing Chinese image-text pairs in multiple different scenes, each containing a sample image and sample Chinese text. The sample images are input into a pre-trained model to obtain predicted Chinese text for the sample images. This pre-trained model is obtained by pre-training based on multiple pre-training images and corresponding descriptive texts for each pre-training image. The pre-trained model is trained based on the predicted Chinese text and sample Chinese text. When a third preset stop condition is reached, the model parameters of the image-text recognition model obtained during training are acquired and transmitted to the terminal-side device. Since the pre-trained model is obtained by pre-training based on multiple pre-training images and corresponding descriptive texts for each pre-training image, it has image description capabilities. Furthermore, by training the pre-trained model using multiple Chinese image-text pairs in multiple different scenes, the efficiency and accuracy of model training can be improved, and the image-text recognition model can be given the ability to recognize Chinese text within images.

[0123] The image text recognition method according to the present invention will be further explained below, using Figure 6 as an example of its application in Chinese image text recognition. Figure 6 shows a flowchart of an image text recognition method according to one embodiment of the present invention. This method includes the steps of a terminal device acquiring Chinese image text pairs in multiple different scenes, constructing a sample set based on the Chinese image text pairs in multiple different scenes, and transmitting it to a cloud-side device. Here, the Chinese image text pairs include sample images and sample Chinese text, and examples of Chinese image text pairs in multiple different scenes include Chinese image text pairs of Street View, Chinese image text pairs of drawings on a network, Chinese image text pairs of documents, and Chinese image text pairs of handwritten characters.

[0124] The cloud-side device inputs sample images into the pre-trained model to obtain predicted Chinese text for the sample images, trains the pre-trained model based on the predicted Chinese text and the sample Chinese text, and when the first pre-set stop condition is reached, it obtains the model parameters of the image text recognition model obtained during training and also transmits the model parameters of the image text recognition model obtained during training to the terminal-side device. Here, the pre-trained model is obtained by performing pre-training based on multiple pre-training images and the descriptive text corresponding to each pre-training image.

[0125] The terminal device receives an image-text recognition request from the user, inputs the image to be recognized into the image-text recognition model, processes it with the image-text recognition model, extracts the Chinese text within the image, and then provides the Chinese text back to the user. Here, the image-text recognition request includes the image to be recognized.

[0126] Furthermore, the terminal device can obtain the model parameters of the image-text recognition model transmitted from the cloud-side device, and then construct an image-text recognition model based on those model parameters.

[0127] In the technical proposal according to the embodiment of the present invention, the pre-trained model is obtained by pre-training based on multiple pre-training images and descriptive text corresponding to each pre-training image, and therefore possesses image description capabilities. Furthermore, by training the pre-trained model using Chinese image-text pairs in multiple different scenes, the efficiency and accuracy of model training can be improved, giving the image-text recognition model the ability to recognize Chinese text within images, thereby improving the efficiency and accuracy of image-text recognition.

[0128] Referring to Figure 7, a flowchart of the data processing process for an image-text recognition model according to one embodiment of the present invention is shown. This data processing process includes the following: Pre-training based on image descriptions: The data processing method for the image-text recognition model according to an embodiment of the present invention can transform a complex character recognition task into a sequence-to-sequence generation task from image to text. To improve the initial model's text generation and image understanding capabilities, the initial model can be pre-trained with very large datasets. In one possible embodiment, a training set is obtained that includes multiple pre-training image-text pairs, each containing a pre-training image and corresponding descriptive text, and weakly supervised data for Chinese image-text pairs is constructed based on the training set. Since the Transformer model can take images and text as input and output text, weakly supervised data can be fed into the Transformer model for pre-training, giving the Transformer model the ability to generate from image to text and laying the foundation for subsequent image-text recognition. In one embodiment, the loss function during the pre-training process was a cross-entropy loss function, and the number of iterations reached 5 or more rounds, ensuring more thorough pre-training and giving the model strong text generation capabilities.

[0129] Multitask Image-Text Recognition Training: A sample set containing Chinese image-text pairs from multiple different scenes is obtained, with each pair containing a sample image and sample Chinese text. Immediately afterward, multitask training is performed based on a pre-trained model. Here, multitask refers to image-text recognition tasks for Chinese image-text pairs from different scenes. Instead of a randomly initialized model, the model parameters are initialized with the weights of the pre-trained model obtained in the pre-training described above, and the model is trained from a pre-trained model with image description capabilities. In multitask training, the image-text recognition tasks from multiple scenes are unified into a consistent task format, the data is distributed and then mixed, and then multitask training is performed. After the multitask model training is complete, the image-text recognition model is obtained. Furthermore, the image-text recognition model can be directly evaluated on multiple downstream tasks. Here, distributing and mixing the data means dividing the data into many batches to ensure data diversity; for example, 1 million data points are divided into many batches, with each batch containing 128 data points.

[0130] Single-task image-text recognition training: After acquiring an image-text recognition model, further single-task fine-tuning can be performed to improve its performance in specific scenes. The training method is the same as multi-task training, and only Chinese image-text pairs in a specified scene are used as data. Compared to single-task fine-tuning, training using both multi-task and single-task methods can effectively enhance the generalization of the model and prevent overfitting.

[0131] In the embodiments of the present invention, a two-stage image-text recognition technology consisting of multitask + single-task configurations driven by ultra-large-scale multi-mode pre-training, i.e., a multi-mode generative image-text recognition model, is proposed. In the embodiments of the present invention, the model configuration is greatly simplified, and the complex character recognition portion is converted into an image-based text description model. Furthermore, in order to improve the image description capability of the model itself and provide a good model foundation for subsequent character recognition, the embodiments of the present invention first widely collect a large amount of weakly trained image-text data from a Chinese network and perform pre-training based on image description. Based on this, the embodiments of the present invention design a multitask + single-task character recognition training method, generating a multitask model by performing unified training of character recognition in multiple scenes, and immediately thereafter fine-tuning the downstream task for character recognition in single scenes.

[0132] Referring to Table 1, a comparison table of model recognition capabilities, the recognition capabilities of the image-text recognition model trained in the embodiment of the present invention compared to other models in image recognition scenes of Street View, network drawings, documents, and handwritten text. Other models include spatial autoregressive models, transformation recognition models, basic models based on OCR data training, and large-scale models based on OCR data training. The comparison table of model recognition capabilities shown in Table 1 is as follows:

[0133] [Table 1] As is clear from Table 1 above, the image-text recognition model trained in the embodiment of the present invention shows a significant improvement in model effectiveness, with an absolute value 6.5 higher for Street View representations, a 4.9 higher value for network drawing scenes, and an average value 3.2 higher. However, when calculating the average value, the amount of data in the dataset must be taken into consideration.

[0134] Referring to FIG. 8, there is shown a flowchart of a data processing process of an image text recognition method according to an embodiment of the present invention. The method includes the following steps: acquiring a to-be-recognized image containing the handwritten Chinese text "我喜歡這黄昏" (meaning "I like this dusk"), extracting image features of the to-be-recognized image and text features of a preset prompt text "what are the characters in the image", and inputting the image features and the text features into an encoder of an image text recognition model to obtain " <s>Obtain the encoded representation of "我喜歡這黄昏", input the encoded representation into the decoder of an image text recognition model to obtain the image text recognition result "我喜歡這黄昏< / s> ", wherein <s>is a start mark,< / s> is an end mark.

[0135] In a possible embodiment, a visual access device can be used to extract image features of the to-be-recognized image. The visual access device processes the to-be-recognized image, specifically converts it into a low-dimensional feature map, further converts it into image features, merges the image features with text features converted by a character embedding layer, and inputs the merged features into the encoder of the image text recognition model. The visual access device selects different visual base models according to different model scales. In one embodiment, a base model with 180 million parameters can use parameters of the top three layers of ResNet-101, and a large model with 470 million parameters can use parameters of the top nine layers of the ViT model. Here, the base model includes 6 encoding layers and 6 decoding layers, and the large model includes 12 encoding layers and 12 decoding layers.

[0136] The image-text recognition model's encoder includes multiple Transformer layers, with each Transformer layer primarily consisting of a Headscale Multi-Head Attention (Headscale MHA) and a Fully Connected Layer (FC). To improve training stability, Layer Norm (LN) layers are added before and after the Headscale Multi-Head Attention layer, before the Fully Connected Layer, and after the first Fully Connected Layer. Furthermore, residual connections are added to both the Headscale Multi-Head Attention and Fully Connected Layers. The image-text recognition model's decoder also includes multiple Transformer layers, with each layer consisting of a Headscale Multi-Head Attention layer, a Cross-Attention layer, and a Fully Connected Layer. The Headscale Multi-Head Attention layer and the Fully Connected Layer are identical, and the Cross-Attention layer is responsible for establishing the relationship between the decoder and encoder. When comparing the queries, keys, and values ​​of the Headscale Multi-Head Attention layer, they are all linear transformations represented by the inputs of the current layer. The queries of the Cross-Attention layer are linear transformations represented by the inputs of the current layer, while the keys and values ​​are linear transformations output from the encoder. In one embodiment, the decoder is trained using a teacher-forced method during the training phase and uses a beam search method during the measurement phase.

[0137] The present invention further proposes an embodiment of an image text recognition device corresponding to the embodiment of the image text recognition method described above, and Figure 9 shows a schematic diagram of the configuration of an image text recognition device according to one embodiment of the present invention. As shown in Figure 9, such a device is applied to a terminal-side device and includes a receiving module 902 configured to receive an image text recognition request containing an image to be recognized, input by a user; a first acquisition module 904 configured to input the image to be recognized into an image text recognition model and acquire the Chinese text within the image to be recognized after processing by the image text recognition model, wherein the image text recognition model is obtained by training a pre-trained model based on a plurality of different scenes of Chinese image text pairs, wherein the pre-trained model is obtained by pre-training based on a plurality of pre-training images and descriptive text corresponding to each pre-training image; and a feedback module 906 configured to provide feedback of the Chinese text to the user.

[0138] In one possible embodiment, such a device further includes a training module for an image-text recognition model, which is configured to acquire a sample set containing Chinese image-text pairs in multiple different scenes, each containing a sample image and sample Chinese text, input the sample images into a pre-trained model to obtain predicted Chinese text for the sample images, and to acquire an image-text recognition model by training the pre-trained model based on the predicted Chinese text and the sample Chinese text.

[0139] In one possible embodiment, examples of Chinese image-text pairs in multiple different scenes include Chinese image-text pairs for Street View, Chinese image-text pairs for drawings on a network, Chinese image-text pairs for documents, and Chinese image-text pairs for handwritten characters.

[0140] In one possible embodiment, such device further includes a specified scene training module configured to receive a training command from a user that includes a specified scene, acquire a plurality of Chinese image-text pairs in the specified scene, and acquire an image-text recognition model in the specified scene by training the image-text recognition model using the plurality of Chinese image-text pairs in the specified scene.

[0141] In one possible embodiment, such apparatus further includes a processing module configured to acquire size data of a sample image, determine the aspect ratio of the sample image based on the size data of the sample image, and process the sample image based on the aspect ratio of the sample image to obtain a processed sample image.

[0142] In one possible embodiment, the processing module is further configured to obtain a processed sample image by converting the length of the sample image to a preset resolution, determining the converted length of the sample image, adjusting the width of the sample image based on the aspect ratio and converted length, determining the converted width of the sample image, and embedding the pixel values ​​of the sample image based on the converted length and converted width of the sample image.

[0143] In one possible embodiment, the image text recognition model training module is configured to further calculate a text loss value by comparing the predicted Chinese text with the sample Chinese text, adjust the model parameters of the pre-trained model based on the text loss value, and then return to the step of inputting a sample image into the pre-trained model to obtain the predicted Chinese text of the sample image, and if a first preset stop condition is reached, obtain the image text recognition model.

[0144] In one possible embodiment, the training module for the image-text recognition model is further configured to extract image features from a first sample image, which is one of several sample images selected from a plurality of Chinese image-text pairs, and to input the image and text features into a pre-trained model to obtain a first predicted Chinese text for the first sample image.

[0145] In one possible embodiment, such device further includes a pre-training module which acquires a training set containing a plurality of pre-training image-text pairs, each containing a pre-training image and a corresponding descriptive text, extracts a first pre-training image-text pair from the plurality of pre-training image-text pairs, the first pre-training image-text pair being one of the plurality of pre-training image-text pairs, inputs the first pre-training image in the first pre-training image-text pair into an initial model to acquire a first predictive text corresponding to the first pre-training image, calculates a pre-training loss value by comparing the first predictive text with the first descriptive text in the first pre-training image-text pair, adjusts the model parameters of the initial model based on the pre-training loss value, and returns to the step of extracting the first pre-training image-text pair from the plurality of pre-training image-text pairs, and is configured to acquire a pre-trained model when a second preset stop condition is reached.

[0146] In one possible embodiment, the sample image has sample location information for sample Chinese text in the sample image, and the image text recognition model training module is further configured to input the sample image into a pre-trained model to obtain predicted Chinese text and predicted location information for the sample image, and to obtain an image text recognition model by training the pre-trained model based on the predicted Chinese text, predicted location information, sample Chinese text, and sample location information.

[0147] In one possible embodiment, such device further includes a training update module configured to receive a revised text corresponding to the Chinese text input by the user, and to obtain an updated image text recognition model by training the image text recognition model based on the revised text and the image to be recognized.

[0148] In the technical proposal according to an embodiment of the present invention, an image text recognition request containing an image to be recognized is received from the user, the image to be recognized is input to an image text recognition model, and after processing by the image text recognition model, the Chinese text within the image to be recognized is obtained. The image text recognition model is obtained by training a pre-trained model based on multiple pairs of Chinese image texts in multiple different scenes, and the pre-trained model is obtained by pre-training based on multiple pre-training images and the corresponding descriptive text for each pre-training image, and the Chinese text is also fed back to the user. Because the pre-trained model is obtained by pre-training based on multiple pre-training images and the corresponding descriptive text for each pre-training image, it has image description capabilities. Furthermore, by training the pre-trained model using multiple pairs of Chinese image texts in multiple different scenes, the resulting image text recognition model is given the ability to recognize Chinese text within images, thereby improving the efficiency and accuracy of image text recognition.

[0149] The above is an exemplary technical proposal for an image-text recognition device according to this embodiment. Since this technical proposal for an image-text recognition device belongs to the same concept as the technical proposal for the image-text recognition method described above, details not described in detail in this technical proposal for an image-text recognition device can be referred to in the description of the technical proposal for the image-text recognition method described above.

[0150] The present invention further proposes an embodiment of a data processing device for an image text recognition model corresponding to the embodiment of the data processing method for the image text recognition model described above, and Figure 10 shows a schematic diagram of the configuration of a data processing device for an image text recognition model according to one embodiment of the present invention. As shown in Figure 10, such a device is applied to a cloud-side device and includes an acquisition module 1002 configured to acquire a sample set containing Chinese image text pairs in multiple different scenes, wherein the Chinese image text pair contains a sample image and sample Chinese text; a second acquisition module 1004 configured to input the sample image into a pre-trained model and acquire the predicted Chinese text of the sample image, wherein the pre-trained model is obtained by performing pre-training based on multiple pre-training images and descriptive text corresponding to each pre-training image; a training module 1006 configured to train the pre-trained model based on the predicted Chinese text and sample Chinese text, and when a third preset stop condition is reached, acquire the model parameters of the image text recognition model obtained during training; and a transmission module 008 configured to transmit the model parameters of the image text recognition model obtained during training to a terminal-side device.

[0151] In the technical solution according to an embodiment of the present invention, the cloud-side device acquires a sample set containing Chinese image-text pairs in multiple different scenes, each containing a sample image and sample Chinese text. The sample images are input into a pre-trained model to obtain predicted Chinese text for the sample images. This pre-trained model is obtained by pre-training based on multiple pre-training images and corresponding descriptive texts for each pre-training image. The pre-trained model is trained based on the predicted Chinese text and sample Chinese text. When a third preset stop condition is reached, the model parameters of the image-text recognition model obtained during training are acquired and transmitted to the terminal-side device. Since the pre-trained model is obtained by pre-training based on multiple pre-training images and corresponding descriptive texts for each pre-training image, it has image description capabilities. Furthermore, by training the pre-trained model using multiple Chinese image-text pairs in multiple different scenes, the efficiency and accuracy of model training can be improved, and the image-text recognition model can be given the ability to recognize Chinese text within images.

[0152] The above is an exemplary technical proposal for a data processing device for an image-text recognition model according to this embodiment. Since this technical proposal for a data processing device for an image-text recognition model belongs to the same concept as the technical proposal for a data processing method for an image-text recognition model described above, details not described in detail in the technical proposal for a data processing device for an image-text recognition model can be referred to in the description of the technical proposal for a data processing method for an image-text recognition model described above.

[0153] Figure 11 shows a block diagram of a computing device according to one embodiment of the present invention. Components of the computing device 1100 include, but are not limited to, a memory 1110 and a processor 1120. The processor 1120 is connected to the memory 1110 via a bus 1130, and the database 1150 is for storing data.

[0154] The computing device 1100 further includes an access device 1140 that enables the computing device 1100 to communicate over one or more networks 1160. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 1140 includes one or more selected from any type of wired or wireless network interface (e.g., a Network Interface Card (NIC)), such as a wireless interface for IEEE 802.11 Wireless Local Area Networks (WLAN), a World Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and the like.

[0155] In one embodiment of the present invention, the above-described components of the computing device 1100 can be connected to each other, for example, via a bus, with other components not shown in Figure 11. It should be understood that the configuration block diagram of the computing device shown in Figure 11 is for illustrative purposes only and does not limit the scope of the present invention. Those skilled in the art can add or replace other components as needed.

[0156] The computing device 1100 may be any type of static or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.), or other types of mobile devices, or static computing devices such as desktop computers or personal computers (PCs). The computing device 1100 may also be a mobile or static server.

[0157] Here, the processor 1120 is configured to execute computer executable instructions, and when these instructions are executed by the processor, the steps of the data processing method or image text recognition method of the image text recognition model described above are realized.

[0158] The above is an exemplary technical proposal for a computing device according to this embodiment. Since this computing device technical proposal belongs to the same concept as the technical proposals for the image-text recognition model's data processing method and image-text recognition method described above, details not described in detail in the computing device technical proposal can be referred to in the descriptions of the image-text recognition model's data processing method and image-text recognition method described above.

[0159] In one embodiment of the present invention, a computer-readable storage medium is proposed which stores computer-executable instructions, and when these computer-executable instructions are executed by a processor, the data processing method or image-text recognition method steps of the image-text recognition model described above are realized.

[0160] The above is an exemplary technical proposal for a computer-readable storage medium according to this embodiment. Since this technical proposal for the storage medium belongs to the same concept as the technical proposals for the data processing method and image-text recognition method of the image-text recognition model described above, details not described in detail in the technical proposal for the storage medium can be referred to in the descriptions of the data processing method and image-text recognition method described above.

[0161] In one embodiment of the present invention, a computer program is further proposed, and when the computer program is executed on a computer, the steps of the data processing method or image text recognition method of the image text recognition model described above are executed on the computer.

[0162] The above is an exemplary technical proposal for a computer program according to this embodiment. Since this technical proposal for the computer program belongs to the same concept as the technical proposals for the data processing method and image text recognition method of the image text recognition model described above, details not described in detail in this technical proposal for the computer program can be referenced from the descriptions in the aforementioned technical proposals for the data processing method and image text recognition method.

[0163] Specific embodiments of the present invention have been described above. Other embodiments are within the scope of the appended claims. In some cases, the operations or steps described in the claims may be performed in a different order than those in the embodiments, and the desired results can still be achieved. Furthermore, the processes depicted in the drawings do not necessarily require the specific order or sequence shown to achieve the desired results. In some embodiments, multitasking and parallel processing may be possible or advantageous.

[0164] The computer instruction includes computer program code, which may be in source code form, object code form, executable file form, or some intermediate form. The computer-readable medium may be any entity or device that can carry the computer program code, recording media, U disks, removable hard disks, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media.

[0165] Furthermore, although the embodiments of each method described above are expressed as a combination of a series of operations for the sake of convenience, the embodiments of the present invention are not limited to the described order of operations, because, according to the embodiments of the present invention, several steps can be performed in other orders or simultaneously, as should be understood by those skilled in the art. Next, all embodiments described herein belong to preferred embodiments, and the operations and modules mentioned are not necessarily essential to embodiments of the present invention, as should also be understood by those skilled in the art.

[0166] In the embodiments described above, the focus of each embodiment is explained, and for parts not described in detail in one embodiment, you can refer to the relevant explanations in other embodiments.

[0167] The preferred embodiments of the present invention disclosed above are for the purpose of assisting in the explanation of the invention. The selectable embodiments do not describe all details in detail, nor do they limit the invention to merely the specific embodiments described. Needless to say, many modifications and changes can be made according to the embodiments of the present invention. The purpose of selecting and specifically describing these embodiments is to enable those skilled in the art to better understand and utilize the invention by better interpreting the principles and practical applications of the embodiments of the present invention. The present invention is limited only to the claims and all their equivalents.

Claims

1. A computer-based image text recognition method applied to a terminal device, The computer receives an image text recognition request from a user that includes an image to be recognized. The steps include: the computer inputting the image to be recognized into an image-text recognition model, and obtaining the Chinese text within the image to be recognized after processing by the image-text recognition model, wherein the image-text recognition model is obtained by training a pre-trained model based on multiple pairs of Chinese image-texts in multiple different scenes, and the pre-trained model is obtained by pre-training based on multiple pre-training images and descriptive texts corresponding to each pre-training image; The computer provides the Chinese text as feedback to the user, Includes, The training method for the aforementioned image-text recognition model is: The steps include: the computer obtaining a sample set containing the Chinese image-text pairs in a plurality of different scenes, wherein the Chinese image-text pairs include a sample image and sample Chinese text, and the sample image has sample position information of the sample Chinese text in the sample image; The computer inputs the sample image into a pre-trained model to obtain the predicted Chinese text and predicted location information of the sample image. The computer acquires the image text recognition model by training the pre-trained model based on the predicted Chinese text, the predicted location information, the sample Chinese text, and the sample location information. including, Image text recognition method.

2. The aforementioned Chinese image-text pairs in multiple different scenes include Chinese image-text pairs from Street View, Chinese image-text pairs from drawings on the network, Chinese image-text pairs from documents, and handwritten Chinese image-text pairs. The image text recognition method according to claim 1.

3. After the step of obtaining an image text recognition model by training the aforementioned pre-trained model based on the predicted Chinese text and the sample Chinese text, The computer receives a training command from a user that includes a specified scene, The computer obtains multiple Chinese image-text pairs in the specified scene, The computer obtains an image-text recognition model for the specified scene by training the image-text recognition model using multiple pairs of Chinese image-texts in the specified scene. Further including, The image text recognition method according to claim 1.

4. After the step of obtaining the sample set, The computer obtains the size data of the sample image, The computer performs the steps of determining the aspect ratio of the sample image based on the size data of the sample image, The computer processes the sample image based on the aspect ratio of the sample image to obtain a processed sample image; Further including, The image text recognition method according to claim 1.

5. The step of obtaining a processed sample image by processing the sample image based on the aspect ratio of the sample image is: The computer performs the steps of converting the length of the sample image to a predetermined resolution and determining the converted length of the sample image, The computer adjusts the width of the sample image based on the aspect ratio and the converted length, and determines the converted width of the sample image. The computer obtains a processed sample image by embedding pixel values ​​in the sample image based on the converted length and converted width of the sample image. including, The image text recognition method according to claim 4.

6. The step of obtaining an image text recognition model by training the aforementioned pre-trained model based on the predicted Chinese text and the sample Chinese text is as follows: The computer performs the steps of comparing the predicted Chinese text with the sample Chinese text to calculate a text loss value, The computer adjusts the model parameters of the pre-trained model based on the text loss value, and then returns to the step of inputting the sample image into the pre-trained model to obtain the predicted Chinese text of the sample image. If the first preset stop condition is reached, the computer then performs the step of obtaining the image text recognition model. including, The image text recognition method according to claim 1.

7. The step of inputting the aforementioned sample image into a pre-trained model to obtain the predicted Chinese text for the sample image is: The computer performs the steps of: extracting image features of a first sample image, which is one of several sample images selected from multiple Chinese image-text pairs; The computer inputs the image features into the pre-trained model to obtain the first predicted Chinese text for the first sample image. including, The image text recognition method according to claim 1.

8. The training method for the aforementioned pre-trained model is: The steps include: the computer obtaining a training set which includes a plurality of pre-training image-text pairs, wherein each pre-training image-text pair includes a pre-training image and a descriptive text corresponding to the pre-training image; The computer extracts a first pre-training image-text pair from the plurality of pre-training image-text pairs, wherein the first pre-training image-text pair is any one selected from the plurality of pre-training image-text pairs. The computer inputs the first pre-training image from the first pre-training image-text pair into the initial model to obtain the first predicted text corresponding to the first pre-training image. The computer calculates a pre-training loss value by comparing the first predicted text with the first descriptive text in the first pre-training image text pair. The computer adjusts the model parameters of the initial model based on the pre-training loss value, and returns to the step of extracting a first pre-training image-text pair from the plurality of pre-training image-text pairs, and if a second preset stop condition is reached, the computer obtains a pre-trained model. including, The image text recognition method according to claim 1.

9. After the step of providing feedback of the aforementioned Chinese text to the user, The computer receives a revised text corresponding to the Chinese text input by the user. The computer further includes the step of obtaining an updated image-text recognition model by training the image-text recognition model based on the revised text and the image to be recognized. The image text recognition method according to claim 1.

10. A method for processing data of an image-text recognition model applied to a cloud-side device, performed by a computer, The steps include: the computer obtaining a sample set containing Chinese image-text pairs in multiple different scenes, wherein each Chinese image-text pair includes a sample image and sample Chinese text, and the sample image has sample position information of the sample Chinese text in the sample image; The steps include: the computer inputting the sample image into a pre-trained model to obtain predicted Chinese text and predicted location information for the sample image, wherein the pre-trained model is obtained by performing pre-training based on a plurality of pre-training images and descriptive text corresponding to each pre-training image; The computer performs training on the pre-trained model based on the predicted Chinese text, the predicted location information, the sample Chinese text, and the sample location information, and when a third preset stop condition is reached, it obtains the model parameters of the image text recognition model obtained during training. The computer transmits the model parameters of the image-text recognition model obtained in the training to the terminal device. including, Data processing methods for image-text recognition models.

11. A computing device including memory and a processor, The memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions. When the computer executable instruction is executed by the processor, a step of the method according to any one of claims 1 to 9 or claim 10 is realized. Computing device.

12. A computer-readable storage medium in which computer executable instructions are stored, When the computer executable instruction is executed by the processor, a step of the method according to any one of claims 1 to 9 or claim 10 is realized. A computer-readable storage medium.

Images