Information extraction model training and information extraction method, device, and medium

Abstract

The embodiments of the present application provide a training method, device, and medium for an information extraction model, and relate to the fields of computer and artificial intelligence technology. The method comprises: obtaining a sample image, wherein the sample image comprises at least one sample information unit; obtaining a model to be trained, wherein the model to be trained comprises an encoder model and at least two decoder models; training the model to be trained through the sample image to obtain an information extraction model, wherein each decoder model learns the decoding results of part or all of the decoder models except itself during the training process. The technical solution of the embodiments of the present application can improve the accuracy of information extraction by the information extraction model.

Description

Technical Field

[0001] The present application relates to the field of computer and artificial intelligence technology, and more specifically, to a training method, device, and medium for information extraction model. Background Art

[0002] In information extraction scenarios, such as extracting information from images (e.g., extracting formulas or text from images), it is usually trained based on an encoder-decoder single-branch model structure, and then extracting information from images based on the trained model. However, this model structure cannot better learn the features of images, especially formula images, and often incorrectly extracts image features. Based on this, how to improve the accuracy of information extraction models in information extraction is a technical problem that needs to be solved urgently. Summary of the invention

[0003] The embodiments of the present application provide a method, device, computer program product or computer program, and computer-readable medium for training an information extraction model, thereby improving the accuracy of information extraction by the information extraction model at least to a certain extent.

[0004] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0005] According to one aspect of an embodiment of the present application, a method for training an information extraction model is provided, the method comprising: obtaining a sample image, the sample image comprising at least one sample information unit; obtaining a model to be trained, the model to be trained comprising an encoder model and at least two decoder models; training the model to be trained using the sample image to obtain an information extraction model, wherein each decoder model learns the decoding results of part or all of the decoder models except itself during the training process.

[0006] According to one aspect of an embodiment of the present application, a training device for an information extraction model is provided, the device comprising: a first acquisition unit, used to acquire a sample image, the sample image comprising at least one sample information unit; a second acquisition unit, used to acquire a model to be trained, the model to be trained comprising an encoder model and at least two decoder models; a training unit, used to train the model to be trained using the sample image to obtain an information extraction model, wherein each decoder model learns a decoding result of part or all of the decoder models except itself during the training process.

[0007] In some embodiments of the present application, based on the aforementioned scheme, the first acquisition unit is configured to: acquire at least one frame of original sample image; adjust the height of each frame of original sample image to a preset image height, or adjust the width of each frame of original sample image to a preset image width; adjust the width of the original sample image or the height of the original sample image according to the adjustment ratio of the height or the adjustment ratio of the width to obtain a preprocessed sample image; select a predetermined number of images from the preprocessed sample image as the sample image.

[0008] In some embodiments of the present application, based on the aforementioned scheme, the first acquisition unit is further configured to: sort the preprocessed sample images according to their height or width; and select a predetermined number of images that are sorted and connected from the preprocessed sample images as the sample images.

[0009] In some embodiments of the present application, based on the aforementioned scheme, the encoder model includes a densely connected convolutional network model.

[0010] In some embodiments of the present application, based on the aforementioned scheme, the training unit is configured to: encode the sample image through the encoder model to obtain sample encoding feature data; decode the sample encoding feature data through the at least two decoder models respectively to obtain at least two groups of sample feature information, wherein each group of sample feature information includes feature information for each sample information unit in the sample image; trigger each of the at least two decoder models to learn part or all of the sample feature information decoded by the decoder model except itself; determine a decoder model as the target decoder model among the at least two decoder models to obtain the information extraction model composed of the encoder model and the target decoder model.

[0011] In some embodiments of the present application, based on the aforementioned scheme, the training unit is further configured to: based on the sample coding feature data, determine the sample positioning information of each sample information unit in the sample image according to different arrangement directions of the sample information units in the sample image through the at least two decoder models; based on the sample positioning information corresponding to the at least two decoder models, respectively decode the sample coding feature data to obtain the at least two groups of sample feature information.

[0012] In some embodiments of the present application, based on the aforementioned scheme, the at least two decoder models include a first decoder model and a second decoder model, and the training unit is further configured to: determine the sample positioning information of each sample information unit in the sample image according to a first arrangement direction of the sample information units in the sample image by the first decoder model; determine the sample positioning information of each sample information unit in the sample image according to a second arrangement direction of the sample information units in the sample image by the second decoder model, and the second arrangement direction is the opposite direction of the first arrangement direction.

[0013] In some embodiments of the present application, based on the aforementioned scheme, the decoder model includes at least two positioning convolution kernels of different scales, and the device also includes: a determination unit, which is used to determine the sample positioning information of the target sample information unit in the sample image for each target sample information unit in the sample image in the following manner: obtaining historical positioning information of the sample information unit for which feature information extraction has been completed in the sample image; performing convolution processing on the historical positioning information respectively through the at least two positioning convolution kernels of different scales to obtain at least two positioning reference information, and the at least two positioning reference information are used to respectively focus on the positioning information of sample information units of different sizes; based on the at least two positioning reference information, determining the sample positioning information of the target sample information unit for which feature information extraction has not been completed in the sample image.

[0014] In some embodiments of the present application, based on the aforementioned scheme, the sample image is also matched with a sample label, and the sample label is used to characterize the real feature information of at least one sample information unit in the sample image, and the training unit is also configured to: before determining a decoder model as the target decoder model among the at least two decoder models, based on the sample label, determine the error information existing in the sample feature information decoded by the at least two decoder models respectively; based on the error information, reversely update the model parameters in the corresponding decoder model through a preset loss function.

[0015] In some embodiments of the present application, based on the aforementioned scheme, the training unit is further configured to: train the model to be trained through the sample images according to a predetermined number of training times to obtain the information extraction reference model of the predetermined number of training times; and average the information extraction reference model of the predetermined number of training times to obtain the information extraction model.

[0016] According to one aspect of an embodiment of the present application, an information extraction method is provided, the method comprising: obtaining a target image, the target image comprising at least one information unit; obtaining an information extraction model obtained by a training method of the information extraction model as described in the above embodiment; and extracting feature information of at least one information unit in the target image through the information extraction model.

[0017] According to one aspect of an embodiment of the present application, an information extraction device is provided, comprising: a third acquisition unit, used to acquire a target image, wherein the target image includes at least one information unit; a fourth acquisition unit, used to acquire an information extraction model obtained by a training method for the information extraction model as described in the above embodiment; and an extraction unit, used to extract feature information of at least one information unit in the target image through the information extraction model.

[0018] In some embodiments of the present application, based on the aforementioned scheme, the information unit includes a character unit, and the at least one information unit constitutes one or more formulas containing the character unit. The information extraction device also includes an editing unit, which is used to edit one or more formulas in the target image to a formula editing area based on the feature information of at least one information unit in the target image after extracting the feature information of at least one information unit in the target image through the information extraction model.

[0019] According to one aspect of an embodiment of the present application, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the training method of the information extraction model as described in the above embodiment, and the information extraction method.

[0020] According to one aspect of an embodiment of the present application, a training device for an information extraction model is also provided, characterized in that it includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing the training method of the information extraction model as described in the above embodiments.

[0021] According to one aspect of an embodiment of the present application, an information extraction device is also provided, characterized in that it includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing the information extraction method described in the above embodiments.

[0022] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which at least one program code is stored. The at least one program code is loaded and executed by a processor to implement the operations performed by the training method of the information extraction model described in the above embodiments, and the operations performed by the information extraction method.

[0023] In the technical solutions provided in some embodiments of the present application, an information extraction model can be trained by a model to be trained including an encoder model and at least two decoder models. Since the information extraction model is obtained by training the model to be trained including an encoder model and at least two decoder models, and each decoder model learns the decoding results of part or all of the decoder models other than itself during the training process, each decoder model can learn the feature information of the information unit learned by part or all of the decoder models other than itself in addition to the feature information of the information unit learned through its own training, so that the trained information extraction model can simultaneously pay attention to and make full use of the feature information of the information unit in the image at different angles, thereby ensuring that the information extraction model has a high information extraction capability. Therefore, the accuracy of information extraction by the information extraction model can be improved.

[0024] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0026] Figure 1 A schematic diagram showing an exemplary system architecture to which the technical solution of the embodiments of the present application can be applied;

[0027] Figure 2 A flowchart of a method for training an information extraction model according to an embodiment of the present application is shown;

[0028] Figure 3 A detailed flow chart of acquiring a sample image according to an embodiment of the present application is shown;

[0029] Figure 4A detailed flow chart of selecting a predetermined number of images from the pre-processed sample images as the sample images according to one embodiment of the present application is shown;

[0030] Figure 5 A detailed flow chart of training the model to be trained by using the sample image according to an embodiment of the present application is shown;

[0031] Figure 6 A detailed flow chart of decoding the sample encoding feature data by respectively using the at least two decoder models according to an embodiment of the present application is shown;

[0032] Figure 7 A detailed flow chart showing how the sample location information of each sample information unit in the sample image is determined by the at least two decoder models according to different arrangement directions of the sample information units in the sample image according to an embodiment of the present application;

[0033] Figure 8 A detailed flow chart of determining sample location information of each target sample information unit in a sample image according to an embodiment of the present application is shown;

[0034] Fig. 9 A schematic diagram showing a framework for determining target positioning information of target sample information units in the target image for which feature information extraction has not been completed according to an embodiment of the present application;

[0035] Fig.10 A flowchart of a method before determining a decoder model as a target decoder model among the at least two decoder models according to an embodiment of the present application is shown;

[0036] Fig.11 A schematic diagram of a framework of a training information extraction model according to an embodiment of the present application is shown;

[0037] Fig.12 A detailed flow chart of training the model to be trained using the sample images according to an embodiment of the present application is shown.

[0038] Fig.13 A flow chart of an information extraction method according to an embodiment of the present application is shown;

[0039] Fig.14 A block diagram of a training device for an information extraction model according to an embodiment of the present application is shown;

[0040] Fig.15 A block diagram of a training device for an information extraction model according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0041] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0042] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0043] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0044] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0045] It should be noted that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0046] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the objects used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those shown or described.

[0047] The embodiments in this application involve technologies related to artificial intelligence, that is, fully automated processing of data (such as image data) is achieved through artificial intelligence. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0048] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solution of the embodiments of the present application can be applied is shown.

[0049] like Figure 1 As shown, the system architecture may include terminal devices (such as Figure 1 The embodiment of the present invention is a network 104 and a server 105. The network 104 is a medium for providing a communication link between the terminal device and the server 105. The network 104 may include various connection types, such as a wired communication link, a wireless communication link, etc.

[0050] In one embodiment of the present application, when a user needs to identify the characteristic information reflected by at least one information unit in a target image, the user can send the target image including at least one information unit to the server 105 through a terminal device. After acquiring the target image, the server 105 acquires an information extraction model, and extracts the characteristic information of at least one information unit in the target image through the information extraction model. The server 105 can obtain the information extraction model by pre-training in the following manner: first, a sample image is acquired, the sample image includes at least one sample information unit, then, a model to be trained is acquired, the model to be trained includes an encoder model and at least two decoder models, and finally, the model to be trained is trained through the sample image to obtain an information extraction model, wherein each decoder model learns the decoding results of part or all of the decoder models except itself during the training process.

[0051] In this embodiment, since the information extraction model is obtained by training the to-be-trained model including an encoder model and at least two decoder models, and each decoder model learns the decoding results of part or all of the decoder models except itself during the training process, each decoder model can learn the feature information of the information unit learned by part or all of the decoder models except itself in addition to the feature information of the information unit learned through its own training, so that the trained information extraction model can simultaneously pay attention to and fully utilize the feature information of the information unit in the image at different angles, thereby ensuring that the information extraction model has a high information extraction capability. Therefore, the accuracy of information extraction by the information extraction model can be improved.

[0052] It should be noted that the training method of the information extraction model provided in the embodiment of the present application can be executed by the server 105, and accordingly, the training device of the information extraction model is generally arranged in the server 105. However, in other embodiments of the present application, the terminal device can also have similar functions as the server, so as to execute the training scheme of the information extraction model provided in the embodiment of the present application.

[0053] It should also be noted that Figure 1 The number of terminal devices, networks and servers in the description is only for illustration. According to the implementation requirements, the server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0054] It should be explained that cloud computing as described above is a computing model that distributes computing tasks on a resource pool composed of a large number of computers, so that various application systems can obtain computing power, storage space and information services as needed. The network that provides resources is called a "cloud". The resources in the "cloud" are infinitely expandable in the eyes of users, and can be obtained at any time, used on demand, and expanded at any time. By establishing a cloud computing resource pool (referred to as a cloud platform, generally referred to as an IaaS (Infrastructure as a Service) platform, various types of virtual resources are deployed in the resource pool for external customers to choose to use. The cloud computing resource pool mainly includes: computing devices (virtualized machines, including operating systems), storage devices, and network devices.

[0055] The implementation details of the technical solution of the embodiment of the present application are described in detail below:

[0056] Figure 2 A flowchart of a method for training an information extraction model according to an embodiment of the present application is shown. The method for training an information extraction model can be performed by a device having a computing processing function, such as Figure 1 , or may be performed by the server 105 shown in Figure 1 Refer to the terminal device shown in Figure 2 As shown, the training method of the information extraction model includes at least steps 210 to 250, which are described in detail as follows:

[0057] In step 210, a sample image is acquired, wherein the sample image includes at least one sample information unit.

[0058] In the present application, the sample information unit in the sample image may be a character unit constituting a formula or text in the image, or may be a graphic unit constituting certain specific patterns in the image.

[0059] In such Figure 2 In one embodiment of step 210, the sample image may be obtained as follows: Figure 3 Follow the steps shown.

[0060] See also Figure 3 , shows a detailed flow chart of obtaining a sample image according to an embodiment of the present application. Specifically, it includes steps 211 to 214:

[0061] Step 211, obtaining at least one frame of original sample image.

[0062] Step 212: adjusting the height of each frame of the original sample image to a preset image height, or adjusting the width of each frame of the original sample image to a preset image width.

[0063] Step 213: adjusting the width of the original sample image or the height of the original sample image according to the adjustment ratio of the height or the adjustment ratio of the width to obtain a preprocessed sample image.

[0064] Step 214: Select a predetermined number of images from the pre-processed sample images as the sample images.

[0065] In this embodiment, the original sample image may refer to the sample image initially collected, which may be one frame or multiple frames, such as one thousand frames or ten thousand frames. The specific number of sample images may be determined according to actual conditions. Each frame of the sample image may include at least one information unit.

[0066] It should be noted that the sample images initially collected may have different sizes. In this case, the original sample images need to be preprocessed. Specifically, the height or width of each frame of the original sample image can be adjusted to a preset image height or preset image width, and the width or height of the original sample image can be adjusted according to the adjustment ratio of the height or width.

[0067] For example, the original sample image includes 5 frames, of which the height and width are 15×30 for the first frame, 5×15 for the second frame, 20×30 for the third frame, 5×30 for the fourth frame, and 15×45 for the fifth frame. Take the example of enlarging the width of the sample image to the preset image width of 60. It can be seen that the width enlargement ratio of the first frame image is 2 times, the width enlargement ratio of the second frame image is 4 times, the width enlargement ratio of the third frame image is 2 times, the width enlargement ratio of the fourth frame image is 2 times, the width enlargement ratio of the first frame image is 2 times, and the width enlargement ratio of the first frame image is 4 / 3 times.

[0068] The image height is enlarged at the same magnification ratio as the image width, and the height and width sizes of the preprocessed sample images are 30×60 for the first frame, 20×60 for the second frame, 40×60 for the third frame, 10×60 for the fourth frame, and 20×60 for the fifth frame.

[0069] After the preprocessed sample images are obtained, a predetermined number of images may be selected from the preprocessed sample images as the sample images.

[0070] In the present application, the height or width of each frame of the original sample image is adjusted to a preset image height or a preset image width. The advantage of this is that model training can be accelerated in the subsequent process.

[0071] In such Figure 3 In one embodiment of step 214, a predetermined number of images are selected from the pre-processed sample images as the sample images, which can be performed as follows: Figure 4 Follow the steps shown.

[0072] See also Figure 4 , shows a detailed flow chart of selecting a predetermined number of images from the preprocessed sample images as the sample images according to an embodiment of the present application. Specifically, it includes steps 2141 to 2142:

[0073] Step 2141: sort the preprocessed sample images according to their height or width.

[0074] Step 2142: Select a predetermined number of images that are connected in order from the pre-processed sample images as the sample images.

[0075] For example, taking the preprocessed sample images with height and width sizes of 30×60 for the first frame, 20×60 for the second frame, 40×60 for the third frame, 10×60 for the fourth frame, and 20×60 for the fifth frame as an example, the preprocessed sample images can be sorted in order of their heights to obtain the sorted fourth frame 10×60, fifth frame 20×60, second frame 20×60, first frame 30×60, and third frame 40×60.

[0076] Further, if the predetermined number is set to 4, the fourth frame, the fifth frame, the second frame, the first frame, or the fifth frame, the second frame, the first frame, and the third frame may be selected as the sample images.

[0077] In this application, when training a model based on sample images, the size of the sample images can be kept consistent in order to optimize the training effect. Based on this, when the height or width of the sample images is consistent, but the width or height of the images is inconsistent, the maximum width or maximum height of the sample images can be used as a reference to supplement the width or height of the sample images so that the height and width of the sample images are completely consistent.

[0078] Prior to this, the preprocessed sample images are sorted, and a predetermined number of sorted and connected images are selected from the preprocessed sample images as the sample images. This can avoid a large gap between the maximum width and the minimum width, or between the maximum height and the minimum height in the sample images, thereby avoiding excessive invalid information supplementation to the width or height of the sample images, thereby reducing the amount of computer calculations on invalid data during model training and saving computer resources.

[0079] Continue to refer to Figure 2 In step 230, a model to be trained is obtained, where the model to be trained includes an encoder model and at least two decoder models.

[0080] In the present application, the information extraction model can be obtained by training the model to be trained. Before training the model to be trained, it is necessary to construct the model to be trained. In the present application, the model to be trained can be constructed by an encoder model and at least two decoder models.

[0081] In the present application, those skilled in the art may understand that the encoder model may essentially belong to a network structure model, and the encoder model may be searched through the network structure before constructing the model to be trained.

[0082] In this application, Neural Architecture Search (NAS) is an effective tool for generating and optimizing network structures. When the length and structure of the network are uncertain, a recurrent neural network is used as a controller to generate fields of the network structure to construct a sub-neural network. The accuracy after training the sub-network is used as the controller feedback signal, and the controller is updated by calculating the policy gradient, and the iterative cycle is repeated continuously. In the next iteration, the controller will have a higher probability of proposing a high-accuracy network structure. Based on this, the encoder model is obtained by network structure search, which has the advantage of obtaining a better encoder model, so that the constructed model to be trained has accurate learning ability.

[0083] In one embodiment of the present application, the encoder model may include any one of a densely connected convolutional network model (DenseNet), a MobileNetV2 model, and an Xception model.

[0084] In one embodiment of the present application, the decoder model may include any one of a GRU model, an LSTM model, and a Transformer model.

[0085] Continue to refer to Figure 2 In step 250, the model to be trained is trained using the sample image to obtain an information extraction model, wherein each decoder model learns the decoding results of part or all of the decoder models except itself during the training process.

[0086] It should be noted that each decoder model learns the decoding results of part or all of the decoder models other than itself during the training process, which means that each decoder model in the at least two decoder models learns the decoding results of part or all of the decoder models other than itself in the at least two decoder models. For example, decoder model a, decoder model b and decoder model c are included. Based on this, for decoder model a, it can learn the decoding results of decoder model b and decoder model c, or only learn the decoding results of decoder model b, or only learn the decoding results of decoder model c.

[0087] In one embodiment of the present application, before training the model to be trained, at least one of the encoder model and the decoder model may be compressed based on any one of a model pruning algorithm, a model distillation algorithm, and a model quantization algorithm.

[0088] In the present application, at least one of the encoder model and the decoder model is compressed. The advantage is that the model volume of the encoder model and the decoder model can be further reduced with little or no loss in model accuracy, thereby further accelerating the calculation speed and saving computer resources.

[0089] It should be noted that the decoding result in step 250 is the feature information of the information unit in the image.

[0090] In such Figure 2 In one embodiment of step 250, the model to be trained is trained by the sample image to obtain an information extraction model, which can be performed as follows: Figure 5 Follow the steps shown.

[0091] See also Figure 5 , shows a detailed flow chart of training the model to be trained by the sample image according to an embodiment of the present application. Specifically including steps 251 to 254:

[0092] Step 251, encode the sample image through the encoder model to obtain sample encoding feature data.

[0093] Step 252: decode the sample encoding feature data respectively through the at least two decoder models to obtain at least two groups of sample feature information, wherein each group of sample feature information includes feature information for each sample information unit in the sample image.

[0094] Step 253, triggering each of the at least two decoder models to learn sample feature information decoded by part or all of the decoder models other than itself.

[0095] Step 254, determining a decoder model from the at least two decoder models as a target decoder model, and obtaining the information extraction model composed of the encoder model and the target decoder model.

[0096] In this application, it should be noted that Figure 5In step 252, the process of decoding the sample encoding feature data by each decoder model is performed independently and does not affect each other, that is, one decoder model can independently decode the sample encoding feature data to obtain a set of sample feature information.

[0097] In such Figure 5 In one embodiment of step 252, the sample encoding feature data is decoded by the at least two decoder models to obtain at least two sets of sample feature information. Figure 6 Follow the steps shown.

[0098] See also Figure 6 , shows a detailed flow chart of decoding the sample encoding feature data by the at least two decoder models respectively according to an embodiment of the present application. Specifically comprising steps 2521 to 2522:

[0099] Step 2521: Based on the sample coding feature data, the sample positioning information of each sample information unit in the sample image is determined by the at least two decoder models according to different arrangement directions of the sample information units in the sample image.

[0100] Step 2522: Based on the sample positioning information corresponding to the at least two decoder models, the sample encoding feature data are decoded respectively to obtain the at least two groups of sample feature information.

[0101] In the present application, it should be noted that the information units in the image may have different arrangement directions, for example, an arrangement direction from left to right, an arrangement direction from right to left, an arrangement direction from top to bottom, and an arrangement direction from bottom to top.

[0102] It can be understood that when the sample positioning information of each information unit in the sample image is determined according to different arrangement directions, the order of determining the sample positioning information of each information unit in the sample image is also different.

[0103] It should be noted that, in the present application, the at least two decoder models can be two decoder models, three decoder models, or four decoder models. Here, the present application does not make any specific limitation on the number of decoder models.

[0104] In such Figure 6 In one embodiment of step 2521, the at least two decoder models may include a first decoder model and a second decoder model.

[0105] Further, the sample location information of each sample information unit in the sample image is determined by the at least two decoder models according to different arrangement directions of the sample information units in the sample image, and the sample location information of each sample information unit in the sample image can be determined as follows: Figure 7 Follow the steps shown.

[0106] See also Figure 7 , shows a detailed flow chart of determining the sample location information of each sample information unit in the sample image according to different arrangement directions of the sample information units in the sample image by the at least two decoder models according to an embodiment of the present application. Specifically including steps 25211 to 25212:

[0107] Step 25211: Determine the sample positioning information of each sample information unit in the sample image according to the first arrangement direction of the sample information units in the sample image through the first decoder model.

[0108] Step 25212: Determine the sample location information of each sample information unit in the sample image according to a second arrangement direction of the sample information units in the sample image through the second decoder model, wherein the second arrangement direction is the opposite direction of the first arrangement direction.

[0109] In order to enable those skilled in the art to better understand the present application, the following description is given by taking a formula recognition scenario in an image as an example.

[0110] For example, for the formula "A+BC", the character "+" is arranged on the right side of the character "A", the character "B" is arranged on the right side of the character "+", the character "-" is arranged on the right side of the character "B", and the character "C" is arranged on the right side of the character "-".

[0111] Further, by determining the sample positioning information of each information unit in the sample image according to the first arrangement direction of the information units in the sample image through the first decoder model, the sample positioning information of each character in the formula "A+BC" can be determined according to the first arrangement direction of "A"→"+"→"B"→"-"→"C". By determining the sample positioning information of each information unit in the sample image according to the second arrangement direction of the information units in the sample image through the second decoder model, the sample positioning information of each character in the formula "A+BC" can be determined according to the second arrangement direction of "C"→"-"→"B"→"+"→"A".

[0112] In such Figure 6 In one embodiment of step 2521, each decoder model may include at least two localization convolution kernels of different scales.

[0113] It should be noted that in the present application, at least two positioning convolution kernels of different scales can be two positioning convolution kernels of different scales, three positioning convolution kernels of different scales, or four positioning convolution kernels of different scales. Here, the present application does not make any specific limitation on the number of positioning convolution kernels.

[0114] Further, for each target sample information unit in the sample image, the following can be used: Figure 8 The steps shown determine the sample location information of the target sample information unit.

[0115] See also Figure 8 , shows a detailed flow chart of determining sample location information of each target sample information unit in a sample image according to an embodiment of the present application. Specifically, it includes steps 25213 to 25215:

[0116] Step 25213, obtaining the historical positioning information of the sample information unit whose feature information has been extracted in the sample image.

[0117] Step 25214: Convolution processing is performed on the historical positioning information respectively through the at least two positioning convolution kernels of different scales to obtain at least two positioning reference information, and the at least two positioning reference information are used to focus on the positioning information of sample information units of different sizes respectively.

[0118] Step 25215: Based on the at least two positioning reference information, determine the sample positioning information of the target sample information unit in the sample image whose feature information extraction has not been completed.

[0119] In the present application, the decoder model includes at least two localization convolution kernels of different scales, which can be used to focus on information units of different sizes in the target image.

[0120] It can be understood that in the present application, if the information unit includes a character unit, and at least one information unit in the target image constitutes one or more formulas containing the character unit, then at least two positioning reference information obtained by convolving the historical positioning information with at least two positioning convolution kernels of different scales can be used to focus on the positioning information of character units of different sizes respectively.

[0121] In order to help those skilled in the art better understand this embodiment, the following will continue to take the formula recognition scene in the image as an example. Fig. 9 This is explained with a specific example.

[0122] See also Fig. 9 , shows a schematic diagram of a framework for determining target positioning information of target sample information units in the target image for which feature information extraction has not been completed according to an embodiment of the present application.

[0123] like Fig. 9 As shown, for the formula "A+BC", if the feature information extraction for the character "A" and the character "+" has been completed historically, the historical positioning information 901 of the character "A" and the character "+" can be obtained, and the historical positioning information of the character "A" and the character "+" can be added to obtain the historical positioning information 902. It should be noted that the positioning information mentioned in this application can essentially be represented by a matrix information.

[0124] Furthermore, by performing convolution processing on the historical positioning information 902 through a 5×5 positioning convolution kernel 903 and an 11×11 positioning convolution kernel 904 respectively, a first positioning reference information 906 and a second positioning reference information 907 can be obtained. Then, the sample positioning information of the character "B" (target information unit) for which feature information extraction has not been completed in the formula "A+BC" can be determined through the first positioning reference information 906 and the second positioning reference information 907.

[0125] Furthermore, in the process of determining the sample positioning information of the character "B" whose feature information extraction has not been completed in the formula "A+BC" through the first positioning reference information 906 and the second positioning reference information 907, the hidden state information 905 of the information unit (i.e., the character "+") that is closest to the target information unit (i.e., the character "B") during feature information extraction can be obtained simultaneously, and the encoded feature data 908 for the target image (i.e., the target image containing the formula "A+BC") can be obtained. Finally, the first positioning reference information 906, the second positioning reference information 907, the hidden state information 905 and the encoded feature data 908 are aggregated to obtain the target positioning information 909 of the target information unit (i.e., the character "B").

[0126] In the present application, by convolving the historical positioning information with positioning convolution kernels of different scales, it is possible to focus on information units of different sizes in the target image, so that more accurate positioning information can be determined for the target information unit in the subsequent process. By aggregating the at least two positioning reference information, the hidden state information, and the encoded feature data, target positioning information of the target information unit with higher accuracy can be obtained, thereby improving the accuracy of the target feature information of the target information unit determined by the target positioning information.

[0127] In one embodiment of the present application, the sample image may also be matched with a sample label, and the sample label is used to characterize the real feature information of at least one information unit in the sample image.

[0128] Furthermore, in Figure 5 Before step 254, that is, before determining a decoder model as a target decoder model among the at least two decoder models, the following steps may also be performed: Fig.10 Steps shown.

[0129] See also Fig.10 , shows a flow chart of a method before determining a decoder model as a target decoder model from the at least two decoder models according to an embodiment of the present application. Specifically comprising steps 2531 to 2532:

[0130] Step 2531: Based on the sample label, determine the error information existing in the sample feature information decoded by the at least two decoder models respectively.

[0131] Step 2532: Based on the error information, reverse update the model parameters in the corresponding decoder model through a preset loss function.

[0132] In the present application, the sample labels are used as supervisory information to reversely update the model parameters in the decoder model, which can also play a role in continuously optimizing the decoder model, thereby improving the performance of the decoding model.

[0133] In order to enable those skilled in the art to better understand the above embodiments, the following will continue to take the formula recognition scene in the image as an example. Fig.11 Let's take a specific example to illustrate.

[0134] See also Fig.11 , showing a framework diagram of a training information extraction model according to an embodiment of the present application.

[0135] like Fig.11 As shown, the model to be trained includes an encoder model 1101, a first decoder model 1102 and a second decoder model 1103.

[0136] First, the encoder model encodes the target image containing the formula "A+BC" to obtain encoded feature data, and then the first decoder model 1102 and the second decoder model 1103 respectively decode the encoded feature data based on the attention mechanism to obtain two sets of feature information.

[0137] In the process of decoding the encoded feature data, the first decoder model 1102 and the second decoder model 1103 can respectively determine the positioning information of each information unit in the target image according to the different arrangement directions of the information units (i.e., character units) in the target image, and then decode the encoded feature data based on the positioning information to obtain two sets of sample feature information.

[0138] For example, the first decoder model 1102 can determine the positioning information of the character "A", the character "+", the character "B", the character "-" and the character "C" according to the arrangement direction of "A" → "+" → "B" → "-" → "C" in the formula "A+BC", and the second decoder model 1103 can determine the positioning information of the character "C", the character "-", the character "B", the character "+" and the character "A" according to the arrangement direction of "C" → "-" → "B" → "+" → "A" in the formula "A+BC". After determining the positioning information of each character unit in the formula "A+BC", the first decoder model 1102 and the second decoder model 1103 can extract feature information of the character unit based on the positioning information of the character unit determined by themselves.

[0139] Furthermore, after obtaining two sets of feature information for the formula "A+BC", the first decoder model 1102 is triggered to learn the feature information decoded by the second decoder model 1103, and the second decoder model 1103 is also triggered to learn the feature information decoded by the first decoder model 1102, so as to optimize the model parameters in the first decoder model 1102 and the second decoder model 1103 respectively.

[0140] Finally, a decoder model is retained as a target decoder model between the first decoder model 1102 and the second decoder model 1103 (for example, the first decoder model 1102 is retained as the target decoder model), and the information extraction model composed of the encoder model and the target decoder model is obtained.

[0141] In the present application, by respectively performing decoding training on each information unit in the image according to different arrangement directions of the information units in the image, the at least two decoder models can simultaneously focus on and fully utilize information from different angles of the information units in the image (such as historical and future information). Furthermore, through mutual learning, the decoder models can fully utilize the complementary information of different arrangement directions of the information units and explore long-distance dependency information, thereby making the decoding ability of the decoder model stronger, thereby helping to improve the accuracy of information extraction by the information extraction model.

[0142] In such Figure 2 In one embodiment of step 250, the model to be trained is trained by the sample image to obtain the information extraction model. Fig.12 Follow the steps shown.

[0143] See also Fig.12 , shows a detailed flow chart of training the model to be trained by the sample image according to an embodiment of the present application. Specifically including steps 255 to 256:

[0144] Step 255: training the model to be trained using the sample images according to a predetermined number of training times to obtain an information extraction reference model for the predetermined number of training times.

[0145] Step 256, averaging the information extraction reference models trained a predetermined number of times to obtain the information extraction model.

[0146] In this embodiment, the model to be trained can be trained for multiple rounds using the sample images, for example, 12 rounds of training, wherein each round of training will obtain an information extraction reference model, and then the obtained multiple information extraction reference models are averaged to obtain the information extraction model. The advantage of doing so is that the accuracy of the model can be improved, and the accuracy of the information extraction model in extracting feature information in the image can be enhanced.

[0147] Next, we will combine Fig.13 An information extraction scheme according to an embodiment of the present application is described.

[0148] See also Fig.13 , which shows a flow chart of an information extraction method according to an embodiment of the present application. The information extraction method can be executed by a device having a computing processing function, such as Figure 1 , or may be performed by the server 105 shown in Figure 1 Refer to the terminal device shown in Fig.13 As shown, the information extraction method at least includes steps 220 to 260, which are described in detail as follows:

[0149] Step 220: Acquire a target image, wherein the target image includes at least one information unit.

[0150] In the present application, the proposed information extraction scheme can be applied to information recognition scenarios of target objects in images, such as formula recognition scenarios in images, text recognition scenarios in images, and certain specific pattern recognition scenarios in images. Furthermore, the target object in the image can be composed of at least one information unit, such as a formula or text in an image can be composed of at least one character unit, and certain specific patterns in an image can be composed of at least one graphic unit.

[0151] It should be noted that the target image may be obtained by capturing the page area containing the target object in the interface, or may be directly obtained locally.

[0152] Step 240: Acquire the information extraction model obtained by the information extraction model training method.

[0153] Step 260: extracting feature information of at least one information unit in the target image through the information extraction model.

[0154] It should be noted that the feature information proposed in the present application may include the shape feature information of the information unit itself, may include the position feature information of the information unit (for example, the relative position relationship with other information units), and may also include the shape feature information and position feature information of the information unit itself. For example, taking the formula recognition scenario in the image as an example, the information unit may be a character unit, and at least one information unit in the target image constitutes one or more formulas containing the character unit. It can be understood that the feature information of the character unit in the formula may include the shape feature information of each character unit and / or the feature information of the relative position relationship between the character units.

[0155] In the present application, each information unit in the target image corresponds to a piece of positioning information in the target image. Before extracting feature information from the information unit, the positioning information of the information unit needs to be determined.

[0156] It should be noted that, in the process of extracting the characteristic information of the information unit in the target image, the positioning information of an information unit may be determined first, and the characteristic information of the information unit defined by the positioning information may be extracted. Then, the positioning information of the next information unit may be determined, and the characteristic information of the next information unit may be extracted. In this way, the characteristic information of the information units may be extracted step by step until the characteristic information of all the information units in the target image is extracted.

[0157] In order to enable those skilled in the art to better understand the present application, the following description is given by taking a formula recognition scenario in an image as an example.

[0158] For example, for the character "B" and the character "-" in the formula "A+BC", the positioning information of the character "B" is determined first, and the feature information of the character "B" is extracted based on the positioning information of the character "B". Then, the positioning information of the character "-" is determined, and the feature information of the character "-" is extracted based on the positioning information of the character "-".

[0159] In this application, we continue to take the formula recognition scenario in the image as an example. Fig.13 In an embodiment after step 260, that is, after extracting feature information of at least one information unit in the target image through the information extraction model, the following scheme may also be performed:

[0160] Based on feature information of at least one information unit in the target image, one or more formulas in the target image are edited into a formula editing area.

[0161] Specifically, in this application scenario, when editing a document, the user can capture the formula image that needs to be edited on the web page, and then, based on the information extraction scheme proposed in this application, extract the first feature information of at least one character unit in the formula image to obtain the target feature information, and then edit the formula in the formula image to the formula editing area based on the target feature information. It can be seen that the information extraction method proposed in this application can bring great convenience and excellent user experience to users in the formula editing process.

[0162] In the present application, an information extraction model can be trained by a model to be trained including an encoder model and at least two decoder models. Since the information extraction model is obtained by training the model to be trained including an encoder model and at least two decoder models, and each decoder model learns the decoding results of part or all of the decoder models other than itself during the training process, each decoder model can learn the feature information of the information unit learned by part or all of the decoder models other than itself in addition to learning the feature information of the information unit through its own training, so that the trained information extraction model can simultaneously pay attention to and make full use of the feature information of the information unit in the image at different angles, thereby ensuring that the information extraction model has a high information extraction capability. Therefore, the accuracy of information extraction by the information extraction model can be improved.

[0163] The following describes an apparatus embodiment of the present application, which can be used to execute the training method of the information extraction model in the above-mentioned embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the embodiment of the training method of the information extraction model in the above-mentioned embodiment of the present application.

[0164] Fig.14 A block diagram of a training device for an information extraction model according to an embodiment of the present application is shown.

[0165] Reference Fig.14 As shown, a training device 1400 for an information extraction model according to an embodiment of the present application includes: a first acquisition unit 1401, a second acquisition unit 1402 and a training unit 1403.

[0166] Among them, the first acquisition unit 1401 is used to acquire a sample image, which includes at least one sample information unit; the second acquisition unit 1402 is used to acquire a model to be trained, which includes an encoder model and at least two decoder models; the training unit 1403 is used to train the model to be trained through the sample image to obtain an information extraction model, wherein each decoder model learns the decoding results of part or all of the decoder models except itself during the training process.

[0167] In some embodiments of the present application, based on the aforementioned scheme, the first acquisition unit 1401 is configured to: acquire at least one frame of original sample image; adjust the height of each frame of original sample image to a preset image height, or adjust the width of each frame of original sample image to a preset image width; adjust the width of the original sample image or the height of the original sample image according to the adjustment ratio of the height or the adjustment ratio of the width to obtain a preprocessed sample image; select a predetermined number of images from the preprocessed sample image as the sample image.

[0168] In some embodiments of the present application, based on the aforementioned scheme, the first acquisition unit 1401 is further configured to: sort the preprocessed sample images according to their height or width; and select a predetermined number of images that are sorted and connected from the preprocessed sample images as the sample images.

[0169] In some embodiments of the present application, based on the aforementioned scheme, the encoder model includes a densely connected convolutional network model.

[0170] In some embodiments of the present application, based on the aforementioned scheme, the training unit 1403 is configured to: encode the sample image through the encoder model to obtain sample encoding feature data; decode the sample encoding feature data through the at least two decoder models respectively to obtain at least two groups of sample feature information, wherein each group of sample feature information includes feature information for each sample information unit in the sample image; trigger each of the at least two decoder models to learn part or all of the sample feature information decoded by the decoder model except itself; determine a decoder model as the target decoder model among the at least two decoder models to obtain the information extraction model composed of the encoder model and the target decoder model.

[0171] In some embodiments of the present application, based on the aforementioned scheme, the training unit 1403 is further configured to: based on the sample coding feature data, determine the sample positioning information of each sample information unit in the sample image according to different arrangement directions of the sample information units in the sample image through the at least two decoder models; based on the sample positioning information corresponding to the at least two decoder models, respectively decode the sample coding feature data to obtain the at least two groups of sample feature information.

[0172] In some embodiments of the present application, based on the aforementioned scheme, the at least two decoder models include a first decoder model and a second decoder model, and the training unit 1403 is also configured to: determine the sample positioning information of each sample information unit in the sample image according to a first arrangement direction of the sample information units in the sample image by the first decoder model; determine the sample positioning information of each sample information unit in the sample image according to a second arrangement direction of the sample information units in the sample image by the second decoder model, and the second arrangement direction is the opposite direction of the first arrangement direction.

[0173] In some embodiments of the present application, based on the aforementioned scheme, the decoder model includes at least two positioning convolution kernels of different scales, and the device also includes: a determination unit, which is used to determine the sample positioning information of the target sample information unit in the sample image for each target sample information unit in the sample image in the following manner: obtaining historical positioning information of the sample information unit for which feature information extraction has been completed in the sample image; performing convolution processing on the historical positioning information respectively through the at least two positioning convolution kernels of different scales to obtain at least two positioning reference information, and the at least two positioning reference information are used to respectively focus on the positioning information of sample information units of different sizes; based on the at least two positioning reference information, determining the sample positioning information of the target sample information unit for which feature information extraction has not been completed in the sample image.

[0174] In some embodiments of the present application, based on the aforementioned scheme, the sample image is also matched with a sample label, and the sample label is used to characterize the real feature information of at least one sample information unit in the sample image. The training unit 1403 is also configured to: before determining a decoder model as the target decoder model among the at least two decoder models, based on the sample label, determine the error information existing in the sample feature information decoded by the at least two decoder models respectively; based on the error information, reversely update the model parameters in the corresponding decoder model through a preset loss function.

[0175] In some embodiments of the present application, based on the aforementioned scheme, the training unit 1403 is further configured to: train the model to be trained through the sample image according to a predetermined number of training times to obtain the information extraction reference model of the predetermined number of training times; and average the information extraction reference model of the predetermined number of training times to obtain the information extraction model.

[0176] The present application also provides an information extraction device, which includes: a third acquisition unit, used to acquire a target image, wherein the target image includes at least one information unit; a fourth acquisition unit, used to acquire an information extraction model obtained by the training method of the information extraction model as described in the above embodiment; and an extraction unit, used to extract feature information of at least one information unit in the target image through the information extraction model.

[0177] In some embodiments of the present application, based on the aforementioned scheme, the information unit includes a character unit, and the at least one information unit constitutes one or more formulas containing the character unit. The information extraction device also includes an editing unit, which is used to edit one or more formulas in the target image to a formula editing area based on the feature information of at least one information unit in the target image after extracting the feature information of at least one information unit in the target image through the information extraction model.

[0178] As another aspect, an embodiment of the present application also provides another information extraction model training device and an information extraction device, including a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing a training method for the information extraction model as described in the above embodiments, and an information extraction method.

[0179] Fig.15 The training device of the information extraction model according to one embodiment of the present application and the block diagram of the information extraction device are shown. For example, the device 1500 can be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0180] Reference Fig.15 , device 1500 may include one or more of the following components: a processing component 1502 , a memory 1504 , a power component 1506 , a multimedia component 1508 , an audio component 1510 , an input / output (I / O) interface 1512 , a sensor component 1514 , and a communication component 1516 .

[0181] The processing component 1502 generally controls the overall operation of the device 1500, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 1502 may include one or more processors 1518 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 1502 may include one or more modules to facilitate the interaction between the processing component 1502 and other components. For example, the processing component 1502 may include a multimedia module to facilitate the interaction between the multimedia component 1508 and the processing component 1502.

[0182] The memory 1504 is configured to store various types of data to support operations on the device 1500. Examples of such data include instructions for any application or method operating on the device 1500, contact data, phone book data, messages, pictures, videos, etc. The memory 1504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0183] The power supply component 1506 provides power to the various components of the device 1500. The power supply component 1506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 1500.

[0184] The multimedia component 1508 includes a screen that provides an output interface between the device 1500 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1508 includes a front camera and / or a rear camera. When the device 1500 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0185] The audio component 1510 is configured to output and / or input audio signals. For example, the audio component 1510 includes a microphone (MIC), and when the device 1500 is in an operation mode, such as a call mode, a recording mode, and a voice information processing mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 1504 or sent via the communication component 1516. In some embodiments, the audio component 1510 also includes a speaker for outputting audio signals.

[0186] I / O interface 1512 provides an interface between processing component 1502 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0187] The sensor assembly 1514 includes one or more sensors for providing various aspects of the status assessment of the device 1500. For example, the sensor assembly 1514 can detect the open / closed state of the device 1500, the relative positioning of components, such as the display and keypad of the device 1500, the sensor assembly 1514 can also search for changes in the position of the search result display device 1500 or a component of the device 1500, the presence or absence of user contact with the device 1500, the orientation or acceleration / deceleration of the device 1500, and the temperature change of the device 1500. The sensor assembly 1514 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 1514 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1514 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0188] The communication component 1516 is configured to facilitate wired or wireless communication between the device 1500 and other devices. The device 1500 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 1516 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1516 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency information processing (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0189] In an exemplary embodiment, the apparatus 1500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above methods.

[0190] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1504 including instructions, and the instructions can be executed by a processor 1518 of the device 1500 to complete the above-mentioned information extraction model training method and information extraction method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0191] As another aspect, the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the training method of the information extraction model described in the above embodiment, and the information extraction method.

[0192] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The computer-readable storage medium stores at least one program code, which is loaded and executed by the processor of the device to implement the operations performed by the training method of the information extraction model described in the above embodiment, and the operations performed by the information extraction method.

[0193] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0194] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0195] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0196] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A training method for an information extraction model, characterized in that: The method comprises: Acquire at least one frame of original sample image; adjust the height of each frame of original sample image to a preset image height, or adjust the width of each frame of original sample image to a preset image width; adjust the width of the original sample image or the height of the original sample image according to the adjustment ratio of the height or the adjustment ratio of the width to obtain a preprocessed sample image; sort the preprocessed sample images according to the height or width of the preprocessed sample images; select a predetermined number of images that are sorted and connected from the preprocessed sample images as sample images, wherein the sample images include at least one sample information unit; Obtain a model to be trained, wherein the model to be trained includes an encoder model and at least two decoder models; The model to be trained is trained using the sample images to obtain an information extraction model, wherein each decoder model learns decoding results of part or all of the decoder models except itself during the training process.

2. The method according to claim 1, characterized in that The step of training the model to be trained by using the sample image to obtain an information extraction model includes: Encoding the sample image by using the encoder model to obtain sample encoding feature data; Decoding the sample encoded feature data by the at least two decoder models respectively to obtain at least two groups of sample feature information, wherein each group of sample feature information includes feature information for each sample information unit in the sample image; Triggering each of the at least two decoder models to learn sample feature information decoded by part or all of the decoder models other than itself; A decoder model is determined as a target decoder model among the at least two decoder models, and the information extraction model consisting of the encoder model and the target decoder model is obtained.

3. The method according to claim 2, characterized in that The decoding of the sample encoding feature data by the at least two decoder models respectively to obtain at least two sets of sample feature information includes: Based on the sample coding feature data, determining, by the at least two decoder models, sample positioning information of each sample information unit in the sample image according to different arrangement directions of the sample information units in the sample image; Based on the sample positioning information corresponding to the at least two decoder models, the sample encoding feature data is decoded respectively to obtain the at least two groups of sample feature information.

4. The method according to claim 3, characterized in that The at least two decoder models include a first decoder model and a second decoder model, and determining the sample positioning information of each sample information unit in the sample image according to different arrangement directions of the sample information units in the sample image by using the at least two decoder models respectively includes: Determining, by the first decoder model, sample positioning information of each sample information unit in the sample image according to a first arrangement direction of the sample information units in the sample image; The sample location information of each sample information unit in the sample image is determined by the second decoder model according to a second arrangement direction of the sample information units in the sample image, wherein the second arrangement direction is the opposite direction of the first arrangement direction.

5. The method according to claim 3, characterized in that: The decoder model includes at least two localization convolution kernels of different scales, and for each target sample information unit in the sample image, the sample localization information of the target sample information unit is determined in the following manner: Obtaining historical positioning information of sample information units from which feature information extraction has been completed in the sample image; The historical positioning information is convolved by the at least two positioning convolution kernels of different scales to obtain at least two positioning reference information, where the at least two positioning reference information are used to focus on the positioning information of sample information units of different sizes respectively; Based on the at least two pieces of positioning reference information, sample positioning information of a target sample information unit in the sample image whose feature information extraction has not been completed is determined.

6. The method according to claim 2, characterized in that The sample image is further matched with a sample label, and the sample label is used to characterize real feature information of at least one sample information unit in the sample image. Before determining a decoder model as a target decoder model among the at least two decoder models, the method further includes: Based on the sample labels, respectively determining error information existing in the sample feature information decoded by the at least two decoder models; Based on the error information, the model parameters in the corresponding decoder model are reversely updated through a preset loss function.

7. The method according to claim 1, characterized in that The step of training the model to be trained by using the sample image to obtain an information extraction model includes: According to a predetermined number of training times, the model to be trained is trained using the sample images to obtain an information extraction reference model for the predetermined number of training times; The information extraction reference model trained a predetermined number of times is averaged to obtain the information extraction model.

8. An information extraction method, characterized in that: The method comprises: Acquire a target image, wherein the target image includes at least one information unit; Acquire an information extraction model obtained by the information extraction model training method according to any one of claims 1 to 7; The feature information of at least one information unit in the target image is extracted through the information extraction model.

9. The method according to claim 8, characterized in that The information unit includes a character unit, and the at least one information unit constitutes one or more formulas including the character unit. After extracting feature information of at least one information unit in the target image through the information extraction model, the method further includes: Based on feature information of at least one information unit in the target image, one or more formulas in the target image are edited into a formula editing area.

10. A training device for an information extraction model, characterized in that: The device comprises: The first acquisition unit is used to acquire at least one frame of original sample image; adjust the height of each frame of original sample image to a preset image height, or adjust the width of each frame of original sample image to a preset image width; adjust the width of the original sample image or the height of the original sample image according to the adjustment ratio of the height or the adjustment ratio of the width to obtain a pre-processed sample image; sort the pre-processed sample images according to the height or width of the pre-processed sample images; select a predetermined number of images that are sorted and connected from the pre-processed sample images as sample images, wherein the sample images include at least one sample information unit; A second acquisition unit is used to acquire a model to be trained, wherein the model to be trained includes an encoder model and at least two decoder models; The training unit is used to train the model to be trained through the sample image to obtain an information extraction model, wherein each decoder model learns the decoding results of part or all of the decoder models except itself during the training process.

11. A training device for an information extraction model, characterized in that: It includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing the training method of the information extraction model as described in any one of claims 1 to 7.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the operations performed by the training method of the information extraction model as described in any one of claims 1 to 7.

13. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium and are suitable for being read and executed by a processor, so that a computer device having the processor executes the information extraction model training method described in any one of claims 1 to 7.

Images