Information extraction method, device and medium

Abstract

The embodiments of the present application provide an information extraction method, device, and medium, which relate to the fields of computer and artificial intelligence technology. The method comprises: obtaining a target image, wherein the target image comprises at least one information unit; obtaining an information extraction model, wherein the information extraction model is obtained by training at least one information unit in a training sample image on training labels of multiple feature types; and extracting the first feature information of at least one information unit in the target image as target feature information through the information extraction model. The technical solution of the embodiments of the present application can improve the accuracy of information extraction.

Description

Technical Field

[0001] The present application relates to the field of computer and artificial intelligence technology, and more specifically, to an information extraction method, device and medium. Background Art

[0002] In information extraction scenarios, such as extracting information from images (e.g., extracting formulas or text from images), the information units in the image are usually segmented, identified, and post-processed in sequence to extract the information in the image. However, in the process of segmenting, identifying, and post-processing the information units in the image in sequence, errors may accumulate, resulting in inaccurate information extraction from the image. Therefore, how to improve the accuracy of information extraction is a technical problem that needs to be solved urgently. Summary of the invention

[0003] The embodiments of the present application provide an information extraction method, apparatus, computer program product or computer program, and computer-readable medium, which can improve the accuracy of information extraction at least to a certain extent.

[0004] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0005] According to one aspect of an embodiment of the present application, there is provided an information extraction method, comprising: acquiring a target image, the target image comprising at least one information unit; acquiring an information extraction model, the information extraction model being trained by training labels of at least one information unit in a training sample image on multiple feature types; and extracting first feature information of at least one information unit in the target image as target feature information through the information extraction model.

[0006] According to one aspect of an embodiment of the present application, an information extraction device is provided, comprising: a first acquisition unit, used to acquire a target image, the target image including at least one information unit; a second acquisition unit, used to acquire an information extraction model, the information extraction model being trained by training labels of at least one information unit in a training sample image on multiple feature types; an extraction unit, used to extract first feature information of at least one information unit in the target image as target feature information through the information extraction model.

[0007] In some embodiments of the present application, based on the aforementioned scheme, the second acquisition unit includes: a third acquisition unit, used to acquire a training sample image; a determination unit, used to determine training labels on multiple feature types for at least one information unit in the training sample image, and obtain multiple training labels, wherein each training label is used to characterize the feature information of the at least one information unit on the corresponding feature type; a fourth acquisition unit, used to acquire a model to be trained, and train the model to be trained through the training sample image and the multiple training labels to obtain the information extraction model.

[0008] In some embodiments of the present application, based on the aforementioned scheme, the third acquisition unit is configured to: acquire at least one frame of original training sample image; scale the height or width of each frame of the original training sample image to a preset image height or a preset image width, and scale the width or height of the original training sample image according to the scaling ratio of the height or the width to obtain a preprocessed training sample image; select a predetermined number of images from the preprocessed training sample images as the training sample images.

[0009] In some embodiments of the present application, based on the aforementioned scheme, the third acquisition unit is configured to: sort the preprocessed training sample images according to the width or height of the preprocessed training sample images; and select a predetermined number of images that are sorted and connected from the preprocessed training sample images as the training sample images.

[0010] In some embodiments of the present application, based on the aforementioned scheme, the information unit includes explicit information units and implicit information units, and the determination unit is configured to: obtain an information unit dictionary matching each feature type, wherein the information unit dictionary records at least feature vectors corresponding to all explicit information units; based on the feature vectors recorded in each information unit dictionary, construct a training label on the corresponding feature type for at least one information unit in the training sample image.

[0011] In some embodiments of the present application, based on the aforementioned scheme, the fourth acquisition unit is configured to: acquire an encoder model and a decoder model through a network structure search, the encoder model is used to encode an image, and the decoder model is used to decode the features encoded by the encoder to obtain second feature information of at least one information unit in the image; based on the encoder model and the decoder model, construct the model to be trained.

[0012] In some embodiments of the present application, based on the aforementioned scheme, the fourth acquisition unit is configured as follows: according to a predetermined number of training times, the model to be trained is trained using the training sample images and the multiple training labels to obtain the information extraction reference model for the predetermined number of training times; and the information extraction reference model for the predetermined number of training times is averaged to obtain the information extraction model.

[0013] In some embodiments of the present application, based on the aforementioned scheme, the fourth acquisition unit is configured to: input the training sample image into the model to be trained, and obtain the third feature information of at least one information unit in the training sample image output by the model to be trained; based on the training label corresponding to each feature type, determine the error information existing in the corresponding feature type in the third feature information; based on the error information existing in each feature type, reversely update the model parameters in the model to be trained respectively through the preset loss function corresponding to each feature type to obtain the information extraction reference model.

[0014] In some embodiments of the present application, based on the aforementioned scheme, the fourth acquisition unit is configured to: select a training label corresponding to a target feature type from the multiple training labels; input the training sample image into the model to be trained, and obtain the third feature information of at least one information unit in the training sample image output by the model to be trained; based on the training label corresponding to the target feature type, determine the target error information existing on the target feature type in the third feature information; based on the target error information, reversely update the model parameters in the model to be trained through a preset loss function corresponding to the target feature type to obtain an intermediate information extraction reference model; use the intermediate information extraction reference model as a new model to be trained, and re-execute the step of selecting a training label corresponding to a target feature type from the multiple training labels until all the labels in the multiple training labels are selected to obtain the information extraction reference model.

[0015] In some embodiments of the present application, based on the aforementioned scheme, the information unit includes a character unit, and the determination unit is configured to: determine a training label on a positioning feature for at least one character unit in the training sample image, wherein the positioning feature is used to at least characterize a relative relationship feature between the at least one character unit; determine a training label on a shape feature for at least one character unit in the training sample image.

[0016] In some embodiments of the present application, based on the aforementioned scheme, the extraction unit is configured to: for each target information unit in the at least one information unit, determine all the information units and some of the information units arranged before the target information unit; based on the feature information of all the information units, predict the first reference feature information of the target information unit; based on the feature information of the some of the information units, predict the second reference feature information of the target information unit; based on the first reference feature information and the second reference feature information of each information unit in the at least one information unit, determine the first feature information.

[0017] In some embodiments of the present application, based on the aforementioned scheme, the information unit includes a character unit, and the at least one information unit constitutes one or more formulas containing the character unit. The device also includes an editing unit, which is used to extract the first feature information of at least one information unit in the target image as the target feature information through the information extraction model, and then edit one or more formulas in the target image to a formula editing area based on the target feature information.

[0018] According to one aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the information extraction method described in the above embodiments.

[0019] According to one aspect of an embodiment of the present application, an information extraction device is also provided, characterized in that it includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing the information extraction method described in the above embodiments.

[0020] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which at least one program code is stored. The at least one program code is loaded and executed by a processor to implement the operations performed by the information extraction method described in the above embodiment.

[0021] In the technical solutions provided in some embodiments of the present application, the information extraction model obtained by training the training labels of at least one information unit in the training sample image on multiple feature types can extract the first feature information of at least one information unit in the target image. Since the training labels of the information unit on multiple feature types take into account the supervisory feature information of the information unit in the image on multiple feature types, the trained information extraction model has the ability to accurately extract information, and by extracting the feature information reflected by at least one information unit in the target image through the information extraction model, the accuracy of information extraction can be improved.

[0022] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0024] Figure 1 A schematic diagram showing an exemplary system architecture to which the technical solution of the embodiments of the present application can be applied;

[0025] Figure 2 A flow chart of an information extraction method according to an embodiment of the present application is shown;

[0026] Figure 3 A detailed flow chart of obtaining an information extraction model according to an embodiment of the present application is shown;

[0027] Figure 4 A detailed flow chart of acquiring training sample images according to an embodiment of the present application is shown;

[0028] Figure 5 A detailed flow chart of determining training labels on multiple feature types for at least one information unit in the training sample image according to an embodiment of the present application is shown;

[0029] Figure 6 A detailed flowchart of obtaining a model to be trained according to an embodiment of the present application is shown;

[0030] Figure 7 A detailed flow chart of training the model to be trained using the training sample images and the multiple training labels according to an embodiment of the present application is shown;

[0031] Figure 8 A detailed flow chart of training the model to be trained using the training sample images and the multiple training labels according to an embodiment of the present application is shown;

[0032] Fig. 9 A detailed flow chart of training the model to be trained using the training sample images and the multiple training labels according to an embodiment of the present application is shown;

[0033] Fig.10 A schematic diagram of a framework of an information extraction model according to an embodiment of the present application is shown;

[0034] Fig.11 A detailed flow chart of extracting first feature information of at least one information unit in the target image according to an embodiment of the present application is shown;

[0035] Fig.12 A block diagram of an information extraction device according to an embodiment of the present application is shown;

[0036] Fig.13 A block diagram of an information extraction device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0037] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0038] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0039] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0040] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0041] It should be noted that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0042] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the objects used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those shown or described.

[0043] The embodiments in this application involve technologies related to artificial intelligence, that is, fully automated processing of data (such as image data) is achieved through artificial intelligence. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0044] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solution of the embodiments of the present application can be applied is shown.

[0045] like Figure 1 As shown, the system architecture may include terminal devices (such as Figure 1 The embodiment of the present invention is a network 104 and a server 105. The network 104 is a medium for providing a communication link between the terminal device and the server 105. The network 104 may include various connection types, such as a wired communication link, a wireless communication link, etc.

[0046] In one embodiment of the present application, when a user needs to identify feature information reflected by at least one information unit in a target image, the user can send the target image to the server 105 through a terminal device. After acquiring the target image, the server 105 acquires an information extraction model, wherein the information extraction model is trained by training labels of at least one information unit in a training sample image on multiple feature types. Then, the server 105 extracts the first feature information of at least one information unit in the target image as target feature information through the information extraction model.

[0047] It should be noted that the first characteristic information proposed in the present application may be all characteristic information of at least one information unit, or may be partial characteristic information of at least one information unit. Specifically, the first characteristic information may include characteristic information of the information unit itself, or may include information on the characteristics of the relative relationship (e.g., relative position relationship) between information units, or may include both characteristic information of the information unit itself and information on the characteristics of the relative relationship between information units.

[0048] For example, taking the scenario of identifying formulas in images, the information unit may be a character unit in the formula. It can be understood that the first feature information of the character unit in the formula may include shape feature information of each character unit and / or relative position relationship feature information between character units.

[0049] In this embodiment, the information extraction model obtained by training at least one information unit in the training sample image on training labels of multiple feature types has the ability to accurately extract information, and by extracting the feature information reflected by at least one information unit in the target image through the information extraction model, the accuracy of information extraction can be improved.

[0050] It should be noted that the information extraction method provided in the embodiment of the present application can be executed by the server 105, and accordingly, the information extraction device is generally arranged in the server 105. However, in other embodiments of the present application, the terminal device can also have similar functions as the server, so as to execute the information extraction scheme provided in the embodiment of the present application.

[0051] It should also be noted that Figure 1 The number of terminal devices, networks and servers in the description is only for illustration. According to the implementation requirements, the server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0052] It should be explained that cloud computing as described above is a computing model that distributes computing tasks on a resource pool composed of a large number of computers, so that various application systems can obtain computing power, storage space and information services as needed. The network that provides resources is called a "cloud". The resources in the "cloud" are infinitely expandable in the eyes of users, and can be obtained at any time, used on demand, and expanded at any time. By establishing a cloud computing resource pool (referred to as a cloud platform, generally referred to as an IaaS (Infrastructure as a Service) platform, various types of virtual resources are deployed in the resource pool for external customers to choose to use. The cloud computing resource pool mainly includes: computing devices (virtualized machines, including operating systems), storage devices, and network devices.

[0053] The implementation details of the technical solution of the embodiment of the present application are described in detail below:

[0054] Figure 2 A flowchart of an information extraction method according to an embodiment of the present application is shown. The information extraction method can be executed by a device having a computing processing function, such as Figure 1 , or may be performed by the server 105 shown in Figure 1 Refer to the terminal device shown in Figure 2 As shown, the information extraction method at least includes steps 210 to 250, which are described in detail as follows:

[0055] In step 210, a target image is acquired, wherein the target image includes at least one information unit.

[0056] In the present application, the proposed information extraction scheme can be applied to information recognition scenarios of target objects in images, such as recognizing formulas in images, recognizing text in images, or recognizing certain specific patterns in images. Furthermore, the target object in the image can be composed of at least one information unit, such as a formula or text in an image can be composed of at least one character unit, and for example, certain specific patterns in an image can be composed of at least one graphic unit.

[0057] In the present application, the target image may be obtained by capturing a page area containing a target object in the interface, or the target image may be directly obtained locally.

[0058] Continue to refer to Figure 2 In step 230, an information extraction model is obtained, where the information extraction model is trained by training labels of at least one information unit in a training sample image on multiple feature types.

[0059] In the present application, information recognition of target objects in an image can be achieved through a pre-built information extraction model, and the information extraction model proposed in the present application can be obtained by training at least one information unit in a training sample image on training labels of multiple feature types.

[0060] In order to enable those skilled in the art to more clearly understand the information extraction model proposed in this application, the technical details of obtaining the information extraction model will be described in detail below:

[0061] In such Figure 2 In one embodiment of step 230, the information extraction model can be obtained as follows: Figure 3 Follow the steps shown.

[0062] See also Figure 3 , shows a detailed flow chart of obtaining an information extraction model according to an embodiment of the present application. Specifically, it includes steps 231 to 233:

[0063] Step 231, obtaining a training sample image.

[0064] Step 232, determining training labels on multiple feature types for at least one information unit in the training sample image, to obtain multiple training labels, wherein each training label is used to characterize feature information of the at least one information unit on a corresponding feature type.

[0065] Step 233, obtaining a model to be trained, and training the model to be trained using the training sample images and the multiple training labels to obtain the information extraction model.

[0066] In such Figure 3 In one embodiment of step 231, the acquisition of the training sample image may be performed as follows: Figure 4 Follow the steps shown.

[0067] See also Figure 4 , shows a detailed flow chart of obtaining a training sample image according to an embodiment of the present application. Specifically, it includes steps 2311 to 2313:

[0068] Step 2311, obtaining at least one frame of original training sample image.

[0069] Step 2312: scale the height or width of each frame of the original training sample image to a preset image height or preset image width, and scale the width or height of the original training sample image according to the scaling ratio of the height or the width to obtain a preprocessed training sample image.

[0070] Step 2313: Select a predetermined number of images from the pre-processed training sample images as the training sample images.

[0071] In this embodiment, the original training sample image may refer to the sample image initially collected, which may be one frame or multiple frames, such as one thousand frames or ten thousand frames, and the specific number of sample images may be determined according to actual conditions. Each frame of the sample image may include at least one information unit.

[0072] It should be noted that the sample images initially collected may have different sizes. In this case, the original sample images need to be preprocessed. Specifically, the height or width of each frame of the original training sample image can be scaled to a preset image height or preset image width, and the width or height of the original training sample image can be scaled according to the scaling ratio of the height or width.

[0073] For example, the original sample image includes 5 frames, of which the height and width are 15×30 for the first frame, 5×15 for the second frame, 20×30 for the third frame, 5×30 for the fourth frame, and 15×45 for the fifth frame. Take the example of enlarging the width of the sample image to the preset image width of 60. It can be seen that the width enlargement ratio of the first frame image is 2 times, the width enlargement ratio of the second frame image is 4 times, the width enlargement ratio of the third frame image is 2 times, the width enlargement ratio of the fourth frame image is 2 times, the width enlargement ratio of the first frame image is 2 times, and the width enlargement ratio of the first frame image is 4 / 3 times.

[0074] The image height is enlarged at the same magnification ratio as the image width, and the height and width sizes of the preprocessed training sample images are 30×60 for the first frame, 20×60 for the second frame, 40×60 for the third frame, 10×60 for the fourth frame, and 20×60 for the fifth frame.

[0075] After the preprocessed training sample images are obtained, a predetermined number of images may be selected from the preprocessed training sample images as the training sample images.

[0076] In the present application, the height or width of each frame of the original training sample image is scaled to a preset image height or preset image width, which has the advantage of speeding up model training in the subsequent process.

[0077] Furthermore, in Figure 3 In step 2313, the step of selecting a predetermined number of images from the pre-processed training sample images as the training sample images may be performed in the following manner:

[0078] First, the preprocessed training sample images may be sorted according to their width or height, and then a predetermined number of images that are connected in sorting are selected from the preprocessed training sample images as the training sample images.

[0079] For example, taking the preprocessed training sample images with height and width sizes of 30×60 for the first frame, 20×60 for the second frame, 40×60 for the third frame, 10×60 for the fourth frame, and 20×60 for the fifth frame as an example, the preprocessed training sample images can be sorted according to their heights to obtain the sorted fourth frame 10×60, fifth frame 20×60, second frame 20×60, first frame 30×60, and third frame 40×60.

[0080] Further, if the predetermined number is set to 4, the fourth frame, the fifth frame, the second frame, the first frame, or the fifth frame, the second frame, the first frame, and the third frame may be selected as the training sample images.

[0081] In the present application, when training a model based on training sample images, in order to optimize the training effect, the size of the training sample images can be kept consistent. Based on this, when the height or width of the training sample images is consistent, but the width or height of the images is inconsistent, the maximum width or maximum height of the training sample images can be used as a reference to supplement the width or height of the training sample images so that the height and width of the training sample images are completely consistent.

[0082] Prior to this, the preprocessed training sample images are sorted, and a predetermined number of sorted and connected images are selected from the preprocessed training sample images as the training sample images. This can avoid a large gap between the maximum width and the minimum width, or between the maximum height and the minimum height in the training sample images, thereby avoiding excessive invalid information supplementation to the width or height of the training sample images, thereby reducing the amount of computer calculations on invalid data during the model training process and saving computer resources.

[0083] In the present application, the information unit may include explicit information units and implicit information units, wherein the explicit unit may refer to an information unit contained in the image that can be directly seen, and the implicit information unit may refer to an information unit contained in the image that cannot be directly seen.

[0084] For example, taking the scenario of recognizing formulas or texts in images as an example, the individual character units that make up the formula or text are explicit information units, and the relative relationship between the individual character units (such as the positional relationship between the base and the exponent in the formula) is an implicit information unit.

[0085] In such Figure 3In one embodiment of step 232, the determining of training labels on multiple feature types for at least one information unit in the training sample image may be performed as follows: Figure 5 Follow the steps shown.

[0086] See also Figure 5 , shows a detailed flow chart of determining training labels on multiple feature types for at least one information unit in the training sample image according to an embodiment of the present application. Specifically including steps 2321 to 2322:

[0087] Step 2321, obtaining an information unit dictionary matching each feature type, wherein the information unit dictionary records at least feature vectors corresponding to all explicit information units.

[0088] Step 2322: Based on the feature vectors recorded in each information unit dictionary, construct a training label of the corresponding feature type for at least one information unit in the training sample image.

[0089] In order to enable those skilled in the art to better understand this embodiment, the following continues to take the application scenario of recognizing formulas or texts in images as an example. Specifically, in the example Figure 2 In step 232, the feature types in this application scenario may include positioning features of character units and shape features of character units.

[0090] Furthermore, the determining of training labels on multiple feature types for at least one information unit in the training sample image may include the following two types:

[0091] The first one is to determine a training label on a positioning feature for at least one character unit in the training sample image, where the positioning feature is used to at least characterize a relative relationship feature between the at least one character unit.

[0092] The second method is to determine a training label based on shape features for at least one character unit in the training sample image.

[0093] In this scenario, on the one hand, the information unit dictionary that matches the positioning feature may include feature vectors of all explicit information units and implicit information units that can be listed, for example, [1 0 0 0 … 0] is a feature vector of an explicit information unit (e.g., character unit "a"), and [0 1 0 0 … 0] is a feature vector of an implicit information unit (e.g., the relative relationship between two character units is the relationship between a base and an exponent). On the other hand, the information unit dictionary that matches the shape feature may only include feature vectors of all explicit information units that can be listed.

[0094] Furthermore, based on the feature vector recorded in each information unit dictionary, a training label of the corresponding feature type is constructed for at least one information unit in the training sample image. For example, the image includes a formula "a b ”. Among them, the eigenvector corresponding to the character unit “a” is [1 0 0 0 … 0], the eigenvector corresponding to the character unit “b” is [0 0 1 0 … 0], and the eigenvector corresponding to the relative relationship between the character unit “a” and the character unit “b” (that is, the relationship between the base and the exponent) is [0 1 0 0 … 0].

[0095] Based on the above situation, for the formula "a b The training labels of the information units in the training sample images of ” on the positioning features are “[1 0 0 0 …… 0]-[0 1 0 0 …… 0]-[0 0 1 0 …… 0]”, and the training labels of the information units in the training sample images of ” on the shape features are “[1 0 0 0 …… 0], [0 0 1 0 ……0]”.

[0096] It can be seen that the training label “[1 0 0 0 … 0]-[0 1 0 0 … 0]-[0 0 1 0 … 0]” mainly reflects the characteristics of the relative relationship between each character unit, and of course also reflects the shape characteristics of each character unit. The training label “[1 0 0 0 … 0], [0 0 1 0 … 0]” only reflects the shape characteristics of each character unit, and does not reflect the characteristics of the relative relationship between each character unit. Therefore, it is essentially a weak supervision label.

[0097] Based on the above scenario, it can be understood that, in the present application, the training labels constructed on various feature types for at least one information unit in the training sample image may include weakly supervised training labels.

[0098] In the present application, by determining training labels on multiple feature types including weakly supervised training labels for at least one information unit in the training sample image, the advantage is that the feature information of the training sample image can be segmented by the training labels on multiple feature types, giving the model more supervision information to achieve supervised training of the model with different types of supervision information, thereby improving the accuracy of model prediction.

[0099] In such Figure 3 In one embodiment of step 233, the acquisition of the model to be trained can be performed as follows: Figure 6 Follow the steps shown.

[0100] See also Figure 6, shows a detailed flowchart of obtaining a model to be trained according to an embodiment of the present application. Specifically including steps 2331 to 2332:

[0101] Step 2331, obtaining an encoder model and a decoder model through network structure search, wherein the encoder model is used to encode the image, and the decoder model is used to decode the features encoded by the encoder to obtain second feature information of at least one information unit in the image.

[0102] Step 2332: construct the model to be trained based on the encoder model and the decoder model.

[0103] Those skilled in the art can understand that the encoder model and the decoder model can essentially belong to a network structure model.

[0104] In this application, Neural Architecture Search (NAS) is an effective tool for generating and optimizing network structures. When the length and structure of the network are uncertain, a recurrent neural network is used as a controller to generate the fields of the network structure to build a sub-neural network. The accuracy of the sub-network after training is used as the controller reward signal, and the controller is updated by calculating the policy gradient, and the iterative cycle continues. In the next iteration, the controller will have a higher probability of proposing a high-accuracy network structure.

[0105] Based on this, the encoder model and decoder model are obtained through network structure search. The advantage is that a better encoder model and decoder model can be obtained, so that the constructed model to be trained has accurate learning ability.

[0106] In such Figure 3 In one embodiment of step 233, the training model to be trained is trained by using the training sample images and the multiple training labels to obtain the information extraction model. Figure 7 Follow the steps shown.

[0107] See also Figure 7 , shows a detailed flow chart of training the model to be trained using the training sample images and the multiple training labels according to an embodiment of the present application. Specifically including steps 2333 to 2334:

[0108] Step 2333, training the model to be trained by using the training sample images and the multiple training labels according to a predetermined number of training times, to obtain an information extraction reference model for the predetermined number of training times;

[0109] Step 2334, averaging the information extraction reference models trained a predetermined number of times to obtain the information extraction model.

[0110] In this embodiment, the model to be trained can be trained multiple times by using the training sample images and the multiple training labels, for example, 12 rounds of training, wherein each round of training will obtain an information extraction reference model, and then the obtained multiple information extraction reference models are averaged to obtain the information extraction model. The advantage of doing so is that the accuracy of the model can be improved, and the accuracy of the information extraction model in extracting feature information in the image can be enhanced.

[0111] In such Figure 7 In one embodiment of step 2333, the training of the model to be trained using the training sample images and the multiple training labels may be performed as follows: Figure 8 Follow the steps shown.

[0112] See also Figure 8 , shows a detailed flow chart of training the model to be trained by using the training sample images and the multiple training labels according to an embodiment of the present application. Specifically including steps 23331 to 23333:

[0113] Step 23331: input the training sample image into the model to be trained, and obtain third feature information of at least one information unit in the training sample image output by the model to be trained.

[0114] Step 23332: Based on the training label corresponding to each feature type, determine the error information existing on the corresponding feature type in the third feature information.

[0115] Step 23333, based on the error information existing on each feature type, the model parameters in the to-be-trained model are reversely updated respectively through the preset loss function corresponding to each feature type to obtain the information extraction reference model.

[0116] In this embodiment, after determining the error information existing on each feature type in the third feature information through the training labels corresponding to each feature type, the model parameters in the model to be trained can be reversely updated based on each error information and through a preset loss function to obtain the information extraction reference model.

[0117] In this embodiment, the model parameters in the model to be trained are reversely updated based on each error information. This has the advantage of shortening the training time of the information extraction reference model and improving the model training efficiency.

[0118] It should be noted that, in the present application, when the model parameters in the to-be-trained model are reversely updated through each error information, the loss parameters used may be different.

[0119] For example, continuing with the application scenario of identifying formulas or texts in images, for the error information existing in the positioning feature type, the model parameters in the model to be trained can be reversely updated through the CELoss loss function, and for the error information existing in the shape feature type, the model parameters in the model to be trained can be reversely updated through the BceWithLogits Loss loss function. The advantage of doing this is that, based on the error information existing in each feature type, the loss function suitable for each feature type is set respectively, which can enhance the training effect of the model, thereby enhancing the accuracy of the information extraction model in extracting feature information in the image.

[0120] In such Figure 7 In another embodiment of step 2333, the training of the model to be trained by using the training sample images and the multiple training labels can be performed as follows: Fig. 9 Follow the steps shown.

[0121] See also Fig. 9 , shows a detailed flow chart of training the model to be trained by using the training sample images and the multiple training labels according to an embodiment of the present application. Specifically including steps 23334 to 23338:

[0122] Step 23334, selecting a training label corresponding to the target feature type from the multiple training labels.

[0123] Step 23335: input the training sample image into the model to be trained, and obtain third feature information of at least one information unit in the training sample image output by the model to be trained.

[0124] Step 23336: Based on the training label corresponding to the target feature type, determine the target error information existing on the target feature type in the third feature information.

[0125] Step 23337, based on the target error information, reversely update the model parameters in the model to be trained through a preset loss function corresponding to the target feature type to obtain an intermediate information extraction reference model.

[0126] Step 23338, uses the intermediate information extraction reference model as a new model to be trained, and re-executes the step of selecting a training label corresponding to a target feature type from the multiple training labels until all the labels in the multiple training labels are selected to obtain the information extraction reference model.

[0127] In this embodiment, it can be understood that, in fact, the model to be trained is trained through each training label in turn based on multiple training labels through iterative training, that is, after the model to be trained is trained through a training label corresponding to a feature type, the model to be trained that was trained last time is retrained through a training label corresponding to another feature type. The advantage of doing so is that the training effect of the model can also be enhanced, thereby enhancing the accuracy of the information extraction model in extracting feature information in the image.

[0128] In order to enable those skilled in the art to better understand the information extraction model mentioned in this application, the following will be combined with Fig.10 Provide explanation.

[0129] See also Fig.10 , showing a schematic diagram of the framework of an information extraction model according to an embodiment of the present application.

[0130] like Fig.10 As shown, the information extraction model is composed of an encoder 1002 and a decoder 1004, wherein, in the process of extracting feature information of at least one information unit in an image, the encoder 1002 first encodes the image data 1001 to obtain encoded feature information 1003, and then the decoder 1004 decodes the encoded feature information 1003 to obtain feature information 1005 of at least one information unit in the image.

[0131] Continue to refer to Figure 2 In step 250, first feature information of at least one information unit in the target image is extracted through the information extraction model as target feature information.

[0132] In such Figure 2 In one embodiment of step 250, the first feature information of at least one information unit in the target image is extracted by the information extraction model as follows: Fig.11 Follow the steps shown.

[0133] See also Fig.11 , shows a detailed flow chart of extracting the first feature information of at least one information unit in the target image according to an embodiment of the present application. Specifically, it includes steps 251 to 254:

[0134] Step 251: for each target information unit in the at least one information unit, determine all information units and part of information units arranged before the target information unit.

[0135] Step 252: predicting first reference characteristic information of the target information unit based on the characteristic information of all the information units.

[0136] Step 253: predict second reference characteristic information of the target information unit based on the characteristic information of the partial information unit.

[0137] Step 254: Determine the first characteristic information based on the first reference characteristic information and the second reference characteristic information of each information unit in the at least one information unit.

[0138] In this embodiment, the various information units in the target image have certain logical relative relationships, such as a relative relationship in spatial logic and a relative relationship in reading order logic.

[0139] In this implementation, the entire information unit arranged before the target information unit may refer to the entire information unit arranged before the target information unit in terms of spatial logic or in terms of reading order logic. The partial information unit arranged before the target information unit may refer to the partial information unit arranged before the target information unit in terms of spatial logic or in terms of reading order logic.

[0140] In this embodiment, the application scenario of recognizing formulas or texts in an image is continued as an example. For example, the target image includes the formula "a b c d ", if it is necessary to predict the first characteristic information of the target information unit "d", on the one hand, the first reference characteristic information of "d" can be predicted based on the characteristic information of all the information units "a", "b", and "c" arranged before the target information unit. On the other hand, the second reference characteristic information of "d" can be predicted based on the characteristic information of the partial information unit "c" arranged before the target information unit. Then, based on the first reference characteristic information and the second reference characteristic information, the first characteristic information is determined.

[0141] In this embodiment, when extracting the first feature information of at least one information unit in the target image, by considering the feature information of all information units and the feature information of some information units arranged before each target information unit, global information and local information can be integrated at the same time, thereby making the prediction of the information extraction model more accurate.

[0142] In the present application, continuing to take the application scenario of formula recognition in an image as an example, the information unit may include a character unit, and the at least one information unit may constitute one or more formulas containing the character unit.

[0143] Specifically, in Figure 2 In an embodiment after step 250, that is, after the first feature information of at least one information unit in the target image is extracted as the target feature information by the information extraction model, the following scheme may also be performed:

[0144] Based on the target feature information, one or more formulas in the target image are edited into a formula editing area.

[0145] Specifically, in this application scenario, when editing a document, the user can capture the formula image that needs to be edited on the web page, and then, based on the information extraction scheme proposed in this application, extract the first feature information of at least one character unit in the formula image to obtain the target feature information, and then edit the formula in the formula image to the formula editing area based on the target feature information. It can be seen that the information extraction method proposed in this application can bring great convenience and excellent user experience to users in the formula editing process.

[0146] The technical solution provided in the present application uses an information extraction model obtained by training at least one information unit in a training sample image on a plurality of feature types of training labels to extract first feature information of at least one information unit in the target image. Since the supervisory feature information of the information unit in the image on a plurality of feature types is taken into account by training at least one information unit in the training sample image on a plurality of feature types of training labels, the information extraction model obtained by training has the ability to accurately extract information, and by extracting the feature information reflected by at least one information unit in the target image through the information extraction model, the accuracy of information extraction can be improved.

[0147] The following describes an apparatus embodiment of the present application, which can be used to execute the information extraction method in the above-mentioned embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the above-mentioned embodiment of the information extraction method of the present application.

[0148] Fig.12 A block diagram of an information extraction device according to an embodiment of the present application is shown.

[0149] Reference Fig.12 As shown, according to an embodiment of the present application, an information extraction device 1200 includes: a first acquisition unit 1201 , a second acquisition unit 1202 , and an extraction unit 1203 .

[0150] Among them, the first acquisition unit 1201 is used to acquire a target image, which includes at least one information unit; the second acquisition unit 1202 is used to acquire an information extraction model, which is obtained by training at least one information unit in a training sample image on training labels of multiple feature types; the extraction unit 1203 is used to extract first feature information of at least one information unit in the target image as target feature information through the information extraction model.

[0151] In some embodiments of the present application, based on the aforementioned scheme, the second acquisition unit 1202 includes: a third acquisition unit, used to acquire a training sample image; a determination unit, used to determine training labels on multiple feature types for at least one information unit in the training sample image, and obtain multiple training labels, wherein each training label is used to characterize feature information of the at least one information unit on a corresponding feature type; a fourth acquisition unit, used to acquire a model to be trained, and train the model to be trained using the training sample image and the multiple training labels to obtain the information extraction model.

[0152] In some embodiments of the present application, based on the aforementioned scheme, the third acquisition unit is configured to: acquire at least one frame of original training sample image; scale the height or width of each frame of the original training sample image to a preset image height or a preset image width, and scale the width or height of the original training sample image according to the scaling ratio of the height or the width to obtain a preprocessed training sample image; select a predetermined number of images from the preprocessed training sample images as the training sample images.

[0153] In some embodiments of the present application, based on the aforementioned scheme, the third acquisition unit is configured to: sort the preprocessed training sample images according to the width or height of the preprocessed training sample images; and select a predetermined number of images that are sorted and connected from the preprocessed training sample images as the training sample images.

[0154] In some embodiments of the present application, based on the aforementioned scheme, the information unit includes explicit information units and implicit information units, and the determination unit is configured to: obtain an information unit dictionary matching each feature type, wherein the information unit dictionary records at least feature vectors corresponding to all explicit information units; based on the feature vectors recorded in each information unit dictionary, construct a training label on the corresponding feature type for at least one information unit in the training sample image.

[0155] In some embodiments of the present application, based on the aforementioned scheme, the fourth acquisition unit is configured to: acquire an encoder model and a decoder model through a network structure search, the encoder model is used to encode an image, and the decoder model is used to decode the features encoded by the encoder to obtain second feature information of at least one information unit in the image; based on the encoder model and the decoder model, construct the model to be trained.

[0156] In some embodiments of the present application, based on the aforementioned scheme, the fourth acquisition unit is configured as follows: according to a predetermined number of training times, the model to be trained is trained using the training sample images and the multiple training labels to obtain the information extraction reference model for the predetermined number of training times; and the information extraction reference model for the predetermined number of training times is averaged to obtain the information extraction model.

[0157] In some embodiments of the present application, based on the aforementioned scheme, the fourth acquisition unit is configured to: input the training sample image into the model to be trained, and obtain the third feature information of at least one information unit in the training sample image output by the model to be trained; based on the training label corresponding to each feature type, determine the error information existing in the corresponding feature type in the third feature information; based on the error information existing in each feature type, reversely update the model parameters in the model to be trained respectively through the preset loss function corresponding to each feature type to obtain the information extraction reference model.

[0158] In some embodiments of the present application, based on the aforementioned scheme, the fourth acquisition unit is configured to: select a training label corresponding to a target feature type from the multiple training labels; input the training sample image into the model to be trained, and obtain the third feature information of at least one information unit in the training sample image output by the model to be trained; based on the training label corresponding to the target feature type, determine the target error information existing on the target feature type in the third feature information; based on the target error information, reversely update the model parameters in the model to be trained through a preset loss function corresponding to the target feature type to obtain an intermediate information extraction reference model; use the intermediate information extraction reference model as a new model to be trained, and re-execute the step of selecting a training label corresponding to a target feature type from the multiple training labels until all the labels in the multiple training labels are selected to obtain the information extraction reference model.

[0159] In some embodiments of the present application, based on the aforementioned scheme, the information unit includes a character unit, and the determination unit is configured to: determine a training label on a positioning feature for at least one character unit in the training sample image, wherein the positioning feature is used to at least characterize a relative relationship feature between the at least one character unit; determine a training label on a shape feature for at least one character unit in the training sample image.

[0160] In some embodiments of the present application, based on the aforementioned scheme, the extraction unit 1203 is configured to: for each target information unit in the at least one information unit, determine all the information units and some of the information units arranged before the target information unit; based on the feature information of all the information units, predict the first reference feature information of the target information unit; based on the feature information of the some of the information units, predict the second reference feature information of the target information unit; based on the first reference feature information and the second reference feature information of each information unit in the at least one information unit, determine the first feature information.

[0161] In some embodiments of the present application, based on the aforementioned scheme, the information unit includes a character unit, and the at least one information unit constitutes one or more formulas containing the character unit. The device also includes an editing unit, which is used to extract the first feature information of at least one information unit in the target image as target feature information through the information extraction model, and then edit one or more formulas in the target image to a formula editing area based on the target feature information.

[0162] As another aspect, an embodiment of the present application also provides another information extraction device, including a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing the information extraction method described in the above embodiments.

[0163] Fig.13 A block diagram of an information extraction device according to an embodiment of the present application is shown. For example, the device 1300 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0164] Reference Fig.13 , the device 1300 may include one or more of the following components: a processing component 1302 , a memory 1304 , a power component 1306 , a multimedia component 1308 , an audio component 1310 , an input / output (I / O) interface 1312 , a sensor component 1314 , and a communication component 1316 .

[0165] The processing component 1302 generally controls the overall operation of the device 1300, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 1302 may include one or more processors 1320 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 1302 may include one or more modules to facilitate interaction between the processing component 1302 and other components. For example, the processing component 1302 may include a multimedia module to facilitate interaction between the multimedia component 1308 and the processing component 1302.

[0166] The memory 1304 is configured to store various types of data to support operations on the device 1300. Examples of such data include instructions for any application or method operating on the device 1300, contact data, phone book data, messages, pictures, videos, etc. The memory 1304 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0167] The power supply component 1306 provides power to the various components of the device 1300. The power supply component 1306 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 1300.

[0168] The multimedia component 1308 includes a screen that provides an output interface between the device 1300 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1308 includes a front camera and / or a rear camera. When the device 1300 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0169] The audio component 1310 is configured to output and / or input audio signals. For example, the audio component 1310 includes a microphone (MIC), and when the device 1300 is in an operating mode, such as a call mode, a recording mode, and a voice information processing mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 1304 or sent via the communication component 1316. In some embodiments, the audio component 1310 also includes a speaker for outputting audio signals.

[0170] I / O interface 1312 provides an interface between processing component 1302 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0171] The sensor assembly 1314 includes one or more sensors for providing various aspects of the status assessment of the device 1300. For example, the sensor assembly 1314 can detect the open / closed state of the device 1300, the relative positioning of components, such as the display and keypad of the device 1300, the sensor assembly 1314 can also search for changes in the position of the device 1300 or a component of the device 1300, the presence or absence of user contact with the device 1300, the orientation or acceleration / deceleration of the device 1300, and the temperature change of the device 1300. The sensor assembly 1314 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 1314 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1314 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0172] The communication component 1316 is configured to facilitate wired or wireless communication between the device 1300 and other devices. The device 1300 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 1316 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1316 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency information processing (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0173] In an exemplary embodiment, the device 1300 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above-mentioned methods.

[0174] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1304 including instructions, and the instructions can be executed by the processor 1320 of the device 1300 to complete the above-mentioned information extraction method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0175] As another aspect, the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the information extraction method described in the above embodiment.

[0176] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The computer-readable storage medium stores at least one program code, which is loaded and executed by the processor of the device to implement the operations performed by the information extraction method described in the above embodiment.

[0177] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0178] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0179] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0180] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An information extraction method, characterized in that: The method comprises: Acquire a target image, wherein the target image includes at least one information unit; Acquire an information extraction model, wherein the information extraction model is obtained by training at least one information unit in a training sample image on a plurality of feature types; Through the information extraction model, for each target information unit in the at least one information unit, all information units and some information units arranged before the target information unit are determined; based on the feature information of all information units, the first reference feature information of the target information unit is predicted; based on the feature information of the some information units, the second reference feature information of the target information unit is predicted; based on the first reference feature information and the second reference feature information of each information unit in the at least one information unit, the first feature information is determined as the target feature information.

2. The method according to claim 1, characterized in that: The obtaining of the information extraction model comprises: Obtain training sample images; Determine a training label on multiple feature types for at least one information unit in the training sample image to obtain multiple training labels, wherein each training label is used to represent feature information of the at least one information unit on a corresponding feature type; A model to be trained is obtained, and the model to be trained is trained using the training sample images and the multiple training labels to obtain the information extraction model.

3. The method according to claim 2, wherein obtaining a training sample image comprises: Obtain at least one frame of original training sample image; Scaling the height or width of each frame of the original training sample image to a preset image height or a preset image width, and scaling the width or height of the original training sample image according to the scaling ratio of the height or the width to obtain a preprocessed training sample image; A predetermined number of images are selected from the preprocessed training sample images as the training sample images.

4. The method according to claim 3, characterized in that The step of selecting a predetermined number of images from the pre-processed training sample images as the training sample images comprises: Sorting the preprocessed training sample images according to the width or height of the preprocessed training sample images; A predetermined number of images that are sequentially connected are selected from the preprocessed training sample images as the training sample images.

5. The method according to claim 2, wherein the information unit comprises an explicit information unit and an implicit information unit, and the step of determining a training label on multiple feature types for at least one information unit in the training sample image comprises: Acquire an information unit dictionary matching each feature type, wherein the information unit dictionary records at least feature vectors corresponding to all explicit information units; Based on the feature vectors recorded in each information unit dictionary, a training label of a corresponding feature type is constructed for at least one information unit in the training sample image.

6. The method according to claim 2, characterized in that The step of obtaining the model to be trained includes: Acquire an encoder model and a decoder model through network structure search, wherein the encoder model is used to encode the image, and the decoder model is used to decode the features encoded by the encoder to obtain second feature information of at least one information unit in the image; Based on the encoder model and the decoder model, the model to be trained is constructed.

7. The method according to claim 2, characterized in that The step of training the model to be trained by using the training sample images and the multiple training labels to obtain the information extraction model includes: According to a predetermined number of training times, the model to be trained is trained using the training sample images and the multiple training labels to obtain an information extraction reference model for the predetermined number of training times; The information extraction reference model trained a predetermined number of times is averaged to obtain the information extraction model.

8. The method according to claim 7, characterized in that The step of training the model to be trained by using the training sample images and the multiple training labels includes: Inputting the training sample image into the model to be trained, and obtaining third feature information of at least one information unit in the training sample image output by the model to be trained; Based on the training label corresponding to each feature type, determining error information existing on the corresponding feature type in the third feature information; Based on the error information existing on each feature type, the model parameters in the model to be trained are reversely updated respectively through the preset loss function corresponding to each feature type to obtain the information extraction reference model.

9. The method according to claim 7, characterized in that: The step of training the model to be trained by using the training sample images and the multiple training labels includes: Selecting a training label corresponding to a target feature type from the multiple training labels; Inputting the training sample image into the model to be trained, and obtaining third feature information of at least one information unit in the training sample image output by the model to be trained; Based on the training label corresponding to the target feature type, determining target error information existing on the target feature type in the third feature information; Based on the target error information, the model parameters in the to-be-trained model are reversely updated by a preset loss function corresponding to the target feature type to obtain an intermediate information extraction reference model; The intermediate information extraction reference model is used as a new model to be trained, and the step of selecting a training label corresponding to a target feature type from the multiple training labels is re-executed until all labels in the multiple training labels are selected to obtain the information extraction reference model.

10. The method according to any one of claims 2 to 9, characterized in that: The information unit includes a character unit, and determining a training label on multiple feature types for at least one information unit in the training sample image includes: Determine a training label on a positioning feature for at least one character unit in the training sample image, wherein the positioning feature is used to at least characterize a relative relationship feature between the at least one character unit; A training label based on shape features is determined for at least one character unit in the training sample image.

11. The method according to claim 1, characterized in that The information unit includes a character unit, the at least one information unit constitutes one or more formulas including the character unit, and after determining the first feature information as target feature information based on the first reference feature information and the second reference feature information of each information unit in the at least one information unit, the method further includes: Based on the target feature information, one or more formulas in the target image are edited into a formula editing area.

12. An information extraction device, characterized in that: The device comprises: A first acquisition unit is used to acquire a target image, wherein the target image includes at least one information unit; A second acquisition unit is used to acquire an information extraction model, wherein the information extraction model is obtained by training at least one information unit in a training sample image on a plurality of feature types; The extraction unit is used to determine, for each target information unit in the at least one information unit, all information units and some information units arranged before the target information unit; predict the first reference feature information of the target information unit based on the feature information of all information units; predict the second reference feature information of the target information unit based on the feature information of the some information units; and determine the first feature information as the target feature information based on the first reference feature information and the second reference feature information of each information unit in the at least one information unit.

13. An information extraction device, characterized in that: The invention comprises a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, and the one or more programs include instructions for performing the information extraction method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one program code, and the at least one program code is loaded and executed by a processor to implement the operations performed by the information extraction method according to any one of claims 1 to 11.

Images