Image information extraction method and device and electronic equipment

Abstract

The invention provides an image information extraction method and device and electronic equipment. Relates to the technical field of image processing. The method comprises the following steps: acquiring image text information of a to-be-processed text image; the image text information comprises a to-be-processed text image and at least one group of text information contained in the to-be-processed text image; any group of text information comprises text content information and text position information corresponding to the text content information; performing information extraction on the image text information by adopting the trained information extraction model to obtain an information extraction result of the to-be-processed text image; the information extraction result comprises at least one key information key in the to-be-processed text image and at least one key information value corresponding to each key information key. According to the method, the technical problem of low accuracy of the extraction result in the prior art is solved, and the accuracy of the image information extraction result is improved by extracting all the key information values corresponding to the key information keys.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image information extraction method, device and electronic device. Background Art

[0002] In practical applications, information often exists in the form of images, such as scanned documents, tickets, forms, ID cards, driver's licenses, etc. These images contain a large amount of structured information. However, manually extracting this structured information is not only time-consuming and labor-intensive, but also prone to errors.

[0003] With the development of deep learning and computer vision technology, image information extraction methods based on deep learning have gradually become a research hotspot. For example, many researchers have begun to study intelligent structured image information extraction models to help users extract image information as quickly as possible, such as key text and table information in images.

[0004] However, in some images, one key information key often corresponds to multiple key information values. However, most of the existing image information extraction models cannot accurately extract all the key information values, resulting in low accuracy of the extraction results. Summary of the invention

[0005] The present application provides an image information extraction method, device and electronic device to solve the technical problem of low accuracy of extraction results in the prior art, and improve the accuracy of image information extraction results by extracting all key information values ​​corresponding to key information keys.

[0006] In a first aspect, the present application provides an image information extraction method, comprising:

[0007] Acquire image text information of a text image to be processed; the image text information includes the text image to be processed and at least one set of text information contained in the text image to be processed; any set of text information includes text content information and text position information corresponding to the text content information;

[0008] Using the trained information extraction model to extract information from the image text information, obtaining an information extraction result of the text image to be processed; the information extraction result includes at least one key information key in the text image to be processed and at least one key information value corresponding to each key information key;

[0009] Among them, the information extraction model includes an encoding module, a first decoding module and a second decoding module; the encoding module is used to obtain a multimodal fusion vector based on the information encoding result of the image text information; the first decoding module is used to implement information key extraction based on the multimodal fusion vector to obtain the key information key; the second decoding module is used to implement information value extraction based on the multimodal fusion vector and the key information key to obtain the key information value.

[0010] In an optional implementation, extracting information keys according to the multimodal fusion vector to obtain the key information keys includes:

[0011] According to the multimodal fusion vector and the preset extraction prompt word, multiple rounds of key information key extraction operations are repeatedly performed until the extraction stop condition is met, and all the extracted key information keys are determined as at least one key information key in the text image to be processed; wherein,

[0012] If the current round is the first extraction, the extraction prompt word is the starting mark word in the word segmentation result corresponding to the text content information; if the current round is not the first extraction, the extraction prompt word is the concatenation result of all key information keys extracted in the previous rounds;

[0013] The extraction stop condition includes: the extracted key information key is a preset extraction end word, or the predicted number of all key information keys meets a preset threshold;

[0014] The extraction process of any round includes:

[0015] Performing low-dimensional vector conversion processing on the extracted prompt word to obtain a corresponding prompt word vector;

[0016] Obtaining an information key prediction vector for representing a key information key to be determined in the current round according to the prompt word vector and the multimodal fusion vector;

[0017] A first probability distribution of the information key prediction vector is determined, and a key information key of the current round is determined according to the first probability distribution.

[0018] In an optional implementation, determining a first probability distribution of the information key prediction vector, and determining the key information key of the current round according to the first probability distribution includes:

[0019] Processing the information key prediction vector according to a preset self-attention weight matrix to obtain a processed information key prediction vector;

[0020] Normalizing the processed information key prediction vector to obtain a probability matrix corresponding to the processed information key prediction vector;

[0021] The key information key of the current round is determined according to the probability matrix and the word segmentation result corresponding to the text content information.

[0022] In an optional implementation, extracting information value according to the multimodal fusion vector and the key information key to obtain the key information value includes:

[0023] According to the multimodal fusion vector and each of the key information keys, multiple rounds of key information value extraction operations are repeatedly performed until the extraction stops when the extraction stop condition is met, and all the extracted key information values ​​are determined as at least one key information value corresponding to each of the key information keys; wherein,

[0024] The extraction stop condition includes: the extracted key information value is a preset extraction end word, or the predicted number of all key information values ​​meets a preset threshold;

[0025] The extraction process of any round includes:

[0026] Determine the image position of the image area where the key information key is located, and perform low-dimensional vector conversion on the image position to obtain a corresponding image position vector;

[0027] Determining, according to the image position vector, an information key self-attention vector corresponding to a splicing result of each of the key information keys;

[0028] Obtaining an information value prediction vector for characterizing a key information value to be determined in a current round according to the multimodal fusion vector and the information key self-attention vector;

[0029] The key information value of the current round is determined according to the probability distribution of the information value prediction vector.

[0030] In an optional implementation, determining the key information value of the current round according to the probability distribution of the information value prediction vector includes:

[0031] Determine a second probability distribution of the information value prediction vector and a third probability distribution of the product of the information value prediction vector and the word vector corresponding to the text content information;

[0032] Determine, according to the multimodal fusion vector and the information value prediction vector, a first weight and a second weight corresponding to the second probability distribution and the third probability distribution, respectively;

[0033] The key information value of the current round is determined according to a weighted result of the first weight and the second probability distribution, and a weighted result of the second weight and the third probability distribution.

[0034] In an optional implementation, the information encoding result includes an image vector, a text word vector, and a text position vector;

[0035] According to the information encoding result of the image text information, a multimodal fusion vector is obtained, including:

[0036] Determine an image self-attention vector corresponding to the image vector, a text word self-attention vector corresponding to the text word vector, and a text position self-attention vector corresponding to the text position vector;

[0037] A multimodal fusion vector corresponding to the text image to be processed is determined according to the image self-attention vector, the text word self-attention vector and the text position self-attention vector.

[0038] In an optional implementation, before obtaining the multimodal fusion vector according to the information encoding result of the image text information, the method further includes:

[0039] Performing low-dimensional vector conversion processing on the text image to be processed to obtain a corresponding image feature vector;

[0040] Performing position coding processing on each pixel point in the text image to be processed to obtain a corresponding image position vector;

[0041] An image vector corresponding to the text image to be processed is obtained according to the image feature vector and the image position vector.

[0042] In an optional implementation, before obtaining the multimodal fusion vector according to the information encoding result of the image text information, the method further includes:

[0043] For any group of text information, segment the text content information to obtain multiple phrases;

[0044] For any word group, the word group is converted into a low-dimensional vector to obtain a corresponding word vector, and the word vector is positionally encoded to obtain a corresponding word position vector;

[0045] Obtaining text word vectors corresponding to each of the text information according to the word vectors and the word position vectors respectively corresponding to each of the word groups in the text information;

[0046] Determining word position information corresponding to each of the phrases in the at least one group of text information according to each of the text position information;

[0047] The word position information is converted into a low-dimensional vector to obtain a text position vector corresponding to each text information.

[0048] In a second aspect, the present application provides an image information extraction device, comprising:

[0049] An information acquisition module, used to acquire image text information of a text image to be processed; the image text information includes the text image to be processed and at least one set of text information contained in the text image to be processed; any set of text information includes text content information and text position information corresponding to the text content information;

[0050] An information extraction module, configured to extract information from the image text information using a trained information extraction model to obtain an information extraction result of the text image to be processed; the information extraction result includes at least one key information key in the text image to be processed and at least one key information value corresponding to each key information key;

[0051] Among them, the information extraction model includes an encoding module, a first decoding module and a second decoding module; the encoding module is used to obtain a multimodal fusion vector based on the information encoding result of the image text information; the first decoding module is used to implement information key extraction based on the multimodal fusion vector to obtain the key information key; the second decoding module is used to implement information value extraction based on the multimodal fusion vector and the key information key to obtain the key information value.

[0052] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0053] The memory stores computer-executable instructions;

[0054] The processor executes the computer-executable instructions stored in the memory to implement the method according to the first aspect.

[0055] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect.

[0056] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.

[0057] The image information extraction method provided by the present application obtains image text information of a text image to be processed; the image text information includes the text image to be processed and at least one group of text information contained in the text image to be processed; any group of text information includes text content information and text position information corresponding to the text content information; a trained information extraction model is used to extract information from the image text information to obtain an information extraction result of the text image to be processed; the information extraction result includes at least one key information key in the text image to be processed and at least one key information value corresponding to each key information key; wherein the information extraction model includes an encoding module, a first decoding module and a second decoding module; the encoding module is used to obtain a multimodal fusion vector according to the information encoding result of the image text information; the first decoding module is used to implement information key extraction according to the multimodal fusion vector, The key information key is obtained; the second decoding module is used to extract the information value according to the multimodal fusion vector and the key information key to obtain the key information value; in the above scheme, the image information in the image text information of the processed text image, the text content information contained in the image, and the local position information corresponding to the text content information are respectively converted into low-dimensional vectors, and the obtained vectors are fused to obtain the multimodal fusion vector; sampling the multimodal fusion vector to extract information from the image can effectively mine the relationship between the image and the text, so as to achieve more accurate extraction results; and in the subsequent decoding process, that is, when extracting information, the key information key is first extracted according to the multimodal fusion vector, and then when extracting the key information value, the extracted key information key is also fused on the basis of the above-mentioned multiple information, which makes up for the information loss problem that may exist when extracting the information value, so as to improve the accuracy of the extraction result. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0059] Figure 1 An application scenario diagram of the image information extraction method provided in this application;

[0060] Figure 2 A schematic diagram of a flow chart of an image information extraction method provided in an embodiment of the present application;

[0061] Figure 3 A schematic diagram of the structure of an information extraction model provided in an embodiment of the present application;

[0062] Figure 4 This is a schematic diagram of the structure of an image information extraction device provided by the present application according to an embodiment;

[0063] Figure 5 It is a block diagram of an electronic device provided in an embodiment of the present application.

[0064] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0065] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0066] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data or user data involved shall comply with the provisions of relevant laws and regulations and shall not violate public order and good morals.

[0067] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0068] In practical applications, information often exists in the form of images, such as scanned documents, tickets, forms, ID cards, driver's licenses, etc. These images contain a lot of structured information, such as name, date, address, amount, etc. Image information extraction aims to extract structured key-value pair information from images.

[0069] In an optional implementation, the image can be matched with the template through a predefined template to extract the key-value pair information. However, the implementation of the above method is highly dependent on the format and layout of the image, and needs to be redefined when the template changes.

[0070] In another optional implementation, the optical character recognition (OCR) method can be used to recognize the text content in the image, and then the key-value pairs can be extracted through the natural language processing (NPL) technology. The above implementation can realize information extraction of images of different formats and layouts by training a variety of different images. However, the accuracy of OCR is greatly affected by factors such as image quality, font, language, etc., resulting in low accuracy of the extraction result.

[0071] In another optional implementation, the joint features of the image and text can also be used for information extraction, usually by using a multimodal learning method to directly extract key-value pair information from the image. The above implementation can improve the accuracy of the extraction results by combining multiple information during the extraction process. However, in some images, there are often situations where one key information key corresponds to multiple key information values. In the above case, most of the existing image information extraction methods cannot accurately extract all the key information values ​​corresponding to the key information key, resulting in low accuracy of the extraction results.

[0072] For example, in an image, for the key information key "name" contained in the image, there are multiple key information values ​​corresponding to it in the image - "small A", "small B" and "small C". In this case, when extracting the key information value corresponding to "name", the existing image information extraction method can only extract "small A", but cannot extract "small B" and "small C", resulting in poor processing effect on complex and diverse text images.

[0073] The image information extraction method provided by the present application is intended to solve the above technical problems of the prior art. Specifically, the image and the text information contained in the image are fused and encoded during the encoding process, which can effectively mine the association between the image and the image text, and extract the information key based on the encoding result to improve the accuracy of the extraction result; further, when extracting the information value, the key information key and the key information value are jointly extracted, which increases the complementarity between the key-value pairs in the extraction process, and further associates the text with the information key and the previously extracted information value, which makes up for the information loss problem that may exist when only the key information key is used to extract the key information value, and realizes the extraction of multiple key information values ​​corresponding to the key information key, thereby improving the extraction effect of the text image.

[0074] Figure 1An application scenario diagram of the image information extraction method provided in the present application. The technical solution provided in the present application can be applied to the application scenario of information extraction from text images in the process of financial and bill data processing. Banks and financial institutions need to process a large number of documents such as bills, invoices and contracts, which are usually digitally archived by scanning or photographing. The scanned or photographed images may have inaccurate extraction results during the information extraction process, affecting the extraction and verification of information. The image information extraction method provided in the present application uses a series of complex algorithms to accurately extract key-value pair information from text images, which can ensure the accuracy and reliability of the information extraction results, and facilitate improving the accuracy and efficiency of subsequent processing of financial business.

[0075] For ease of understanding, the following combination Figure 1 The application scenarios to which the embodiments of the present application are applicable are described. Figure 1 The image extraction technology provided in this application involves an image acquisition device 11, a text information acquisition device 12 and an information extraction device 13.

[0076] Specifically, the text image containing text information such as financial bills is photographed or scanned by the image acquisition device 11 to obtain the text image to be processed. The text image to be processed is transmitted to the text information acquisition device 12 for text information extraction processing, and the text image to be processed is transmitted to the information extraction device 13 for information extraction processing.

[0077] In the text information acquisition device 12, the text information in the text image to be processed is extracted by OCR technology to obtain at least one group of text information; wherein each group of text information includes text content information and text position information corresponding to the text content information. The extracted text information is transmitted to the information extraction device 13 for information extraction processing.

[0078] In the information extraction device 13, when a text image to be processed and at least one group of text information contained in the text image to be processed are received, the text image to be processed and each group of text information are used as the image text information corresponding to the text image to be processed; a trained information extraction model is obtained, and the image text information is extracted using the information extraction model to obtain the information extraction result of the text image to be processed; the information extraction result includes at least one key information key in the text image to be processed and at least one key information value corresponding to each key information key; wherein the information extraction model includes an encoding module, a first decoding module and a second decoding module; the encoding module is used to obtain a multimodal fusion vector according to the information encoding result of the image text information; the first decoding module is used to implement information key extraction according to the multimodal fusion vector to obtain the key information key; the second decoding module is used to implement information value extraction according to the multimodal fusion vector and the key information key to obtain the key information value.

[0079] In the above scheme, the information of the image, the text content information contained in the image, and the text position information corresponding to the text content information are integrated to extract information from the image, which can effectively explore the relationship between the image and the text, and achieve more accurate extraction results; and when extracting information, the key information keys are first extracted through the fusion results of multiple information, and then when extracting the key information values, the extracted key information keys are also integrated on the basis of the above-mentioned multiple information, which makes up for the possible information loss problem when extracting information values, and improves the accuracy of the extraction results.

[0080] It should be understood that the image information extraction technology provided in this application can be applied to various scenarios such as file management, mobile office, educational examinations and legal contract management in addition to financial document processing scenarios. The embodiments of this application do not specifically limit the application scenarios of the image information extraction technology provided.

[0081] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0082] Figure 2 The following is a flow chart of a method for extracting image information provided in an embodiment of the present application. The method can be executed by an image information extraction device, which can be a server or an electronic device. The following description takes an electronic device as an example. The method in this embodiment can be implemented by software, hardware, or a combination of software and hardware, such as Figure 2As shown, the method includes the following steps.

[0083] S201, obtaining image text information of a text image to be processed.

[0084] In this application, the text image to be processed can be understood as a digital form of a paper document, including text, charts, images, signatures, etc. The way to obtain the document image includes scanners, digital cameras, mobile phone cameras, etc.

[0085] When obtaining the text image to be processed, optical character recognition technology can be used to identify and process the text image to be processed to obtain at least one group of text information contained in the image; wherein any group of text information can include text content information and text position information corresponding to the text content information.

[0086] Exemplarily, the text content information may be understood as a group of characters recognized in the image; the text position information may be understood as the coordinate position data corresponding to the group of characters in the image.

[0087] The text image to be processed and at least one set of recognized text information are used as image text information of the text image to be processed, so as to perform structured information extraction on the text image to be processed based on the image text information and obtain key-value pair information contained in the image.

[0088] S202: extract information from the image text information using the trained information extraction model to obtain an information extraction result of the text image to be processed.

[0089] In the present application, the information extraction model may be a machine learning model. For example, an information extraction model constructed by a neural network and obtained by training the key-value pair information contained in the historical text image data and the image. Of course, the information extraction model may also be other types of models, which are not specifically limited.

[0090] It should be understood that the image text information of the text image to be processed is input into the information extraction model, and the model outputs the extraction result after completing the information extraction. The information extraction result includes at least one key information key in the text image to be processed and at least one key information value corresponding to each key information key.

[0091] Since some images may include multiple sets of key-value pairs, and some key-value pairs may be the case where one key information key corresponds to multiple key information values, when extracting the key information value, the extraction result of the key information key can be used as one of the reference information for extraction, in order to improve the accuracy of the extraction result.

[0092] Figure 3This is a schematic diagram of the structure of an information extraction model provided in an embodiment of the present application. Figure 3 The information extraction model includes an encoding module, a first decoding module and a second decoding module. Specifically, the encoding module is used to obtain a multimodal fusion vector according to the information encoding result of the image text information; the first decoding module is used to extract information keys according to the multimodal fusion vector to obtain the key information key; the second decoding module is used to extract information values ​​according to the multimodal fusion vector and the key information key to obtain the key information value.

[0093] In the information extraction model, a fusion vector is obtained by encoding and fusing various information in the input model. The fusion vector is decoded to extract the structured key-value pair information in the image and obtain the information extraction result of the text image to be processed.

[0094] First, in the encoding process, the image information, text content information and text position information in the image text information are encoded and fused to obtain a multimodal fusion vector, so that each element in the image can be dynamically associated with other image positions and text contents in the global range, thereby comprehensively capturing and integrating the rich information in the image context, and improving the accuracy of the extraction results in the subsequent extraction process.

[0095] Secondly, in the decoding process, the first decoding module is used to extract the key information keys of the image based on the multimodal fusion vector output by the above encoding module, and obtain all the key information keys contained in the text image to be processed; next, the second decoding module is used to extract the key information values ​​of the image based on the multimodal fusion vector output by the above encoding module and the key information keys output by the first decoding module, and obtain all the key information values ​​in the text image to be processed that have a corresponding relationship with each key information key.

[0096] The image information extraction method provided by the present application obtains image text information of a text image to be processed; the image text information includes the text image to be processed and at least one group of text information contained in the text image to be processed; any group of text information includes text content information and text position information corresponding to the text content information; a trained information extraction model is used to extract information from the image text information to obtain an information extraction result of the text image to be processed; the information extraction result includes at least one key information key in the text image to be processed and at least one key information value corresponding to each key information key; wherein the information extraction model includes an encoding module, a first decoding module and a second decoding module; the encoding module is used to obtain a multimodal fusion vector according to the information encoding result of the image text information; the first decoding module is used to implement information key extraction according to the multimodal fusion vector, The key information key is obtained; the second decoding module is used to extract the information value according to the multimodal fusion vector and the key information key to obtain the key information value; in the above scheme, the image information in the image text information of the processed text image, the text content information contained in the image, and the local position information corresponding to the text content information are respectively converted into low-dimensional vectors, and the obtained vectors are fused to obtain the multimodal fusion vector; sampling the multimodal fusion vector to extract information from the image can effectively mine the relationship between the image and the text, so as to achieve more accurate extraction results; and in the subsequent decoding process, that is, when extracting information, the key information key is first extracted according to the multimodal fusion vector, and then when extracting the key information value, the extracted key information key is also fused on the basis of the above-mentioned multiple information, which makes up for the information loss problem that may exist when extracting the information value, so as to improve the accuracy of the extraction result.

[0097] Next, the process of extracting information from text images using the information extraction model is introduced in detail.

[0098] First, in the present application, when the information extraction model receives the image text information of the text image to be processed, it performs low-dimensional vector conversion on the image, text content and text position contained in the information, and fuses the converted vectors to obtain a multimodal fusion vector.

[0099] Optionally, the process of performing low-dimensional vector conversion on an image in the present application may specifically include: performing low-dimensional vector conversion processing on the text image to be processed to obtain a corresponding image feature vector; performing position encoding processing on each pixel point in the text image to be processed to obtain a corresponding image position vector; and obtaining an image vector corresponding to the text image to be processed based on the image feature vector and the image position vector.

[0100] It can be explained that, in an optional implementation, the information extraction model provided in the present application can be a Transformer model. The model structure causes the data input to the model to be two-dimensional data, but since the text image to be processed contains multiple channels, it is three-dimensional data. In order to make the text image to be processed match the input data of the model, the image size is preprocessed before the low-dimensional vector conversion processing is performed on the text image to be processed.

[0101] Specifically, for any text image to be processed Among them, H, W and C represent the height, width and number of channels of the image respectively. During the processing, images of different channels can be spliced ​​together to obtain two-dimensional data to realize the input of the matching model; and in order to reduce the amount of calculation of the model for a single image, the spliced ​​image can be cropped to obtain multiple sub-images. Here, p represents the image size of the sub-image, and N represents the image data of the sub-image.

[0102] Specifically, a convolutional neural network of a preset size can be used to map the text image to be processed into two dimensions to obtain a two-dimensional sub-image. Exemplarily, the expression for the cropping process can be:

[0103] x p =conv(x)

[0104] On this basis, low-dimensional vector conversion processing is performed on each sub-image respectively, and the image vector corresponding to the text image to be processed is obtained according to the image vector corresponding to each sub-image after being processed.

[0105] Specifically, a learnable image embedding matrix can be used to perform low-dimensional vector conversion processing on each two-dimensional sub-image to obtain an image feature vector of the text image to be processed. Optionally, the learnable image embedding matrix can be a pre-generated linear layer. Exemplarily, the expression for the image low-dimensional vector conversion processing can be:

[0106]

[0107] in, Represents the image feature vector.

[0108] Furthermore, in order to reference the spatial information and global information of the image in the subsequent image-based extraction process, the text image to be processed can be position-encoded to obtain the corresponding image position vector. Since the text image x to be processed is split into multiple sub-images in the above implementation, each sub-image can be specifically expressed as x1, x2, x3... Therefore, when calculating the position encoding result of the text image to be processed, each sub-image can be position-encoded respectively, and then the position encoding result corresponding to the text image to be processed, that is, the image position vector, can be obtained according to the position encoding result of each sub-image. Exemplarily, the expression for obtaining the image position vector can be:

[0109]

[0110] in, The image position vector representing the text image to be processed.

[0111] On this basis, the image vector and image position vector obtained above can be fused to obtain the image vector corresponding to the text image to be processed. Exemplarily, the expression for obtaining the image vector can be:

[0112]

[0113] Optionally, the process of converting text content information into a low-dimensional vector in the present application may specifically include: for any group of text information, performing word segmentation processing on the text content information to obtain multiple phrases; for any phrase, performing low-dimensional vector conversion processing on the phrase to obtain a corresponding word vector, and performing position encoding processing on the word vector to obtain a corresponding word position vector; according to the word vector and the word position vector corresponding to each phrase in the text information, respectively, obtaining the text word vector corresponding to each text information.

[0114] In this application, in order to facilitate the low-dimensional vector conversion processing of the text content, a word segmenter can be used to segment the text content information to obtain multiple phrases. Each phrase uses t top In order to distinguish each group of text content information, a start marker can be set for each group of text content information. The start marker is represented by [CLS]. The text content information after word segmentation can be represented as: [CLS], ....

[0115] Since the text content information will be calculated in the form of a matrix after being converted into a low-dimensional vector, but the word lengths obtained after word segmentation of different groups of text content information are different, in order to facilitate subsequent calculations, for text content information with shorter lengths, a padding marker can be added after the word group after word segmentation. The padding marker can be represented by [PAD].

[0116] When converting a phrase into a low-dimensional vector, a learnable text embedding matrix can be used to convert each phrase after word segmentation into a low-dimensional vector to obtain a word vector corresponding to each phrase. Exemplarily, the expression for converting a phrase into a low-dimensional vector can be:

[0117]

[0118] in, Represents word vector.

[0119] In order to comprehensively consider the spatial position and global information of the text content information, the word vector of each phrase can be positionally encoded to obtain a word position vector. Exemplarily, the expression for obtaining the word position vector can be:

[0120]

[0121] in, Represents the word position vector.

[0122] On this basis, the word vector and word position vector obtained above can be fused to obtain the text word vector corresponding to the text content information. Exemplarily, the expression for obtaining the text word vector can be:

[0123]

[0124] in, Represents a text word vector.

[0125] Optionally, the process of converting text position information into a low-dimensional vector in the present application may specifically include: determining the word position information corresponding to each phrase in the at least one group of text information based on each of the text position information; performing low-dimensional vector conversion processing on each of the word position information to obtain a text position vector corresponding to each of the text information.

[0126] In this application, in order to integrate more information, when encoding the text position information, in addition to encoding the position coordinates of the text content, the area size of the text area corresponding to the text content and the area shape are also encoded.

[0127] Specifically, before processing, the text position coordinates corresponding to the text content are determined based on the results of optical character recognition, and then the region size is determined based on the text position coordinates, that is, the region height h and width w. On this basis, the region shape can be represented by the distance from the region edge to the region centroid. For example, the horizontal coordinate distance from the upper left corner of the text region to the text block centroid can be calculated. The vertical coordinate distance from the upper left corner of the text area to the center of mass of the text block The horizontal coordinate distance from the lower right corner of the text area to the center of mass of the text block And the vertical coordinate distance from the lower corner of the text area to the centroid of the text block

[0128] The above text position information is converted into a low-dimensional vector to obtain a corresponding text position vector. Exemplarily, the expression for obtaining the text position vector may be:

[0129]

[0130] in, Represents the text position vector; W x and W y Indicates encoding parameters for different directions.

[0131] On the basis of the above implementation manner, the step of obtaining a multimodal fusion vector according to the information encoding result in the present application may include: determining the image self-attention vector corresponding to the image vector, the text word self-attention vector corresponding to the text word vector, and the text position self-attention vector corresponding to the text position vector; determining the multimodal fusion vector corresponding to the text image to be processed according to the image self-attention vector, the text word self-attention vector and the text position self-attention vector.

[0132] Specifically, an encoder can be used to encode each vector separately to obtain the self-attention vector corresponding to each vector, and then the respective attention vectors are fused to obtain a multimodal fusion vector.

[0133] Exemplarily, the expression for obtaining the multimodal fusion vector may be:

[0134]

[0135] Among them, H e represents a multimodal vector; specifically, H e It can also be expressed as

[0136] In an optional implementation, a multimodal Transformer self-attention mechanism may be used in the encoder to implement encoding processing of each vector to obtain a multimodal fusion vector. Exemplarily, the calculation expression may include:

[0137]

[0138] Among them, W Q , W K , W V , All are learnable matrices; d represents the hidden layer representation dimension; Q represents the Query matrix, K represents the Key matrix, and V represents the Value matrix.

[0139] In this application, the multimodal Transformer self-attention mechanism adopts multi-head self-attention, which enables the model to promote each element in the image to establish dynamic associations with other image positions and text content in the global scope, thereby comprehensively capturing and integrating the rich information in the image context, providing sufficient data support for subsequent information extraction, and improving the accuracy of information extraction results.

[0140] At this point, the encoding module is used to complete the encoding of the image text of the text image to be processed, and the corresponding multimodal fusion vector is obtained. The multimodal vectors help the encoding module to extract the key features in the image and text, so as to obtain more accurate extraction results in the subsequent decoding process. Next, the decoding module is used to decode the multimodal fusion vector to obtain the key-value pair information in the text image to be processed. Specifically, two decoding modules are used to extract information respectively to obtain the key information key and key information value in the text image to be processed.

[0141] It can be explained that, since there are usually multiple pairs of key-value pairs in the text to be processed, the extraction of key information keys and key information values ​​can be modeled as a sequence generation problem, that is, the first decoding module is directly used to extract multiple key information keys, and the second decoding module is used to extract multiple key information values. In this way, during the extraction process, the association between the key information keys and the key information values ​​in the image can also be used as one of the extraction reference information to improve the accuracy of the subsequent extraction results.

[0142] Optionally, the extraction of key information keys can be understood as a process of obtaining multiple key information keys by cyclically performing extraction steps. In the present application, the extraction process of key information keys can be: according to the multimodal fusion vector and the preset extraction prompt word, multiple rounds of key information key extraction operations are repeatedly performed until the extraction stop condition is met, and all extracted key information keys are determined as at least one key information key in the text image to be processed.

[0143] In the above method, the extraction prompt word includes the starting mark word in the word segmentation result corresponding to the text content information or the extracted key information key. Optionally, if the current round is the first extraction, the extraction prompt word is the starting mark word in the word segmentation result corresponding to the text content information; if the current round is not the first extraction, the extraction prompt word is the concatenation result of all key information keys extracted in the previous rounds.

[0144] It should also be noted that the extraction stop condition includes: the extracted key information key is a preset extraction end word, or the predicted number of all key information keys meets a preset threshold.

[0145] Next, the extraction process of the key information key is illustrated by taking the extraction operation process of any round as an example. The extraction operation process is: performing low-dimensional vector conversion processing on the extracted prompt word to obtain the corresponding prompt word vector; obtaining the information key prediction vector for characterizing the key information key to be determined in the current round according to the prompt word vector and the multimodal fusion vector; determining the first probability distribution of the information key prediction vector, and determining the key information key of the current round according to the first probability distribution.

[0146] For example, the current round of extraction is not the first extraction, and accordingly, the extracted prompt word is the concatenation result of all key information keys extracted in the previous round. In order to facilitate the subsequent decoding process after information fusion, the concatenation result can be encoded to obtain the prompt word self-attention vector corresponding to the extracted prompt word.

[0147] Among them, the prompt word self-attention vector can be expressed as

[0148] Specifically, in the current decoding module, a decoder may be used to decode the multimodal fusion vector and the above-mentioned prompt word self-attention vector, that is, to perform vector prediction processing to obtain an information key prediction vector representing the key information key to be determined in the current round. Exemplarily, the expression for obtaining the information key prediction vector may include:

[0149]

[0150] in, Represents the information key prediction vector.

[0151] In an optional implementation, the decoder may be implemented based on a Transformer decoder architecture, and the specific implementation will not be described in detail herein.

[0152] In the present application, the information key prediction vector contains the index corresponding to each candidate key information key, and the probability distribution corresponding to each index in the information key prediction vector is calculated, that is, the first probability distribution is determined, and the key information key of the current round is determined based on the first probability distribution.

[0153] Optionally, the process of determining the key information key of the current round according to the first probability distribution in the present application may include: processing the information key prediction vector according to a preset self-attention weight matrix to obtain a processed information key prediction vector; normalizing the processed information key prediction vector to obtain a probability matrix corresponding to the processed information key prediction vector; determining the key information key of the current round according to the probability matrix and the word segmentation result corresponding to the text content information.

[0154] Specifically, a preset self-attention weight matrix is ​​used to process the above-mentioned information key prediction vector to obtain a processed vector; the vector contains multiple numerical values, each of which can be used as an extraction index of a key information key, and the extraction index corresponds to the word index of each phrase in the word segmentation result of the text content information, so that the phrase in the word segmentation result can be found according to the extraction index, and then the key information key can be determined.

[0155] On the basis of obtaining the vector representing the extraction index, the probability distribution corresponding to each extraction index can be calculated, that is, the first probability distribution. Exemplarily, the expression for obtaining the first probability distribution can be:

[0156]

[0157] Among them, p(q t |q 1:t-1 ) represents the first probability distribution; Wq represents the self-attention weight matrix.

[0158] Further, the extraction index corresponding to the maximum probability value in the first probability distribution is determined, and based on the word index corresponding to the above extraction index is determined in the word segmentation result, and the phrase corresponding to the word index is determined, and the phrase is determined as the key information key of the current round.

[0159] The key information key generation step is repeated in the above manner until the decoded key information key is an end marker word, or the number of all the decoded key information keys reaches the maximum prediction length, then the prediction is stopped and all the key information keys obtained are used as the key information keys corresponding to the text image to be processed.

[0160] Since there may be multiple key information keys in the text image to be processed, in order to facilitate the distinction between the key information keys, the key information key prediction decoder may use a special separator to separate each key information key when outputting each key information key.

[0161] Next, the key information values ​​are extracted according to the multimodal fusion vector and the extracted key information keys to obtain all the key information values. The key information keys are used as one of the extraction reference data, which can reduce the search space in the process of extracting key information values. While ensuring the extraction accuracy, it can also improve the extraction efficiency.

[0162] Optionally, the extraction of key information values ​​can also be understood as a process of obtaining multiple key information values ​​by cyclically performing extraction steps. In the present application, the process of extracting key information values ​​may include: repeatedly performing multiple rounds of key information value extraction operations according to the multimodal fusion vector and each of the key information keys, until the extraction is stopped when the extraction stop condition is met, and determining all the extracted key information values ​​as at least one key information value corresponding to each of the key information keys.

[0163] In the above extraction process, the extraction stop condition may be: the extracted key information value is a preset extraction end word, or the predicted number of all key information values ​​meets a preset threshold.

[0164] Next, the extraction process of the key information value is illustrated by taking the extraction operation process of any round as an example. The extraction operation process is as follows: determine the image position of the image area where the key information key is located, and perform a low-dimensional vector conversion on the image position to obtain the corresponding image position vector; determine the information key self-attention vector corresponding to the splicing result of each key information key according to the image position vector; obtain the information value prediction vector used to characterize the key information value to be determined in the current round according to the multimodal fusion vector and the information key self-attention vector; determine the key information value of the current round according to the probability distribution of the information value prediction vector.

[0165] Specifically, the key information keys extracted by the first decoding module are obtained, and the image positions of the key information keys in the image, that is, the regional coordinate information of the image region where the key information keys are located, are determined according to the text position information corresponding to the text contents contained in the text image to be processed.

[0166] Further, referring to the above implementation, the region coordinate information can be converted into a low-dimensional vector to obtain a corresponding image position vector; the text position vector obtained by converting the text position information into a low-dimensional vector can also be searched to obtain the corresponding image position vector, and the method for obtaining the vector is not limited. Exemplarily, in this application, the obtained image position vector can be used express.

[0167] At the same time, the above key information keys can also be spliced ​​to obtain a splicing result, and the splicing result can be converted into a low-dimensional vector to obtain an information key vector corresponding to the corresponding key information key. express.

[0168] On this basis, in order to obtain richer features in the key information key, the information key vector corresponding to the key information key and the image position vector corresponding to the image area where the key information key is located can be encoded to obtain the encoded information key self-attention vector. Exemplarily, the expression for obtaining the information key self-attention vector can be:

[0169]

[0170] in, Represents the information key self-attention vector.

[0171] It can be explained that if the current round of extraction process is the first extraction, then in the current decoding module, a decoder can be used to decode the multimodal fusion vector and the information key self-attention vector, that is, to perform vector prediction processing to obtain an information value prediction vector representing the key information value to be determined in the current round. Exemplarily, the expression for obtaining the information value prediction vector may include:

[0172]

[0173] in, Characterizes the information value prediction vector.

[0174] It should also be noted that if the current round of extraction is not the first extraction, the current extraction result includes the key information values ​​extracted in the previous round. Based on this, the key information values ​​obtained are spliced ​​to obtain the splicing result, and the splicing result is also converted to a low-dimensional vector and encoded to obtain the information value self-attention vector corresponding to the corresponding key information value. In the decoding process, the information value self-attention vector is also processed as one of the reference data to obtain the information value prediction vector representing the key information value to be determined in the current round.

[0175] In an optional implementation, the decoder may be implemented based on a Transformer decoder architecture, and the specific implementation will not be described in detail herein.

[0176] Furthermore, the information value prediction vector includes an index corresponding to each candidate key information value, and the probability distribution corresponding to each index in the information value prediction vector is calculated, and the key information value of the current round is determined according to the corresponding probability distribution.

[0177] It should be noted that in order to improve the accuracy of the information value determination result, the probability distribution in the present application, in addition to the second probability distribution corresponding to the information value prediction vector, also includes a third probability distribution of the product of the information value prediction vector and the word vector corresponding to the text content information; accordingly, the process of determining the key information value of the current round according to the probability distribution of the information value prediction vector in the present application may include: determining the second probability distribution of the information value prediction vector, and the third probability distribution of the product of the information value prediction vector and the word vector corresponding to the text content information; determining the first weight and the second weight corresponding to the second probability distribution and the third probability distribution respectively according to the multimodal fusion vector and the information value prediction vector; determining the key information value of the current round according to the weighted result of the first weight and the second probability distribution, and the weighted result of the second weight and the third probability distribution.

[0178] On this basis, the present application uses a preset self-attention weight to process the above-mentioned information value prediction vector to obtain a processed vector; and the second probability distribution corresponding to each value in the calculated vector is respectively calculated; exemplarily, the expression of the second probability distribution can be:

[0179]

[0180] Among them, p r (r t |r 1:t-1 ) represents the second probability distribution; Wr represents the self-attention weight matrix.

[0181] It should be noted that, since the key information value exists in the image text, increasing the probability of the words already existing in the image text can help the second decoder extract more accurate key information values. Therefore, when calculating the probability distribution corresponding to the information value prediction vector in the present application, on the basis of calculating the second probability distribution corresponding to the information value prediction vector, a third probability distribution corresponding to the product of the information value prediction vector and the word vector corresponding to the text content information is also calculated.

[0182] Based on this, the word vector corresponding to the text content information is used as the self-attention weight corresponding to the information value prediction vector, and the information key prediction vector is processed to obtain the processed vector; and the third probability distribution corresponding to each value in the calculated vector is respectively calculated; exemplarily, the expression of the third probability distribution can be:

[0183]

[0184] Among them, p c (r t |r 1:t-1 ) characterizes the third probability distribution.

[0185] On this basis, the weights corresponding to the second probability distribution and the third probability distribution are calculated respectively, so as to obtain the final probability distribution according to the weights of the respective probability distributions.

[0186] Since the first vector of the multimodal representation matrix output by the multimodal encoder fuses information such as images, text blocks, and text block coordinates, the key information value representation vector output by the first decoder that outputs the key information value can be concatenated with the first vector of the multimodal fusion vector to obtain the associated information vector of the next key value and multimodal information (image, text block, and text block coordinates). In this way, when input into a fully connected layer, the weight of the probability distribution of the attention score and the next word of the key value can be predicted, thereby improving the prediction accuracy of the key information value.

[0187] Specifically, obtain the first vector value in the multimodal fusion vector output by the encoder, that is, select the multimodal fusion vector Vector in On this basis, the vector is fused with the information value prediction vector, and the fused vector is processed using the sigmoid function to obtain a mapping value between 0 and 1, and the value is used as the first weight value corresponding to the second probability distribution. Exemplarily, the expression for obtaining the first weight value can be:

[0188]

[0189] p gen Indicates the first weight value.

[0190] Furthermore, 1-p gen As the second weight value corresponding to the third probability distribution.

[0191] On this basis, based on the weighted result according to the first weight and the second probability distribution, and the weighted result according to the second weight and the third probability distribution, a final probability distribution is obtained, and the key information value of the current round is determined according to the final probability distribution. Exemplarily, the expression for obtaining the final probability distribution can be:

[0192] p final (r t |r 1:t-1 )=(1-p gen )*p c (r t |r 1:t-1 )+p gen *p r (r t |r 1:t-1 )

[0193] Among them, p final (rt |r 1:t-1 ) represents the final probability distribution.

[0194] It should be explained that the final probability distribution contains multiple numerical values, each of which can be used as an extraction index for the key information value, and the extraction index corresponds to the word index of each phrase in the word segmentation result of the text content information. In this way, the phrase in the word segmentation result can be found according to the extraction index, and then the key information value can be determined.

[0195] In this application, in order to use the information extraction model to complete the extraction of the above key-value pair information during application, the model needs to be trained in advance.

[0196] During the training process, the cross entropy error can be used to calculate the prediction loss of key information keys and key information values, and the model parameters can be updated according to the calculated loss. After multiple iterative updates, the trained information extraction model is obtained.

[0197] Specifically, in any round of training, the real labels of key information keys and key values ​​are defined as Q * For and R * , the prediction results of the model output are expressed as Q and R, on this basis, the definition and Respectively represent the loss functions corresponding to the key information key and the key information value; where,

[0198]

[0199] The above two loss functions are optimized simultaneously during the training process. On this basis, the model loss is defined as:

[0200]

[0201] in, Characterized as model loss.

[0202] In this application, since the key information key and key value have different sequence lengths and vocabulary sizes, which makes the weight α difficult to adjust, this application adopts the introduction of non-deterministic loss to optimize the model. On this basis, the obtained model loss can also be expressed as:

[0203]

[0204] Among them, σ1 and σ2 represent two learnable parameters. The advantage of this uncertainty loss is that it simulates the homoscedastic uncertainty of each task and provides task-related weights for multi-task learning. During the training process, teacher-guided and self-guided methods are used for training according to a certain probability (exemplarily, the probability is set to 50% in this application). If it is teacher-guided, the word input at each time step t is the word given by the label. If it is self-guided learning, each word input is a model predicted word.

[0205] In an optional embodiment, the encoding module, the first decoding module and the second decoding module in the information extraction model in the present application are trained simultaneously. During training, the Adam optimizer is used to optimize the entire model. The Dropout rate of each module is set to 0.3. The learning rate of the model can be set to 1e-4 during training. At the same time, the learning rate ramp-up method is used to train the model. The learning rate increases by 10% in each training stage. When the learning rate is greater than or equal to 1e-4, the learning rate is reduced by 10% every other stage. In this way, a better information extraction model can be obtained.

[0206] Figure 4 This is a schematic diagram of the structure of an image information extraction device provided by the present application according to an embodiment. Figure 4 The image information extraction device 40 includes: an information acquisition module 401 and an information extraction module 402; wherein,

[0207] The information acquisition module 401 is used to acquire image text information of the text image to be processed; the image text information includes the text image to be processed and at least one set of text information contained in the text image to be processed; any set of text information includes text content information and text position information corresponding to the text content information;

[0208] An information extraction module 402 is used to extract information from the image text information using a trained information extraction model to obtain an information extraction result of the text image to be processed; the information extraction result includes at least one key information key in the text image to be processed and at least one key information value corresponding to each key information key;

[0209] Among them, the information extraction model includes an encoding module, a first decoding module and a second decoding module; the encoding module is used to obtain a multimodal fusion vector based on the information encoding result of the image text information; the first decoding module is used to implement information key extraction based on the multimodal fusion vector to obtain the key information key; the second decoding module is used to implement information value extraction based on the multimodal fusion vector and the key information key to obtain the key information value.

[0210] In an optional implementation manner, the information extraction module, when executing information key extraction according to the multimodal fusion vector to obtain the key information key, is specifically used to:

[0211] According to the multimodal fusion vector and the preset extraction prompt word, multiple rounds of key information key extraction operations are repeatedly performed until the extraction stop condition is met, and all the extracted key information keys are determined as at least one key information key in the text image to be processed; wherein,

[0212] If the current round is the first extraction, the extraction prompt word is the starting mark word in the word segmentation result corresponding to the text content information; if the current round is not the first extraction, the extraction prompt word is the concatenation result of all key information keys extracted in the previous rounds;

[0213] The extraction stop condition includes: the extracted key information key is a preset extraction end word, or the predicted number of all key information keys meets a preset threshold;

[0214] The extraction process of any round includes:

[0215] Performing low-dimensional vector conversion processing on the extracted prompt word to obtain a corresponding prompt word vector;

[0216] Obtaining an information key prediction vector for representing a key information key to be determined in the current round according to the prompt word vector and the multimodal fusion vector;

[0217] A first probability distribution of the information key prediction vector is determined, and a key information key of the current round is determined according to the first probability distribution.

[0218] In an optional implementation manner, the information extraction module, when executing determining the first probability distribution of the information key prediction vector and determining the key information key of the current round according to the first probability distribution, is specifically further configured to:

[0219] Processing the information key prediction vector according to a preset self-attention weight matrix to obtain a processed information key prediction vector;

[0220] Normalizing the processed information key prediction vector to obtain a probability matrix corresponding to the processed information key prediction vector;

[0221] The key information key of the current round is determined according to the probability matrix and the word segmentation result corresponding to the text content information.

[0222] In an optional implementation manner, the information extraction module, when extracting the information value according to the multimodal fusion vector and the key information key to obtain the key information value, is specifically used to:

[0223] According to the multimodal fusion vector and each of the key information keys, multiple rounds of key information value extraction operations are repeatedly performed until the extraction stops when the extraction stop condition is met, and all the extracted key information values ​​are determined as at least one key information value corresponding to each of the key information keys; wherein,

[0224] The extraction stop condition includes: the extracted key information value is a preset extraction end word, or the predicted number of all key information values ​​meets a preset threshold;

[0225] The extraction process of any round includes:

[0226] Determine the image position of the image area where the key information key is located, and perform low-dimensional vector conversion on the image position to obtain a corresponding image position vector;

[0227] Determining, according to the image position vector, an information key self-attention vector corresponding to a splicing result of each of the key information keys;

[0228] Obtaining an information value prediction vector for characterizing a key information value to be determined in a current round according to the multimodal fusion vector and the information key self-attention vector;

[0229] The key information value of the current round is determined according to the probability distribution of the information value prediction vector.

[0230] In an optional implementation manner, the information extraction module, when executing the probability distribution of the information value prediction vector to determine the key information value of the current round, is specifically further used to:

[0231] Determine a second probability distribution of the information value prediction vector and a third probability distribution of the product of the information value prediction vector and the word vector corresponding to the text content information;

[0232] Determine, according to the multimodal fusion vector and the information value prediction vector, a first weight and a second weight corresponding to the second probability distribution and the third probability distribution, respectively;

[0233] The key information value of the current round is determined according to a weighted result of the first weight and the second probability distribution, and a weighted result of the second weight and the third probability distribution.

[0234] In an optional implementation, the information encoding result includes an image vector, a text word vector, and a text position vector;

[0235] The information acquisition module, when executing the information encoding result according to the image text information to obtain the multimodal fusion vector, is specifically used to:

[0236] Determine an image self-attention vector corresponding to the image vector, a text word self-attention vector corresponding to the text word vector, and a text position self-attention vector corresponding to the text position vector;

[0237] A multimodal fusion vector corresponding to the text image to be processed is determined according to the image self-attention vector, the text word self-attention vector and the text position self-attention vector.

[0238] In an optional embodiment, the device is also used for:

[0239] Before obtaining the multimodal fusion vector according to the information encoding result of the image text information, performing low-dimensional vector conversion processing on the text image to be processed to obtain a corresponding image feature vector;

[0240] Performing position coding processing on each pixel point in the text image to be processed to obtain a corresponding image position vector;

[0241] An image vector corresponding to the text image to be processed is obtained according to the image feature vector and the image position vector.

[0242] In an optional implementation, the device is further used to: before obtaining the multimodal fusion vector according to the information encoding result of the image text information, for any group of text information, perform word segmentation processing on the text content information to obtain multiple word groups;

[0243] For any word group, the word group is converted into a low-dimensional vector to obtain a corresponding word vector, and the word vector is positionally encoded to obtain a corresponding word position vector;

[0244] Obtaining text word vectors corresponding to each of the text information according to the word vectors and the word position vectors respectively corresponding to each of the word groups in the text information;

[0245] Determining word position information corresponding to each of the phrases in the at least one group of text information according to each of the text position information;

[0246] The word position information is converted into a low-dimensional vector to obtain a text position vector corresponding to each text information.

[0247] Figure 5 It is a block diagram of an electronic device provided in an embodiment of the present application, and the device may be a computer, a digital broadcast terminal, etc.

[0248] See also Figure 5, the device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output interface 812 , a sensor component 814 , and a communication component 816 .

[0249] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0250] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0251] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.

[0252] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0253] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0254] The input / output interface 812 provides an interface between the processing component 802 and the peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0255] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of the components, such as the display and keypad of the device 800, and the sensor assembly 814 can also detect the position change of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a solid image (Complementary Metal Oxide Semiconductor, CMOS) sensor or a semiconductor image (Charge-coupled Device, CCD) sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0256] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0257] In an exemplary embodiment, the device 800 can be implemented by one or more application-specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0258] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by a processor 820 of the device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0259] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the above-mentioned aircraft collision prediction method.

[0260] An embodiment of the present application also provides a chip for running instructions, which is used to execute the technical solution of the aircraft collision prediction method in the above embodiment.

[0261] An embodiment of the present application further provides a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed on a computer, the computer executes the technical solution of the aircraft collision prediction method in the above embodiment.

[0262] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When at least one processor executes the computer program, it can implement the technical solution of the collision prediction method of the aircraft in the above embodiment.

[0263] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0264] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0265] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0266] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for extracting image information, characterized in that: include: Acquire image text information of a text image to be processed; the image text information includes the text image to be processed and at least one set of text information contained in the text image to be processed; any set of text information includes text content information and text position information corresponding to the text content information; Using the trained information extraction model to extract information from the image text information, obtaining an information extraction result of the text image to be processed; the information extraction result includes at least one key information key in the text image to be processed and at least one key information value corresponding to each key information key; The information extraction model includes an encoding module, a first decoding module and a second decoding module; the encoding module is used to obtain a multimodal fusion vector according to the information encoding result of the image text information; the first decoding module is used to extract the information key according to the multimodal fusion vector to obtain the key information key; The second decoding module is used to extract the information value according to the multimodal fusion vector and the key information key to obtain the key information value.

2. The method according to claim 1, characterized in that Extracting information keys according to the multimodal fusion vector to obtain the key information keys includes: According to the multimodal fusion vector and the preset extraction prompt word, multiple rounds of key information key extraction operations are repeatedly performed until the extraction stop condition is met, and all the extracted key information keys are determined as at least one key information key in the text image to be processed; wherein, If the current round is the first extraction, the extraction prompt word is the starting mark word in the word segmentation result corresponding to the text content information; if the current round is not the first extraction, the extraction prompt word is the concatenation result of all key information keys extracted in the previous rounds; The extraction stop condition includes: the extracted key information key is a preset extraction end word, or the predicted number of all key information keys meets a preset threshold; The extraction process of any round includes: Performing low-dimensional vector conversion processing on the extracted prompt word to obtain a corresponding prompt word vector; Obtaining an information key prediction vector for representing a key information key to be determined in the current round according to the prompt word vector and the multimodal fusion vector; A first probability distribution of the information key prediction vector is determined, and a key information key of the current round is determined according to the first probability distribution.

3. The method according to claim 2, characterized in that Determining a first probability distribution of the information key prediction vector, and determining a key information key of the current round according to the first probability distribution, comprises: Processing the information key prediction vector according to a preset self-attention weight matrix to obtain a processed information key prediction vector; Normalizing the processed information key prediction vector to obtain a probability matrix corresponding to the processed information key prediction vector; The key information key of the current round is determined according to the probability matrix and the word segmentation result corresponding to the text content information.

4. The method according to claim 1, characterized in that: Extracting information value according to the multimodal fusion vector and the key information key to obtain the key information value includes: According to the multimodal fusion vector and each of the key information keys, multiple rounds of key information value extraction operations are repeatedly performed until the extraction stops when the extraction stop condition is met, and all the extracted key information values ​​are determined as at least one key information value corresponding to each of the key information keys; wherein, The extraction stop condition includes: the extracted key information value is a preset extraction end word, or the predicted number of all key information values ​​meets a preset threshold; The extraction process of any round includes: Determine the image position of the image area where the key information key is located, and perform low-dimensional vector conversion on the image position to obtain a corresponding image position vector; Determining, according to the image position vector, an information key self-attention vector corresponding to a splicing result of each of the key information keys; According to the multimodal fusion vector and the information key self-attention vector, an information value prediction vector for characterizing the key information value to be determined in the current round is obtained; The key information value of the current round is determined according to the probability distribution of the information value prediction vector.

5. The method according to claim 4, characterized in that Determining the key information value of the current round according to the probability distribution of the information value prediction vector includes: Determine a second probability distribution of the information value prediction vector and a third probability distribution of the product of the information value prediction vector and the word vector corresponding to the text content information; Determine, according to the multimodal fusion vector and the information value prediction vector, a first weight and a second weight corresponding to the second probability distribution and the third probability distribution, respectively; The key information value of the current round is determined according to a weighted result of the first weight and the second probability distribution, and a weighted result of the second weight and the third probability distribution.

6. The method according to claim 1, characterized in that The information encoding result includes an image vector, a text word vector and a text position vector; According to the information encoding result of the image text information, a multimodal fusion vector is obtained, including: Determine an image self-attention vector corresponding to the image vector, a text word self-attention vector corresponding to the text word vector, and a text position self-attention vector corresponding to the text position vector; A multimodal fusion vector corresponding to the text image to be processed is determined according to the image self-attention vector, the text word self-attention vector and the text position self-attention vector.

7. The method according to claim 6, characterized in that Before obtaining the multimodal fusion vector according to the information encoding result of the image text information, the method further includes: Performing low-dimensional vector conversion processing on the text image to be processed to obtain a corresponding image feature vector; Performing position coding processing on each pixel in the text image to be processed to obtain a corresponding image position vector; An image vector corresponding to the text image to be processed is obtained according to the image feature vector and the image position vector.

8. The method according to claim 6, characterized in that Before obtaining the multimodal fusion vector according to the information encoding result of the image text information, the method further includes: For any group of text information, segment the text content information to obtain multiple phrases; For any word group, the word group is converted into a low-dimensional vector to obtain a corresponding word vector, and the word vector is positionally encoded to obtain a corresponding word position vector; Obtaining text word vectors corresponding to each of the text information according to the word vectors and the word position vectors respectively corresponding to each of the word groups in the text information; Determining word position information corresponding to each of the phrases in the at least one group of text information according to each of the text position information; The word position information is converted into a low-dimensional vector to obtain a text position vector corresponding to each text information.

9. An image information extraction device, characterized in that: The device comprises: An information acquisition module, used to acquire image text information of a text image to be processed; the image text information includes the text image to be processed and at least one set of text information contained in the text image to be processed; any set of text information includes text content information and text position information corresponding to the text content information; An information extraction module, configured to extract information from the image text information using a trained information extraction model to obtain an information extraction result of the text image to be processed; the information extraction result includes at least one key information key in the text image to be processed and at least one key information value corresponding to each key information key; Among them, the information extraction model includes an encoding module, a first decoding module and a second decoding module; the encoding module is used to obtain a multimodal fusion vector based on the information encoding result of the image text information; the first decoding module is used to implement information key extraction based on the multimodal fusion vector to obtain the key information key; the second decoding module is used to implement information value extraction based on the multimodal fusion vector and the key information key to obtain the key information value.

10. An electronic device, characterized in that: include: A processor and a memory communicatively connected to the processor; The memory stores computer-executable instructions; When executing the computer-executable instructions, the processor is used to implement the image information extraction method as described in any one of claims 1 to 8.

Images