Model training method, information extraction method and device

Abstract

The invention provides a model training method and device and an information extraction method and device, and relates to the technical field of information positioning. The method comprises the following steps: acquiring a sample image, a sample text feature of a sample keyword in the sample image, and a sample image feature of the position of the sample keyword in the sample image; based on an attention mechanism, taking the sample text features as query vectors, taking the sample image features as key-value pair vectors, and constructing a model; using the model to predict the position of the sample keyword in the sample image, obtaining sample prediction position information corresponding to the sample keyword, and determining a sample sub-image according to the sample image and the sample prediction position information; and according to the first text feature corresponding to the sample keyword and the sample sub-graph, training the model to obtain a trained model. According to the embodiment of the invention, generalization information extraction can be carried out on different card images, and the extraction efficiency of the information in the card images is improved.

Description

Technical Field

[0001] The present application relates to the field of information positioning technology, and in particular to a model training method, an information extraction method, a model training device, an information extraction device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] At present, in order to extract information from a specific card image, it is usually necessary to construct an information extraction rule corresponding to the specific information arrangement structure corresponding to the card image, and then extract the required information from the card image according to the information extraction rule.

[0003] However, different card images correspond to different information arrangement structures. The above method cannot quickly locate the position information corresponding to the information to be extracted in different card images, and it is even more impossible to use an information extraction rule to obtain the text information in the card image that does not match the information extraction rule. That is, generalized information extraction cannot be achieved, which reduces the efficiency of extracting information from card images. Summary of the invention

[0004] The present application provides a model training method, an information extraction method, a model training device, an information extraction device, an electronic device, a computer-readable storage medium and a computer program product, which can realize generalized information extraction of different card images and improve the efficiency of extracting information from card images.

[0005] In a first aspect, the present application provides a model training method, which includes: obtaining a sample image, sample text features of a sample keyword in the sample image, and sample image features of the position of the sample keyword in the sample image; based on an attention mechanism, using the sample text features as a query vector and the sample image features as a key-value pair vector to construct a model; using the model to predict the position of the sample keyword in the sample image, obtaining the sample prediction position information corresponding to the sample keyword, and determining the sample subgraph based on the sample image and the sample prediction position information; training the model based on the first text features and the sample subgraph corresponding to the sample keyword to obtain a trained model.

[0006] In the second aspect, the present application provides an information extraction method, which includes: obtaining an image to be verified, and extracting keywords to be verified in the image to be verified; using a trained model to locate the keywords to be verified, and obtaining target position information of the keywords to be verified in the image to be verified; wherein the trained model is obtained according to the training method of any one of the models in the embodiments of the present application; according to the target position information, extracting information to be verified that matches the target position information from the image to be verified; inputting the information to be verified into a preset text detection model for text recognition, and obtaining text information corresponding to the keywords to be verified.

[0007] In a third aspect, the present application provides a model training device, which includes: a first acquisition module, used to acquire a sample image, sample text features of a sample keyword in the sample image, and sample image features of the position of the sample keyword in the sample image; a construction module, used to construct a model based on an attention mechanism, using the sample text features as a query vector and the sample image features as a key-value pair vector; a prediction module, used to use the model to predict the position of the sample keyword in the sample image, obtain the sample prediction position information corresponding to the sample keyword, and determine the sample subgraph based on the sample image and the sample prediction position information; a training module, used to train the model based on the first text features and the sample subgraph corresponding to the sample keyword to obtain a trained model.

[0008] In a fourth aspect, the present application provides an information extraction device, which includes: a second acquisition module, used to acquire an image to be verified and extract a keyword to be verified in the image to be verified; a positioning module, used to use a trained model to locate the keyword to be verified, and obtain target position information of the keyword to be verified in the image to be verified; wherein the trained model is obtained according to a training method of any one of the models in the embodiments of the present application; a matching module, used to extract information to be verified that matches the target position information from the image to be verified based on the target position information; a text recognition module, used to input the information to be verified into a preset text detection model for text recognition, and obtain text information corresponding to the keyword to be verified.

[0009] In a fifth aspect, the present application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, and one or more of the computer programs are executed by the at least one processor so that the at least one processor can execute the above-mentioned model training method, or implement the above-mentioned information extraction method.

[0010] In a sixth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-mentioned model training method, or implements the above-mentioned information extraction method.

[0011] In the seventh aspect, the present application provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying a computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device implements the above-mentioned model training method, or implements the above-mentioned information extraction method.

[0012] In the embodiment provided by the present application, in the training process of the information positioning model, first, by obtaining the sample image, the sample text features of the sample keyword in the sample image, and the sample image features of the position of the sample keyword in the sample image, the features of the sample keyword in the sample image can be accurately described in two dimensions of the sample text features and the sample image features, which is convenient for subsequent analysis; then, according to the attention mechanism, the sample text features are used as the query vector, and the sample image features are used as the key-value pair vector to build a model, and by using the model to realize the generalization processing of the text features and image features in the sample image, it is no longer necessary to extract the features in the sample image for a specific information arrangement structure, and a sample subgraph matching the information extraction requirements can be obtained; further, according to the first text features and the sample subgraph corresponding to the sample keyword, the model is trained so that the trained model can be applicable to locating the keywords to be verified in various different types of images to be verified, and the target position information of the keywords to be verified in the images to be verified is obtained, so as to improve the positioning accuracy of the keywords to be verified in the images to be verified, and then based on the obtained target position information of the keywords to be verified in the images to be verified, the text information corresponding to the keywords to be verified can be quickly and accurately extracted, so as to improve the information extraction efficiency of the images to be verified.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are used to provide a further understanding of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. By describing the detailed exemplary embodiments with reference to the accompanying drawings, the above and other features and advantages will become more apparent to those skilled in the art.

[0015] Figure 1A schematic diagram of the structure of a card image provided in an embodiment of the present application is shown.

[0016] Figure 2 A flow chart showing a method for training a model provided in an embodiment of the present application is shown.

[0017] Figure 3 A flowchart of a method for acquiring text features and image features provided in an embodiment of the present application is shown.

[0018] Figure 4 A schematic diagram of a network structure for feature alignment provided in an embodiment of the present application is shown.

[0019] Figure 5 A flow chart of a method for aligning text and images provided in an embodiment of the present application is shown.

[0020] Figure 6 A schematic diagram of the training process of an information positioning model provided in an embodiment of the present application is shown.

[0021] Figure 7 A flow chart of an information extraction method provided in an embodiment of the present application is shown.

[0022] Figure 8 A block diagram showing the composition of a model training device provided in an embodiment of the present application.

[0023] Fig. 9 A block diagram showing the composition of an information extraction device provided in an embodiment of the present application is shown.

[0024] Fig.10 A block diagram of the composition of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the technical solution of the present application, the exemplary embodiments of the present application are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0026] In the absence of conflict, the embodiments of the present application and the features therein may be combined with each other.

[0027] As used herein, the term "and / or" includes any and all combinations of one or more related enumerated items. The terms used herein are only used to describe specific embodiments and are not intended to limit the present application. As used herein, the singular forms "a" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "including" and / or "made of..." are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded. "Connected" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0028] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this application, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined herein.

[0029] At present, the common method for extracting relevant information from card images includes the following steps: first, performing text detection and recognition on the entire image of a specific card image, and then training a preset classification model according to the information arrangement structure corresponding to the specific card image to obtain an information extraction model corresponding to the specific card image.

[0030] However, since different card images correspond to different information arrangement structures, for example, Figure 1 A schematic diagram of the structure of a card image provided by an embodiment of the present application is shown. Figure 1 As shown, Figure 1 The first card image 110 shown in (a) and Figure 1 The second card certificate image 120 shown in (b) is two card certificate images of the same type. The first card certificate image 110 includes: a first information name and a value of the first information, a second information name and a value of the second information, a third information name and a value of the third information, a fourth information name and a value of the fourth information, ..., an nth information name and a value of the nth information; n represents the number of information names and the number of information values ​​included in the first card certificate image 110.

[0031] The second card image 120 includes: a first information name and a value of the first information, a second information name and a value of the second information, a third information name and a value of the third information, a fourth information name and a value of the fourth information, ..., a k-1th information name and a value of the k-1th information, a k-th information name and a value of the k-1th information, a k+1th information name and a value of the k+1th information, a k+2th information name and a value of the k+2th information, a k+3th information name and a value of the k+3th information, ..., an mth information name and a value of the mth information. m represents the number of information names and the number of information values ​​included in the second card image 120, and k is an integer greater than 1 and less than m.

[0032] Depend on Figure 1 It can be seen that the page arrangement structure of each information name and information value in the first card image 110 is different from the page arrangement structure of each information name and information value in the second card image 120. If the information extraction model corresponding to the first card image 110 is used to extract each information name and its corresponding value in the second card image 120, it will cause information extraction errors and reduce the accuracy of extracting the information in the second card image 120.

[0033] According to the training method of the model of the embodiment of the present application, in the training process of the information positioning model, first, by obtaining the sample image, the sample text features of the sample keyword in the sample image, and the sample image features of the position of the sample keyword in the sample image, the features of the sample keyword in the sample image can be accurately described through the two dimensions of the sample text features and the sample image features, which is convenient for subsequent analysis; then, according to the attention mechanism, the sample text features are used as the query vector, and the sample image features are used as the key-value pair vector to build a model, and by using the model to realize the generalization processing of the text features and image features in the sample image, it is no longer necessary to extract the features in the sample image for a specific information arrangement structure, and the sample subgraph matching the information extraction requirements can be obtained; further, according to the first text features and the sample subgraph corresponding to the sample keyword, the model is trained so that the trained model can be applicable to locating the keywords to be verified in various different types of images to be verified, and the target position information of the keywords to be verified in the images to be verified is obtained, so as to improve the positioning accuracy of the keywords to be verified in the images to be verified, and then based on the obtained target position information of the keywords to be verified in the images to be verified, the text information corresponding to the keywords to be verified can be quickly and accurately extracted, and the information extraction efficiency of the images to be verified can be improved.

[0034] The training method of the model according to the embodiment of the present application can be performed by an electronic device such as a terminal device or a server. The terminal device can be a vehicle-mounted device, a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The server can be an independent physical server, a server cluster composed of multiple physical servers, or a cloud server capable of cloud computing. The method can be implemented by a processor calling a computer-readable program instruction stored in a memory. Alternatively, the method can be executed by a server.

[0035] Figure 2 A flow chart of a model training method provided in an embodiment of the present application is shown. Figure 2 , the training method of the model includes:

[0036] Step S21, obtaining a sample image, a sample text feature of a sample keyword in the sample image, and a sample image feature of a position of the sample keyword in the sample image.

[0037] Among them, sample text features of sample keywords can be extracted from sample images by using natural language processing technology. Sample text features are extracted by word embedding (e.g., using Word2Vec or BERT (Bidirectional Encoder Representations from Transformers) model to extract text features); then, the extracted features are converted into vector representations of fixed dimensions, and further pooling or encoding operations are performed to obtain the final text feature vector as the sample text feature.

[0038] For another example, a convolutional neural network (CNN) is used to extract sample image features from a sample image. The sample image features may be output by a convolutional layer or by a global pooling layer.

[0039] In some embodiments, the sample text features and sample image features are features obtained by preprocessing the acquired sample images. The sample images may include multiple ones, each sample image corresponds to a training sample subset, a training sample subset includes multiple groups of sample features, each group of sample features corresponds to a sample keyword in the sample image, and each group of sample features includes sample text features and sample image features.

[0040] It should be noted that the sample text features and sample image features in a set of sample features are both features used to characterize a sample keyword. Moreover, in the set of sample features, the sample text features and sample image features are also features of different dimensions characterizing information at the same position in the image, so as to improve the accuracy of describing the sample keywords in the sample image.

[0041] Step S22, based on the attention mechanism, the sample text features are used as query vectors and the sample image features are used as key-value pair vectors to build a model.

[0042] The construction of the model may include the following steps: first, the sample text features are used as query vectors, and the sample image features are used as key-value pair vectors, and then, the attention score is calculated based on the query vector (i.e., the sample text features) and the key-value pair vector (i.e., the sample image features). For example, the attention score is calculated by dot product, cosine similarity or other similarity metrics.

[0043] Furthermore, the softmax function is used to normalize the attention scores to obtain the attention weights of each key vector. Finally, the normalized attention weights are used to perform weighted summation on the value vectors to obtain the attention-weighted image feature representation, which is used as the output of the model.

[0044] The model constructed in the above manner may also be referred to as an information positioning model, which can be used to predict the position of the sample keyword in the sample image to obtain the sample prediction position information corresponding to the sample keyword.

[0045] Step S23, using the model to predict the position of the sample keyword in the sample image, obtaining sample prediction position information corresponding to the sample keyword, and determining the sample sub-image according to the sample image and the sample prediction position information.

[0046] Among them, the sample prediction position information corresponding to the sample keyword can be mapped to the sample image, so as to clarify which part of the sample image is the image corresponding to the sample prediction position information, and use this part of the image as the sample sub-image.

[0047] Since the information location model is a model constructed based on sample text features and sample image features, and the sample text features and sample image features are features of different dimensions that represent the information at the same position in the image, and both the sample text features and the sample image features are features used to represent a sample keyword, therefore, by processing the sample image through the information location model, it is possible to clarify the position information corresponding to the sample keyword in the sample image, and then predict the sample predicted position information corresponding to the sample keyword, highlighting the position expression in the sample image for subsequent analysis and positioning.

[0048] In some exemplary embodiments, a sample sub-graph is determined based on a sample image and sample prediction position information, including: intercepting the sample image based on the sample prediction position information corresponding to a sample keyword to obtain a sample sub-graph corresponding to the sample prediction position information.

[0049] The sample prediction position information corresponding to the sample keyword can be displayed in the form of coordinates. For example, a coordinate system is constructed with the long side of the sample image as the horizontal coordinate and the short side of the sample image as the vertical coordinate. In this coordinate system, the sample prediction position information corresponding to the sample keyword can be expressed as (x1, y1, x2, y2, off_x1, off_y1, off_x2, off_y2).

[0050] Among them, the coordinates (x1, y1, x2, y2) represent the position information of the sample keyword (such as "document number") in the sample image (such as, the coordinates (x1, y1) represent the position coordinates of the upper left corner of the part of the image occupied by the sample keyword, and the coordinates (x2, y2) represent the position coordinates of the lower right corner of the part of the image occupied by the sample keyword, etc.), and the coordinates (off_x1, off_y1, off_x2, off_y2) represent the position information of the sample value corresponding to the sample keyword (such as "123456") in the sample image (such as, the coordinates (off_x1, off_y1) represent the position coordinates of the upper left corner of the part of the image occupied by the sample value corresponding to the sample keyword, and the coordinates (x2, y2) represent the position coordinates of the lower right corner of the part of the image occupied by the sample value corresponding to the sample keyword, etc.).

[0051] By adopting the sample prediction position information corresponding to the sample keyword represented in the form of coordinates, it is possible to more accurately capture a sub-image in the sample image to obtain a sample sub-image that matches the sample prediction position information, and further perform text analysis on the sample sub-image. Compared with the prior art solution of performing text analysis on the original sample image, the present application can perform a more detailed analysis on the captured sample sub-image, thereby improving the accuracy of text feature extraction.

[0052] Step S24: training the model according to the first text feature corresponding to the sample keyword and the sample subgraph to obtain a trained model.

[0053] The trained model is used to locate the keyword to be verified in the image to be verified, and obtain the target position information of the keyword to be verified in the image to be verified.

[0054] In some embodiments, if the real text corresponding to the sample keyword is {identifier name: identifier value}, the first text feature corresponding to the sample keyword can be expressed as: {identifier name} in the picture, and {identifier value} in the picture. Compared with the feature representation method of only using "such as a picture of {identifier}" in the prior art, the first text feature corresponding to the sample keyword can be made more specific and detailed, so that when the first text feature corresponding to the sample keyword is used to analyze the sample sub-image, the analysis accuracy of the sample sub-image can be improved.

[0055] In some exemplary embodiments, step S24 trains the model according to the first text feature corresponding to the sample keyword and the sample subgraph to obtain the trained model, including: analyzing the sample subgraph according to the first text feature corresponding to the sample keyword to obtain an analysis result; determining a target loss value corresponding to the loss function according to the analysis result; adjusting the parameters of the model according to the target loss value to obtain the trained model.

[0056] The first text feature is a manually annotated text feature.

[0057] Based on the first text feature corresponding to the sample keyword, the information included in the sample sub-image can be analyzed to identify important information, image structure and other information in the sample sub-image, and obtain analysis results. The analysis results clarify the degree of match between the sample sub-image and the first text feature corresponding to the sample keyword, for example, the degree of similarity. Then, based on the obtained analysis results, the loss value output by the loss function is adjusted to obtain the target loss value. The target loss value reflects the difference between the first text feature corresponding to the sample sub-image and the sample keyword; and then back-propagation is used to adjust the parameters of the information localization model (such as the number of network layers in the information localization model, pooling parameters, etc.) through the target loss value, so that the adjusted information localization model is more suitable for locating information in the sample image.

[0058] It should be noted that, during the above training process, the above steps can be repeated, and the information localization model can be iteratively trained using multiple sample images until a preset maximum number of iterations is reached or the target loss value corresponding to the loss function converges.

[0059] After multiple iterations of training, the information localization model can gradually learn how to locate the position of the keywords in the image, so as to extract information about the keywords in the image and improve the accuracy of information localization.

[0060] In some exemplary embodiments, the sample sub-graph is analyzed according to the first text feature corresponding to the sample keyword to obtain an analysis result, including: extracting image features of the sample sub-graph to obtain image features; processing the image features using an autoregressive method to obtain a second text feature; determining a first similarity between the image feature and the first text feature corresponding to the sample keyword, and a second similarity between the second text feature and the first text feature; and determining the analysis result based on the first similarity and the second similarity.

[0061] Among them, by comparing the image feature to be processed and the first text feature corresponding to the sample keyword, the first similarity between the image feature to be processed and the first text feature corresponding to the sample keyword is determined, so that the first similarity can characterize the degree of information on whether the image feature to be processed can truly reflect the first text feature corresponding to the sample keyword.

[0062] It should be noted that the first similarity is obtained by converting the image features and text features into data (such as numbers in a matrix, etc.) respectively, and then calculating the similarity of the converted data; the first similarity is information that characterizes the degree to which the image features to be processed can truly reflect the real text features corresponding to the sample keywords.

[0063] Since the second text feature is a text feature obtained by processing the image feature to be processed in an autoregressive manner, for example, an autoregressive image model that can constrain the mutual dependence between image pixels can be used to analyze the image feature to be processed, clarify the inherent laws and characteristics of the image feature to be processed, and enable the obtained second text feature to capture the text information in the image feature to be processed, so as to facilitate the analysis of the image feature to be processed in a textual manner.

[0064] Furthermore, the second text feature is matched with the first text feature corresponding to the sample keyword in terms of similarity to obtain a second similarity to reflect the difference between the two text features.

[0065] Therefore, based on the first similarity and the second similarity, the analysis result of the feature analysis of the sample subgraph based on the first text feature corresponding to the sample keyword can be clearly identified, so as to use the analysis result to further adjust the parameters of the trained model to make the trained model more accurate.

[0066] In some exemplary embodiments, the analysis result is determined based on the first similarity and the second similarity, including: calculating the similarity mean of the first similarity and the second similarity; when the similarity mean is greater than a preset similarity threshold, determining the analysis result is that the sample image is a difficult sample; when the similarity mean is less than or equal to the preset similarity threshold, determining the analysis result is that the sample image is an easy sample.

[0067] The similarity mean can be calculated in any of the following ways:

[0068] 1) Calculate the arithmetic mean of the first similarity and the second similarity;

[0069] 2) Calculating the geometric mean of the first similarity and the second similarity;

[0070] 3) Calculating the root mean square value of the first similarity and the second similarity, that is, first respectively calculating the square value of the first similarity and the square value of the second similarity, then taking the average of the square value of the first similarity and the square value of the second similarity, and finally performing a square root operation on the average of the square value of the first similarity and the square value of the second similarity to obtain the root mean square value;

[0071] 4) Calculating the harmonic mean of the first similarity and the second similarity;

[0072] 5) Calculate the weighted average of the first similarity and the second similarity.

[0073] By comparing the similarity mean with the preset similarity threshold, the category of the sample image (i.e., a difficult sample or an easy sample) can be determined based on the comparison result, thereby adjusting the parameters of the information localization model based on the category of the sample image, which can improve the adjustment accuracy of the parameters of the information localization model and speed up the training of the information localization model.

[0074] In some exemplary embodiments, based on the analysis results, a target loss value corresponding to the loss function is determined, including: when the analysis result is that the sample image is a difficult sample, weighting the loss value output by the loss function according to a preset weight value to obtain the target loss value; when the analysis result is that the sample image is an easy sample, determining the target loss value to be the loss value output by the loss function.

[0075] Among them, difficult samples refer to those samples that are difficult to predict for the information localization model. Usually, the characteristics of these difficult samples are quite different from the knowledge that the information localization model has learned, which makes it difficult for the information localization model to accurately locate the information in the difficult samples.

[0076] During the training process, since difficult samples can provide more information, they can help the information localization model to better learn the localization features, thereby improving the generalization ability of the information localization model. However, since the prediction of difficult samples is difficult, it is necessary to weight the loss value output by the loss function according to the preset weight value to obtain the target loss value, so that the parameters of the trained information localization model can be adjusted through the target loss value, so that the parameters of the information localization model can be updated and the convergence speed of the information localization model can be accelerated.

[0077] Easy samples refer to those samples that are less difficult to predict for the information localization model. The characteristics of these easy samples are relatively consistent with the knowledge that the information localization model has learned, and the information localization model can easily accurately locate the information in these easy samples. During the training process, it is only necessary to use the loss value output by the loss function as the target loss value, without weighting the loss value output by the loss function, to meet the adjustment requirements of the parameters of the information localization model, thereby accelerating the convergence speed of the information localization model.

[0078] According to the embodiments of the present application, by obtaining the sample image, the sample text features of the sample keywords in the sample image, and the sample image features of the position of the sample keywords in the sample image, the features of the sample keywords in the sample image can be accurately described in two dimensions of the sample text features and the sample image features, which is convenient for subsequent analysis; then, according to the attention mechanism, the sample text features are used as the query vector, and the sample image features are used as the key-value pair vector to construct a model, and by using the model to achieve generalization processing of the text features and image features in the sample image, it is no longer necessary to extract the features in the sample image for a specific information arrangement structure, and a sample subgraph matching the information extraction requirements can be obtained; further, according to the first text features and the sample subgraph corresponding to the sample keyword, the model is trained so that the trained model can be applicable to locating the keywords to be verified in a variety of different types of images to be verified, obtaining the target position information of the keywords to be verified in the images to be verified, and improving the positioning accuracy of the keywords to be verified in the images to be verified, and then based on the obtained target position information of the keywords to be verified in the images to be verified, the text information corresponding to the keywords to be verified can be quickly and accurately extracted, thereby improving the information extraction efficiency of the images to be verified.

[0079] Figure 3 A schematic diagram of a method for acquiring text features and image features provided by an embodiment of the present application is shown. In some embodiments, the sample image includes a plurality of sample keywords.

[0080] Reference Figure 3In the above step S21, obtaining the sample image, the sample text features of the sample keyword in the sample image, and the sample image features of the position of the sample keyword in the sample image may further include the following steps.

[0081] Step S211, obtaining sample position information of each sample keyword in the sample image.

[0082] A sample keyword indicates key information corresponding to a sample information in a sample image, and the sample keyword corresponds to different sample position information in different sample images. For example, a sample keyword representing a "graphic number" corresponds to the sample position information "in the upper left corner of the first sample image" in the first sample image, while the sample position information corresponding to the sample keyword in the second sample image is "in the upper right corner of the second sample image".

[0083] Step S212: Generate a text description sentence set corresponding to each sample keyword according to each sample keyword and its corresponding sample position information.

[0084] For a sample image, the sample keyword extracted from the sample image corresponds one-to-one to the sample position information of the sample keyword in the sample image.

[0085] In some embodiments, the above correspondence is represented by a preset text template. For example, the preset text template is designed as "{key_name}, its value on the {position}". "Key_name" represents the sample keyword in the sample image, and "position" represents the sample position information of the sample keyword in the sample image. "Key_name" can also be represented as a synonym corresponding to the sample keyword.

[0086] Then, each sample keyword and its corresponding sample position information obtained are represented by the above-mentioned preset text template, and a set of text description sentences corresponding to each sample keyword can be obtained. The text description sentence set includes multiple text description sentences, and each text description sentence is a sentence represented by the above-mentioned preset text template.

[0087] The above method is used to obtain a set of text description sentences, which can enrich the sample position information, so as to obtain the sample position information corresponding to the same sample keyword expressed in different texts, which is convenient for subsequent model training.

[0088] In some embodiments, the sample keywords include initial sample keywords and derived keywords with the same meaning as the initial sample keywords. The step S212 generates a set of text description sentences corresponding to each sample keyword according to each sample keyword and its corresponding sample position information, which can be implemented in the following manner: generating a first type of text description sentence according to the initial sample keyword and its corresponding sample position information; and generating a second type of text description sentence according to the derived keyword and the sample position information corresponding to the initial sample keyword; and determining a set of text description sentences corresponding to the sample keyword according to the first type of text description sentence and the second type of text description sentence.

[0089] The first type of text description sentence is a sentence including the initial sample keyword and its corresponding sample position information (e.g., the approximate position of the initial sample keyword in the sample image, such as on the left side or on the right side of the sample image, etc.). The first type of text description sentence can be determined according to a preset text template.

[0090] For example, the preset text template is set to "{key_name}, its value on the {position}", the initial sample keyword is set to "key1", and the sample position information corresponding to the initial sample keyword is "down", then the first type of text description sentence can be represented as "{key1}, its value on the {down}".

[0091] Similarly, the second type of text description sentence is a sentence including a derived keyword (e.g., a keyword determined by semantic expansion or related word extraction) and sample position information corresponding to the initial sample keyword, and the second type of text description sentence can also be determined according to the above preset text template. For example, if the derived keyword is set to "key2", the second type of text description sentence can be represented as "{key2}, its value on the {down}".

[0092] It should be noted that since the derived keywords are derived from words with the same or similar meanings as the initial sample keywords, the derived keywords can include keywords related to the initial sample keywords (such as words with different emphases) and with more detailed information, so that the generated second category of text description sentences can complement the first category of text description sentences, thereby enriching the elements in the obtained text description sentence set.

[0093] In some embodiments, after obtaining the text description sentence set, repeated sentences in the text description sentence set may be removed to optimize the text description sentence set and ensure the accuracy and fluency of the sentences.

[0094] Furthermore, using sentences in a text description sentence set corresponding to sample keywords as a text data set for training an information location model can enrich the training data and improve the training accuracy of the information location model.

[0095] Step S213, analyzing the text description sentences in the text description sentence set corresponding to each sample keyword, determining the initial sample text features corresponding to each sample keyword, and using the image coding network to extract features from the sample image to obtain initial sample image features.

[0096] Among them, the image coding network can be implemented using a Swin transformer model or a vision transformer (ViT) model, and this application does not impose any restrictions on this.

[0097] It should be noted that the Swin transformer model is a deep learning model for semantic segmentation, which can convert the input image into a high-resolution segmented image to extract semantic information in the image.

[0098] For example, using an image coding network to extract features from a sample image to obtain initial sample image features can be achieved in the following manner: first, preprocessing operations (such as image scaling, image cropping, and normalization) are performed on the input sample image to adapt to the input requirements of the image coding network; then, a pre-trained image coding network (such as VGG, ResNet, EfficientNet, etc.) is used to extract features from the preprocessed sample image to obtain multiple candidate sample image features; further, a suitable layer is selected from the multiple candidate sample image features as a feature, for example, the output of a convolutional layer before the fully connected layer (or the output of the global pooling layer) is selected as the initial sample image feature, so as to capture the key information in the sample image.

[0099] In some exemplary embodiments, the text description sentence set includes multiple text description sentences. The step S213 of analyzing the text description sentences in the text description sentence set corresponding to each sample keyword to determine the initial sample text features corresponding to each sample keyword can be implemented in the following manner: performing word segmentation on the multiple text description sentences corresponding to the sample keywords to obtain word segmentation result information; and performing feature extraction on the word segmentation result information using a text encoding network to obtain the initial sample text features.

[0100] The word segmentation result information at least includes sample keywords. The text encoding network can be implemented using a large language model (LLM) that supports multiple languages, and this application does not limit this.

[0101] In some embodiments, the word segmentation result information is extracted using a text encoding network to obtain initial sample text features, including: converting the word segmentation result information into a numerical representation, such as converting each word into a vector of a fixed dimension through word embedding technology to obtain a text sequence after word segmentation; then, using a text encoding network (such as a recurrent neural network or a long short-term memory network, etc.) to encode the text sequence after word segmentation to capture the dependency and context information in the text sequence after word segmentation. Further, the output of the text encoding network is aggregated to obtain the initial sample text features.

[0102] The aggregation method can be to take the output of the last time step, or to perform average pooling (or maximum pooling, etc.) on the output of all time steps. In deep learning, the time step is a unit used to represent the time relationship in sequence data, which describes the time order of different events or behaviors.

[0103] It should be noted that since the quality of the word segmentation represented by the word segmentation result information is different, and the degree of vector conversion of each word using word embedding technology is also different, there are certain differences in the initial text features finally obtained.

[0104] By processing multiple text description sentences corresponding to sample keywords in the above manner, initial sample text features corresponding to the sample keywords can be extracted. Moreover, since the text description sentences in the text description sentence set corresponding to the sample keywords are constructed based on the sample keywords and derived keywords with similar meanings, the dimensions of the initial sample text features can be enriched, which facilitates the subsequent training of the information positioning model.

[0105] Step S214, according to the attention mechanism, align the initial sample text features and the initial sample image features corresponding to each sample keyword to obtain the sample text features corresponding to each sample keyword and the sample image features of the position of each sample keyword in the sample image.

[0106] Among them, the initial sample text features and the initial sample image features corresponding to each sample keyword are aligned, so that the initial sample text features and the initial sample image features can refer to each other's feature expressions to enhance their respective feature information, thereby making the obtained sample text features and sample image features more accurate.

[0107] For example, if the initial sample text feature is a text feature describing "key", and the initial sample image feature is an image feature describing "down", but in the sample image, the actual position information corresponding to the sample keyword "key" is "right", then by aligning the initial sample text feature and the initial sample image feature, the obtained sample text feature can be a text feature describing "key", and correspondingly, the sample image feature can be an image feature describing "right". In this way, the sample text feature and the sample image feature match the sample keyword in the sample image and its corresponding actual position information.

[0108] In some exemplary embodiments, in step S214, according to the attention mechanism, the initial sample text features and the initial sample image features corresponding to each sample keyword are aligned to obtain the sample text features corresponding to each sample keyword and the sample image features of the position of each sample keyword in the sample image. This can be implemented in the following manner: the initial sample text features are processed according to the self-attention mechanism to obtain the processed text features; the initial sample image features are processed according to the deformable attention mechanism to obtain the processed image features; the processed text features and the processed image features are updated according to the cross-modal attention mechanism, respectively, to obtain the sample text features corresponding to the sample keywords and the sample image features of the positions of the sample keywords in the sample image.

[0109] For example, Figure 4 A schematic diagram of a network structure of feature alignment provided by an embodiment of the present application is shown. Figure 4 As shown, the network structure includes a self-attention module 401, a deformable attention module 402, a first cross-modal attention module 403, a second cross-modal attention module 404, a first framework flexible network 405 and a second framework flexible network 406.

[0110] Among them, the self-attention module 401 is a module that uses the self-attention mechanism (Self-Attention, SA) for data processing; the deformable attention module 402 is a module that uses the deformable self-attention mechanism (Deformable Self-Attention, DSA) for data processing.

[0111] The first cross-modal attention module 403 and the second cross-modal attention module 404 are both modules capable of implementing a cross-modal attention mechanism (Cross Attention, CA).

[0112] Both the first frame flexible network 405 and the second frame flexible network 406 may be implemented by using a frame flexible network (Frame Flexible Network, FFN).

[0113] Figure 5 A flowchart of a method for aligning text and images provided in an embodiment of the present application is shown. Figure 5 As shown, using Figure 4 The network structure in processes the acquired initial sample text features and initial sample image features to obtain sample text features of sample keywords and sample image features of the positions of sample keywords in sample images. The method includes the following steps.

[0114] Step S501, obtaining initial sample text features.

[0115] Step S502: input the initial sample text features into the self-attention module 401 for processing to obtain processed text features.

[0116] It should be noted that, while executing steps S501 to S502, steps S503 to S504 may also be executed.

[0117] Step S503, obtaining initial sample image features.

[0118] Step S504, input the initial sample image features into the deformable attention module 402 for processing to obtain processed image features.

[0119] After obtaining the processed text features and the processed image features, the processed text image and the processed image features may be cross-referenced and processed with a cross-modal attention mechanism to continue with the following steps.

[0120] In step S505 , the first cross-modal attention module 403 updates the text features using the processed text features as a query vector and the processed image features as a key-value pair vector.

[0121] The number of query vectors may be the number of text codes or a preset query vector threshold (eg, 300, 350, etc.), which is not limited in this application and will not be described in detail.

[0122] In step S506, the second cross-modal attention module 404 updates the image features using the processed image features as a query vector and the processed text features as a key-value pair vector.

[0123] Step S507 : the first framework flexible network 405 processes the updated text features to obtain sample text features corresponding to the sample keywords.

[0124] Among them, the first framework flexible network 405 can dynamically adjust the amount of calculation according to the number of input frames to significantly reduce the memory cost for storing text features.

[0125] Step S508 : the second framework flexible network 406 processes the updated image features to obtain sample image features of the positions of the sample keywords in the sample images.

[0126] By processing the text features and image features respectively in the above manner, the sample image features of the sample keyword at the position in the sample image finally obtained can be integrated with the text information corresponding to the sample keyword to improve the refinement of the sample image features; correspondingly, the sample text features corresponding to the sample keyword finally obtained can also be integrated with the image information corresponding to the sample keyword to improve the refinement of the sample text features.

[0127] In this embodiment, by using each sample keyword and its corresponding sample position information, a text description sentence set corresponding to each sample keyword is generated, so that each type of text description sentence in the text description sentence set can characterize the sample position information of the sample keyword in the sample image, so as to enrich the position expression corresponding to the sample keyword; then, the text description sentences in the text description sentence set corresponding to each sample keyword are analyzed, for example, the sentences are segmented to obtain the segmentation results, and the text features and image features of the segmentation results are extracted, so that the initial sample text features and initial sample image features corresponding to each sample keyword can be obtained, and the sample keyword is characterized from two dimensions; further, according to the attention mechanism, the initial sample text features and initial sample image features corresponding to each sample keyword are aligned, so that the obtained sample text features of each sample keyword and the sample image features of the position of the sample keyword in the sample image are all feature expressions for the same sample keyword at the same position in the sample image, so that the sample text features and sample image features corresponding to each sample keyword are matched one by one, which is convenient for subsequent model training.

[0128] In some exemplary embodiments, the model training method also includes: determining the to-be-processed position information that needs to be weighted in the sample prediction position information based on the sample position information of the sample keyword in the sample image, and determining the position weight coefficient corresponding to the to-be-processed position information based on the aspect ratio of the sample image; using the position weight coefficient to weight the to-be-processed position information to obtain weighted position information; and updating the sample prediction position information based on the weighted position information.

[0129] Among them, since the sample position information of the sample keyword in the sample image has a certain directional tendency, the position information to be processed that needs to be weighted in the sample prediction position information can be clarified based on the directional tendency. For example, if the sample position information of the sample keyword in the sample image tends to represent that the sample keyword is on the right side of the sample image (such as the coordinates of the upper right corner, or the coordinates of the lower right corner, etc.), the position coordinates representing the right side of the sample image in the sample position information of the sample keyword in the sample image can be extracted as the position information to be processed.

[0130] Then, by obtaining the width and height of the sample image, the aspect ratio of the sample image is calculated, and the position weight coefficient corresponding to the position information to be processed is determined using the aspect ratio of the sample image as a scale. For example, if the aspect ratio of the sample image is 2:1, the position weight coefficient corresponding to the position information to be processed can be set to 2. Since the aspect ratio of the sample image can characterize the layout and shape of the sample image, the position weight coefficient determined based on the aspect ratio of the sample image can more accurately reflect the position information to be processed.

[0131] In some embodiments, the position weight coefficient corresponding to the position information to be processed may also be determined based on information such as the importance of the position of the sample keyword in the sample image and the density of the sample keyword.

[0132] Further, the position information to be processed is weighted using the position weight coefficient, for example, the position weight coefficient is multiplied by each position coordinate in the position information to be processed, so as to obtain the weighted position information. Then, the weighted position information is integrated with the original sample prediction position information to obtain the updated sample prediction position information.

[0133] For example, the weighted position information is used to directly replace the original position information to be processed in the sample prediction position information to obtain the updated sample prediction position information.

[0134] For another example, a weighted averaging method is used to integrate the position information of the sample prediction position information excluding the original position information to be processed and the weighted position information to obtain updated sample prediction position information.

[0135] The above method is used to weight the pending position information that needs to be weighted in the sample prediction position information, and based on the weighted position information, the sample prediction position information is updated. This can make the updated sample prediction position information more accurately reflect the actual position of the sample keyword in the sample image, thereby training the information positioning model based on the updated sample prediction position information, which can improve the positioning accuracy of the information positioning model.

[0136] Figure 6FIG. 1 is a schematic diagram showing a training process of an information positioning model provided in an embodiment of the present application. Figure 6 As shown in FIG. 6 , in the training process of the information localization model, it is necessary to sequentially pass through a text encoder 601, an image encoder 602, a feature enhancement module 603, a cross-modal decoder 604, and a parameter adjustment module 605 based on a loss function to obtain a trained information localization model.

[0137] First, the preset text template is designed as “{key_name}, its value on the {position}”, where “key_name” represents the sample keyword in the sample image, and “position” represents the sample position information of the sample keyword in the sample image.

[0138] For example, the sample position information can be "left side (left)", "right side (right)", "down (down)", "up (up)", "top right (right top)", "bottom right (right down)", etc. The above sample position information is only an example for explanation and can be set according to actual needs (such as the sample keyword in certain sample card images and the sample position relationship of the sample keyword in the sample card image). This application does not impose any restrictions on this and will not go into details here.

[0139] In some embodiments, the sample keywords may include an initial sample keyword and a derivative keyword with the same meaning as the initial sample keyword (e.g., a synonym of the initial sample keyword, etc.). For example, the initial sample keyword is input into a natural language processing tool based on artificial intelligence technology to find a derivative keyword with the same meaning as the initial sample keyword; then, the initial sample keyword and the derivative keyword are respectively substituted into the above text template to generate a text description sentence set, which includes a first type of text description sentence and a second type of text description sentence.

[0140] For example, the first type of text description sentence is expressed as “{key1}, its value on the {down}”, “{key2}, its value on the {right down}”, …, “{keyn}, its value on the {right top}”, where “keyn” represents the nth initial sample keyword, and n is an integer greater than or equal to 1.

[0141] The second type of text description sentences are expressed as “{key1} is also called {a synonym of key1}, its value is on the {right}”, “{key2} is also called {a synonym of key2}, its value is on the {right down}”, …, “{keyn} is also called {a synonym of keyn}, its value is on the {right top}”.

[0142] There may be multiple derived keywords, so the second type of text description sentence may be described as a sentence including multiple different derived keywords, thereby expanding the sample keyword and its position information in the sample image.

[0143] It should be noted that different sample images include different numbers of sample keywords. By extracting text features of each sample image, the sample keywords in different sample images can be determined. In other words, the content and number of sample keywords vary.

[0144] Further, each text description sentence in the text description sentence set is segmented to obtain segmentation result information, which includes at least the sample keyword. For example, each text description sentence in the text description sentence set is input into a preset segmenter for segmentation, with the sample keyword as an independent unit, and then other information in the text description sentence except the sample keyword is segmented to ensure the integrity of the sample keyword.

[0145] Taking "{key1}, its value on the {down}" as an example, the corresponding word segmentation result information includes "key1" and the word segmentation result of "its value on the {down}".

[0146] Then, the word segmentation result information is input as sample text into the text encoder 601 for feature extraction to obtain the initial sample text feature T0; at the same time, the sample image is input into the image encoder 602 for feature extraction to obtain the initial sample image feature V0.

[0147] Furthermore, the initial sample text feature T0 and the initial sample image feature V0 are input into the feature enhancement model 603 for feature enhancement, and the enhanced features of two different dimensions are aligned to obtain enhanced text features and enhanced image features.

[0148] Among them, the method of aligning the enhanced features of two different dimensions can be adopted in this application. Figure 4-5 The method shown is implemented and will not be described in detail here.

[0149] It should be noted that the obtained enhanced text features and enhanced image features can be used to describe the same sample keyword at the same position, so that the enhanced text features and enhanced image features can be more suitable for training the information location model.

[0150] For example, if the initial sample text feature T0 is a text feature describing "key1", and the initial sample image feature V0 is an image feature describing "down", but in the sample image, the actual position information corresponding to the sample keyword "key1" is "right", then the enhanced features of two different dimensions are aligned through the feature enhancement model 603, so that the enhanced text feature obtained is the text feature describing "key1", and correspondingly, the enhanced image feature is the image feature describing "right". In this way, the enhanced text feature and the enhanced image feature match the sample keyword in the sample image and its corresponding actual position information.

[0151] Then, the enhanced text features and the enhanced image features are input into the cross-modal decoder 604 for processing to obtain an information localization model (or, referred to as feature space F).

[0152] Among them, the cross-modal decoder 604 can be implemented based on the attention mechanism. The cross-modal decoder 604 can use the enhanced text features as the query vector and the enhanced image features as the key-value pair vector to construct an information location model; the information location model can predict the position of the sample keyword in the sample image, so as to obtain the sample prediction position information corresponding to the sample keyword; for example, a coordinate system is constructed with the long side of the sample image as the horizontal coordinate and the short side of the sample image as the vertical coordinate, then in this coordinate system, the sample prediction position information corresponding to the sample keyword can be expressed as (x1, y1, x2, y2, off_x1, off_y1, off_x2, off_y2).

[0153] Among them, the coordinates (x1, y1, x2, y2) represent the location information of the sample keyword (such as "document number") in the sample image, and the coordinates (off_x1, off_y1, off_x2, off_y2) represent the location information of the sample value corresponding to the sample keyword (such as "123456") in the sample image.

[0154] In some embodiments, off_x1 represents the position information of the sample value corresponding to the sample keyword in the sample image, and the offset relative to the position information of the sample keyword in the sample image.

[0155] Since the sample prediction position information corresponding to the sample keyword is represented by 8 values, and different coordinates can be adjusted with corresponding weights according to the position information in the text description to update the sample prediction position information, the updated sample prediction position information is more accurate.

[0156] For example, if the position information in the text description is "right", the corresponding weight coefficient a may be added to the coordinate values ​​"x1, x2, off_x1, and off_x2" respectively, and the weight coefficient a is a coefficient determined according to the actual aspect ratio of the sample image.

[0157] In some embodiments, the sample prediction position information corresponding to the sample keyword can also be updated as follows: based on the sample position information of the sample keyword in the sample image, determine the pending position information that needs to be weighted in the sample prediction position information, and determine the position weight coefficient corresponding to the pending position information based on the aspect ratio of the sample image; use the position weight coefficient to weight the pending position information to obtain the weighted position information; and update the sample prediction position information based on the weighted position information.

[0158] For example, in a sample image with an aspect ratio of 60:30, if the position information to be processed represents that the sample keyword is on the right side of the sample image, the weight coefficients added to the coordinate values ​​"x1, x2, off_x1 and off_x2" are 60 / 30=2 respectively, and the other coordinate values ​​in the sample prediction position information are not weighted, so as to obtain the updated sample prediction position information.

[0159] For another example, in the sample image with an aspect ratio of 60:30, if the position information to be processed represents the lower right side of the sample keyword in the sample image, the weight coefficients added to the coordinate values ​​"x2, y2, off_x2 and off_y2" can be 2 respectively, and the other coordinate values ​​in the sample prediction position information are not weighted, so as to obtain the updated sample prediction position information.

[0160] If the model is made to pay more attention to the direction of x2, the weighted position coordinate of the coordinate value "x2" can be determined as "2*|x2_gt-x2_pred|", where x2_gt represents the true value of the coordinate and x1_pred represents the predicted value of the coordinate.

[0161] Then, according to the sample prediction position information corresponding to the sample keyword, the sample image is intercepted to obtain a sample sub-image corresponding to the sample prediction position information; and the image features of the sample sub-image are extracted to obtain the image features to be processed; the image features to be processed are processed by autoregression to obtain the second text features; the first similarity between the image features to be processed and the first text features corresponding to the sample keyword, and the second similarity between the second text features and the first text features corresponding to the sample keyword are determined; the first similarity and the second similarity are input into the parameter adjustment module 605 based on the loss function to adjust the model parameters.

[0162] The autoregressive method is a method for statistically processing time series, for example, an autoregressive model is used to characterize the autoregressive method. The autoregressive model uses the feature information of the same variable (such as the image feature to be processed) in previous periods to predict the feature information of the image feature to be processed in the current period, and assumes that the image feature to be processed changes linearly in different periods.

[0163] For example, by using an autoregressive model to predict the image feature to be processed, a second text feature (such as a string of text data, etc.) describing the image feature to be processed can be obtained, so that the second text feature can represent the text feature information of the image feature to be processed.

[0164] Specifically, the parameter adjustment module 605 based on the loss function needs to first average the first similarity and the second similarity to obtain the similarity mean, then determine whether the similarity mean is greater than a preset similarity threshold, and adjust the parameters of the model according to the determination result.

[0165] For example, if the similarity mean is greater than the preset similarity threshold, it means that the sample image is a difficult sample, and it is necessary to use the preset weight value to weight the loss value output by the loss function to obtain the target loss value, and then use the target loss value to adjust the parameters of the information positioning model. After the parameter adjustment is completed, the trained model is obtained.

[0166] For another example, if the similarity mean is less than or equal to the preset similarity threshold, it means that the sample image is an easy sample. The loss value output by the loss function can be directly used to adjust the parameters of the information localization model. After the parameter adjustment is completed, the trained model is obtained.

[0167] Among them, the loss function can be implemented using absolute error loss (L1 Loss, Mean Absolute Error, MAE) or mean square error loss (L2 Loss, Mean Squared Error, MSE).

[0168] It should be noted that, in the above training process, the information localization model can be iteratively trained for different sample images, so that the trained model finally obtained can be more universal, that is, suitable for information localization of card images with various information arrangement structures.

[0169] Figure 7 A schematic diagram of a flow chart of an information extraction method provided in an embodiment of the present application is shown. Figure 7 , the information extraction method includes:

[0170] Step S701: Acquire an image to be verified, and extract keywords to be verified from the image to be verified.

[0171] The keyword to be verified is text information that needs to be identified in the image to be verified. For example, the keyword to be verified may be a "certificate number" in a card image.

[0172] Step S702: Use the trained model to locate the keyword to be verified, and obtain target position information of the keyword to be verified in the image to be verified.

[0173] Among them, the trained model is obtained according to the training method of any model in the embodiments of the present application.

[0174] Step S703: extracting information to be verified that matches the target position information from the image to be verified according to the target position information.

[0175] The target position information can be used to extract a sub-image to be verified that matches the target position information from the image to be verified, and then the sub-image to be verified is analyzed to obtain the information to be verified. For example, an autoregressive model is used to perform feature analysis on the sub-image to be verified to obtain the information to be verified.

[0176] Step S704: input the information to be verified into a preset text detection model for text recognition to obtain text information corresponding to the keyword to be verified.

[0177] The preset text detection model may be a preset optical character recognition (OCR) model capable of performing text recognition.

[0178] According to an embodiment of the present application, by using a trained model to locate the keyword to be verified extracted from the image to be verified, the target position information of the keyword to be verified in the image to be verified is obtained, and the trained model can be used to realize generalized recognition of text features and image features in the image to be verified. There is no need to extract features in the image to be verified for a specific information arrangement structure, and the target position information of the keyword to be verified in the image to be verified can be obtained, so that the information to be verified that matches the target position information can be more accurately extracted from the image to be verified based on the target position information; further, the information to be verified is input into a preset text detection model for text recognition, and the text information corresponding to the keyword to be verified is obtained, thereby realizing accurate recognition of the text information.

[0179] It can be understood that the above-mentioned various method embodiments mentioned in this application can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, this application will not go into details. Those skilled in the art can understand that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined by its function and possible internal logic.

[0180] In addition, the present application also provides a model training device, which can be used to implement any model training method provided in the present application. The corresponding technical solutions and descriptions are referred to in the corresponding records of the method part and will not be repeated here.

[0181] Figure 8 A block diagram showing the composition of a model training device provided in an embodiment of the present application.

[0182] Reference Figure 8 , the embodiment of the present application provides a model training device 800, the model training device 800 includes:

[0183] The first acquisition module 801 is used to acquire a sample image, a sample text feature of a sample keyword in the sample image, and a sample image feature of a position of the sample keyword in the sample image.

[0184] The sample text feature is a text feature that characterizes a sample keyword in a sample image, and the sample image feature is an image feature that characterizes the position of the sample keyword in the sample image.

[0185] The construction module 802 is used to construct a model based on the attention mechanism, using the sample text features as the query vector and the sample image features as the key-value pair vector.

[0186] The prediction module 803 is used to use the model to predict the position of the sample keyword in the sample image, obtain the sample prediction position information corresponding to the sample keyword, and determine the sample sub-image according to the sample image and the sample prediction position information.

[0187] The training module 804 is used to train the model according to the first text feature corresponding to the sample keyword and the sample subgraph to obtain a trained model.

[0188] The trained model is used to locate the keyword to be verified in the image to be verified, and obtain the target position information of the keyword to be verified in the image to be verified.

[0189] It should be noted that the model training device in this example can implement any model training method provided in the embodiments of the present application, which will not be repeated here.

[0190] According to the embodiment provided by the present application, a sample image, sample text features of a sample keyword in the sample image, and sample image features of the position of the sample keyword in the sample image are obtained by a first acquisition module, and the features of the sample keyword in the sample image can be accurately described in two dimensions of sample text features and sample image features, which is convenient for subsequent analysis; then, a construction module is used to construct a model based on an attention mechanism, using the sample text features as a query vector and the sample image features as a key-value pair vector, and the model is constructed. By using the model, generalized processing of text features and image features in the sample image is achieved, and there is no need to extract features in the sample image for a specific information arrangement structure, so that a sample subgraph matching the information extraction requirements can be obtained; further, a training module is used to train the model based on the first text features and sample subgraphs corresponding to the sample keywords, so that the trained model can be applied to locate the keywords to be verified in a variety of different types of images to be verified, obtain the target position information of the keywords to be verified in the images to be verified, and improve the positioning accuracy of the keywords to be verified in the images to be verified, and then based on the obtained target position information of the keywords to be verified in the images to be verified, the text information corresponding to the keywords to be verified can be quickly and accurately extracted, thereby improving the efficiency of information extraction from the images to be verified.

[0191] The present application also provides an information extraction device, which can be used to implement any information extraction device method provided in the present application. The corresponding technical solutions and descriptions can be found in the corresponding records in the method part and will not be repeated here.

[0192] Fig. 9 A block diagram showing the composition of an information extraction device provided in an embodiment of the present application is shown.

[0193] Reference Fig. 9 , the embodiment of the present application provides an information extraction device 900, the information extraction device 900 comprising:

[0194] The second acquisition module 901 is used to acquire the image to be verified and extract the keyword to be verified in the image to be verified.

[0195] The positioning module 902 is used to use the trained model to locate the keyword to be verified and obtain the target position information of the keyword to be verified in the image to be verified; wherein the trained model is obtained according to the training method of any model in the examples of this application.

[0196] The matching module 903 is used to extract the information to be verified that matches the target position information from the image to be verified according to the target position information;

[0197] The text recognition module 904 is used to input the information to be verified into a preset text detection model to perform text recognition, and obtain text information corresponding to the keyword to be verified.

[0198] It should be noted that the information extraction device in this example can implement any information extraction method provided in the embodiments of the present application, which will not be described in detail here.

[0199] According to the embodiment provided by the present application, the image to be verified is obtained by using the second acquisition module, and the keywords to be verified in the image to be verified are extracted, and the positioning module is used to locate the keywords to be verified extracted from the image to be verified based on the trained model to obtain the target position information of the keywords to be verified in the image to be verified. The trained model can be used to realize generalized recognition of text features and image features in the image to be verified, and there is no need to extract features in the image to be verified for a specific information arrangement structure, so as to obtain the target position information of the keywords to be verified in the image to be verified; the matching module is used to extract information to be verified that matches the target position information from the image to be verified according to the target position information, so that the obtained information to be verified is more accurate; further, the text recognition module is used to input the information to be verified into a preset text detection model for text recognition, so as to obtain text information corresponding to the keywords to be verified, and accurate recognition of the text information is realized.

[0200] The training device of the above model and each module in the information extraction device can be implemented in whole or in part by software, hardware and their combination. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0201] Fig.10 A block diagram of the composition of an electronic device provided in an embodiment of the present application is shown.

[0202] Reference Fig.10, an embodiment of the present application provides an electronic device, the electronic device comprising: at least one processor 1001, at least one memory 1002, and one or more I / O interfaces 1003. The processor 1001, the memory 1002, and the I / O interface 1003 are interconnected via a bus 1004. The memory 1002 stores one or more computer programs, and the one or more computer programs are executed by at least one processor 1001, so that at least one processor 1001 can implement any model training method or information extraction method recorded in the above embodiments.

[0203] Each module in the above electronic device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0204] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored, wherein the computer program implements the above-mentioned model training method or information extraction method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0205] An embodiment of the present application also provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned model training method, or the information extraction method.

[0206] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium).

[0207] As is known to those of ordinary skill in the art, the term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is known to those of ordinary skill in the art that communication media typically contain computer-readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.

[0208] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0209] The computer program instructions for performing the operation of the present application can be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions can be executed completely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). In some embodiments, by using the state information of a computer-readable program instruction to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA), the electronic circuit can execute a computer-readable program instruction, thereby realizing various aspects of the present application.

[0210] The computer program product described herein may be implemented in hardware, software, or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0211] Various aspects of the present application are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0212] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0213] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0214] The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of an instruction, and the module, a program segment or a part of an instruction contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, and the execution order between the boxes can be determined according to the functions involved in each box. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0215] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly noted, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present application as set forth in the appended claims.

Claims

1. A model training method, characterized in that: include: Acquire a sample image, a sample text feature of a sample keyword in the sample image, and a sample image feature of a position of the sample keyword in the sample image; Based on the attention mechanism, the sample text features are used as query vectors, and the sample image features are used as key-value pair vectors to construct a model; Using the model to predict the position of the sample keyword in the sample image, obtaining sample prediction position information corresponding to the sample keyword, and determining a sample sub-image according to the sample image and the sample prediction position information; The model is trained according to the first text feature corresponding to the sample keyword and the sample subgraph to obtain a trained model.

2. The method according to claim 1, characterized in that The determining of the sample sub-image according to the sample image and the sample predicted position information comprises: According to the sample prediction position information corresponding to the sample keyword, the sample image is intercepted to obtain a sample sub-image corresponding to the sample prediction position information.

3. The method according to claim 1, characterized in that The step of training the model according to the first text feature corresponding to the sample keyword and the sample subgraph to obtain the trained model includes: Analyzing the sample subgraph according to a first text feature corresponding to the sample keyword to obtain an analysis result, wherein the first text feature is a manually annotated text feature; According to the analysis result, determining a target loss value corresponding to the loss function; According to the target loss value, the parameters of the model are adjusted to obtain the trained model.

4. The method according to claim 3, characterized in that The step of analyzing the sample subgraph according to the first text feature corresponding to the sample keyword to obtain an analysis result includes: Extracting image features from the sample sub-image to obtain image features; Processing the image feature in an autoregressive manner to obtain a second text feature; Determining a first similarity between the image feature and a first text feature corresponding to the sample keyword, and a second similarity between the second text feature and the first text feature; The analysis result is determined according to the first similarity and the second similarity.

5. The method according to claim 4, characterized in that The determining the analysis result according to the first similarity and the second similarity includes: Calculating a similarity average of the first similarity and the second similarity; When the similarity mean is greater than a preset similarity threshold, determining that the analysis result is that the sample image is a difficult sample; When the similarity mean is less than or equal to the preset similarity threshold, it is determined that the analysis result is that the sample image is an easy sample.

6. The method according to claim 5, wherein determining the target loss value corresponding to the loss function according to the analysis result comprises: When the analysis result indicates that the sample image is the difficult sample, weighting the loss value output by the loss function according to a preset weight value to obtain the target loss value; When the analysis result is that the sample image is the easy sample, the target loss value is determined to be the loss value output by the loss function.

7. The method according to claim 1, characterized in that The sample image includes a plurality of sample keywords; The acquiring of the sample image, the sample text features of the sample keyword in the sample image, and the sample image features of the position of the sample keyword in the sample image includes: Obtaining sample position information of each of the sample keywords in the sample image; According to each of the sample keywords and the corresponding sample position information, a text description sentence set corresponding to each of the sample keywords is generated; Analyze the text description sentences in the text description sentence set corresponding to each of the sample keywords, determine the initial sample text features corresponding to each of the sample keywords, and use an image coding network to extract features from the sample image to obtain the initial sample image features; According to the attention mechanism, the initial sample text features and the initial sample image features corresponding to each of the sample keywords are aligned to obtain the sample text features corresponding to each of the sample keywords and the sample image features of the position of each of the sample keywords in the sample image.

8. The method according to claim 7, characterized in that The text description sentence set includes a plurality of the text description sentences; The analyzing the text description sentences in the text description sentence set corresponding to each of the sample keywords to determine the initial sample text features corresponding to each of the sample keywords includes: Performing word segmentation on a plurality of text description sentences corresponding to the sample keywords respectively to obtain word segmentation result information, wherein the word segmentation result information at least includes the sample keywords; A text encoding network is used to perform feature extraction on the word segmentation result information to obtain the initial sample text features.

9. The method according to claim 7, characterized in that: The method of aligning the initial sample text features and the initial sample image features corresponding to each sample keyword according to the attention mechanism to obtain the sample text features corresponding to each sample keyword and the sample image features of the position of each sample keyword in the sample image includes: Processing the initial sample text features according to the self-attention mechanism to obtain processed text features; Processing the initial sample image features according to a deformable attention mechanism to obtain processed image features; According to the cross-modal attention mechanism, the processed text features and the processed image features are updated respectively to obtain the sample text features corresponding to the sample keywords and the sample image features of the positions of the sample keywords in the sample images.

10. The method according to claim 7, characterized in that The sample keywords include initial sample keywords and derived keywords with the same meaning as the initial sample keywords; The step of generating a text description sentence set corresponding to each sample keyword according to each sample keyword and its corresponding sample position information includes: Generate a first type of text description sentence based on the initial sample keyword and its corresponding sample position information; and generate a second type of text description sentence based on the derived keyword and the sample position information corresponding to the initial sample keyword; According to the first category of text description sentences and the second category of text description sentences, a text description sentence set corresponding to the sample keyword is determined.

11. The method according to any one of claims 1 to 10, characterized in that The method further comprises: Determine, according to the sample position information of the sample keyword in the sample image, the position information to be processed that needs to be weighted in the sample prediction position information, and determine the position weight coefficient corresponding to the position information to be processed according to the aspect ratio of the sample image; Using the position weight coefficient to weight the position information to be processed to obtain weighted position information; The sample prediction position information is updated according to the weighted position information.

12. An information extraction method, characterized in that: include: Acquire an image to be verified, and extract keywords to be verified from the image to be verified; Using the trained model to locate the keyword to be verified, and obtaining target position information of the keyword to be verified in the image to be verified; wherein the trained model is obtained according to the model training method according to any one of claims 1 to 11; Extracting information to be verified that matches the target position information from the image to be verified according to the target position information; The information to be verified is input into a preset text detection model for text recognition to obtain text information corresponding to the keyword to be verified.

13. A model training device, characterized in that: include: A first acquisition module is used to acquire a sample image, a sample text feature of a sample keyword in the sample image, and a sample image feature of a position of the sample keyword in the sample image; A construction module, for constructing a model based on an attention mechanism, taking the sample text features as a query vector and the sample image features as a key-value pair vector; A prediction module, used to predict the position of the sample keyword in the sample image using the model, obtain sample prediction position information corresponding to the sample keyword, and determine a sample sub-image according to the sample image and the sample prediction position information; The training module is used to train the model according to the first text feature corresponding to the sample keyword and the sample subgraph to obtain a trained model.

14. An information extraction device, characterized in that: include: A second acquisition module is used to acquire the image to be verified and extract the keyword to be verified in the image to be verified; a positioning module, used to use the trained model to locate the keyword to be verified, and obtain the target position information of the keyword to be verified in the image to be verified; wherein the trained model is obtained according to the training method of the model according to any one of claims 1 to 11; A matching module, configured to extract information to be verified that matches the target position information from the image to be verified according to the target position information; The text recognition module is used to input the information to be verified into a preset text detection model to perform text recognition and obtain text information corresponding to the keyword to be verified.

15. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, and the one or more computer programs are executed by the at least one processor so that the at least one processor can execute the model training method described in any one of claims 1 to 11, or the information extraction method described in claim 12.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When executed by a processor, the computer program implements the model training method as described in any one of claims 1 to 11, or the information extraction method as described in claim 12.

17. A computer program product, characterized in that It includes a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device implements the training method of the model as described in any one of claims 1 to 11, or the information extraction method as described in claim 12.

Images