Card identification model training method and card structured information identification method

Abstract

The embodiment of the invention discloses a training method of a card identification model and an identification method of card structured information, which are used for solving the problem of low card information identification efficiency. Comprising the steps of obtaining a plurality of sample card pictures of to-be-extracted structured information, inputting the sample card pictures into a card identification model, identifying multiple pieces of key information of the sample card pictures through the card identification model, extracting all keywords in the sample card pictures and position data of key values corresponding to the keywords based on the multiple pieces of key information, a sample card picture is input into a picture encoder to obtain a picture feature; the text feature and the picture feature are mapped to a multi-modal space; position prediction is performed on a keyword and a key value; a spatial position coordinate and a spatial position offset coordinate are output; and updating model parameters in the card identification model according to the corresponding real position coordinates in the sample card picture.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a training method for a card recognition model and a method for recognizing structured information of cards. Background Technology

[0002] In daily life, card and document information recognition is a common application scenario, such as recognizing ID cards, driver's licenses, business licenses, and value-added tax invoices. When a business system receives a user's card or document, it needs to combine and output the information contained in the card or document image in the format of <keyword, value>, such as <name, Zhang San>.

[0003] In related technologies, the extraction of structured information from cards and certificates typically involves setting multiple templates or training different models for the same type of card or certificate. This can be done by differentiating keywords between templates or by varying the relative positions of the same keywords within the card or certificate image. Furthermore, text detection and recognition technologies need to be integrated when extracting structured information from card or certificate images. Setting different templates or training different models is overly cumbersome, and problems during text detection and recognition can significantly impact the extracted structured information, leading to low accuracy and inefficient card and certificate information recognition. Summary of the Invention

[0004] The purpose of this application is to provide a training method for a card recognition model and a method for recognizing structured card information, so as to solve the problem of low efficiency in recognizing card information.

[0005] To solve the above-mentioned technical problems, the embodiments of this application are implemented as follows: In a first aspect, embodiments of this application provide a training method for a card and document recognition model, comprising: acquiring multiple sample card and document images from which structured information is to be extracted; inputting the sample card and document images into a card and document recognition model; identifying multiple key information of the sample card and document images through the card and document recognition model; and extracting the location data of all keywords and the key values ​​corresponding to the keywords in the sample card and document images based on the multiple key information, wherein the key information is used to describe the sample card and document images. The location data of the keywords and key values ​​are input into the text encoder to obtain text features, and the sample card image is input into the image encoder to obtain image features. The text features and the image features are mapped to a multimodal space. The location of the keywords and key values ​​of the text features in the multimodal space is predicted, and the spatial location coordinates of the keywords and the spatial location offset coordinates of the key values ​​are output. The spatial location offset coordinates include the offset of the key value relative to the spatial location coordinates. Based on the spatial coordinates of the keywords and the spatial offset coordinates of the key values, as well as the position coordinates of the keywords and the position coordinates of the key values ​​in the sample card image, the model parameters in the card recognition model are updated.

[0006] Secondly, embodiments of this application provide a method for identifying structured information of cards and certificates, including: A card image containing structured information to be identified is obtained. The card image is then input into a trained card recognition model, which identifies multiple key pieces of information in the card image. Based on these key pieces of information, the location data of all keywords in the card image and the key values ​​corresponding to those keywords are obtained. The card recognition model is trained using the card recognition model training method described above. The card recognition model extracts text features of the location data of multiple keywords and key values, as well as image features of the card image; The card recognition model maps the text features and image features into a multimodal space, and predicts the positions of the keywords and key values ​​of the text features in the multimodal space, outputting the spatial position coordinates of the keywords and the spatial position offset coordinates of the key values. The spatial position offset coordinates include the offset of the key value relative to the spatial position coordinates. Based on the spatial coordinates of multiple keywords and the spatial offset coordinates of the key values, extract the text from the card image; The text in the card image is identified using a text recognition model to obtain the text information corresponding to the keywords and key values.

[0007] Thirdly, embodiments of this application provide a training apparatus for a card recognition model, comprising: The first input module is used to acquire multiple sample card images for extracting structured information, input the sample card images into the card recognition model, and extract multiple keywords and the location data of the key values ​​corresponding to the keywords from the sample card images through the key information in the card recognition model. The second input module is used to input the location data of the keywords and key values ​​into the text encoder to obtain text features, and to input the sample card image into the image encoder to obtain image features; The mapping module is used to map the text features and the image features to a multimodal space, predict the position of the keywords and key values ​​of the text features in the multimodal space, and output the spatial position coordinates of the keywords and the spatial position offset coordinates of the key values. The spatial position offset coordinates include the offset of the key value relative to the spatial position coordinates. An update module is used to update the model parameters in the card recognition model based on the spatial location coordinates of multiple keywords and the spatial location offset coordinates of the key values, as well as the location coordinates of the keywords and the location coordinates of the key values ​​in the sample card image.

[0008] Fourthly, embodiments of this application provide a device for recognizing structured information of cards and certificates, comprising: acquiring multiple sample card and certificate images from which structured information is to be extracted; inputting the sample card and certificate images into a card and certificate recognition model; recognizing multiple key information of the sample card and certificate images through the card and certificate recognition model; and extracting the location data of all keywords and the key values ​​corresponding to the keywords in the sample card and certificate images based on the multiple key information, wherein the key information is used to describe the sample card and certificate images; The first extraction module is used to extract text features of the location data of multiple keywords and key values, as well as image features of the card image, through the card recognition model; The prediction module is used to map the text features and the image features into a multimodal space through the card recognition model, and to predict the position of the keywords and key values ​​of the text features in the multimodal space, and output the spatial position coordinates of the keywords and the spatial position offset coordinates of the key values, wherein the spatial position offset coordinates include the offset of the key value relative to the spatial position coordinates; The second extraction module is used to extract text from the card image based on the spatial coordinates of the multiple keywords and the spatial offset coordinates of the key values; The recognition module is used to recognize the text in the card image using a text recognition model, and obtain the text information corresponding to the keywords and key values.

[0009] Fifthly, embodiments of this application provide an electronic device, including a processor and a memory electrically connected to the processor, the memory storing a computer program, and the processor being used to call and execute the computer program from the memory to implement the above-mentioned training method for a card recognition model, or to implement the above-mentioned method for recognizing structured information of a card.

[0010] Sixthly, embodiments of this application provide a computer-readable storage medium for storing a computer program that can be executed by a processor to implement the above-described training method for a card recognition model or the above-described method for recognizing structured information of a card.

[0011] Seventhly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the above-mentioned training method for a card recognition model, or to implement the above-mentioned method for recognizing structured information of a card.

[0012] Eighthly, embodiments of this application provide a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the above-described training method for a card recognition model, or implements the above-described method for recognizing structured information of a card.

[0013] The technical solution of this application embodiment acquires multiple sample card images from which structured information is to be extracted. These sample card images are then input into a card recognition model, which identifies multiple key pieces of information within the images. Based on this key information, the location data of all keywords and their corresponding key values ​​within the sample card images are extracted. This key information describes the sample card images. It is evident that without pre-setting the number or range of keywords, multiple key pieces of information describing the sample card images can be acquired, and the keywords within the key information and their corresponding key values ​​can be inferred and identified. Therefore, it is unnecessary to set multiple different types of card templates; even with different types of sample card images, this card recognition model can extract the key information describing the sample card images, identify the keywords within them, and the location data of their corresponding key values.

[0014] The location data of keywords and key values ​​are input into a text encoder to obtain text features, and sample card images are input into an image encoder to obtain image features. These text and image features are mapped to a multimodal space. The location of keywords and key values ​​in the text features within the multimodal space is predicted, outputting the spatial coordinates of the keywords and the spatial offset coordinates of the key values. The spatial offset coordinates include the offset of the key value relative to its spatial coordinates. Based on the spatial coordinates and spatial offset coordinates of multiple keywords and key values, as well as their location coordinates in the sample card images, the model parameters in the card recognition model are updated. It is evident that the card recognition model, through the text features of the location data of keywords and key values ​​in multiple sample card images, and the image features of the sample card images, predicts the location of keywords and key values, continuously updating the model parameters. This ensures that the latest sample card image input to the card recognition model outputs spatial coordinates of keywords and spatial offset coordinates of key values ​​that are consistent with or close to their actual locations in the sample card image. Therefore, the trained card recognition model combines the text features of keyword and key value location data with the image features of the card image, making full use of the visual features of the card image to predict the location of keywords and key values. It does not need to rely excessively on text detection and recognition technology. Subsequently, it can accurately recognize text based on the text corresponding to the location information output by the trained card recognition model, preventing errors when recognizing the entire sample card image through text detection and recognition technology. Furthermore, the trained card recognition model can adapt to various types of cards. Even if a new card is encountered that has not been used in the training phase, the location coordinates of keywords and key values ​​can be accurately extracted, improving the efficiency of card information recognition. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in one or more embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic flowchart illustrating a training method for a card recognition model according to an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a card recognition model according to an embodiment of this application; Figure 3 This is a schematic flowchart of a method for identifying structured information of a card according to an embodiment of this application; Figure 4 This is a schematic flowchart illustrating a training method for a card recognition model according to another embodiment of this application; Figure 5 This is a schematic diagram of the structure of a training device for a card recognition model according to an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a card structured information recognition device according to an embodiment of this application; Figure 7 This is a schematic diagram of the hardware structure of a training device for a card recognition model according to an embodiment of this application. Detailed Implementation

[0017] This application provides a method for training a card recognition model and a method for recognizing structured card information to solve the problem of low accuracy in recognizing card information.

[0018] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0019] The training method for the card recognition model provided in this application embodiment can be executed by an electronic device or by software installed in an electronic device. Specifically, the electronic device can be a terminal device or a server device. The terminal device can include smartphones, laptops, smart wearable devices, vehicle terminals, etc., and the server device can include an independent physical server, a server cluster composed of multiple servers, or a cloud server capable of cloud computing.

[0020] The following description, in conjunction with the accompanying drawings, details a training method for a card recognition model provided in this application through specific embodiments and application scenarios.

[0021] Figure 1 The diagram illustrates a schematic flowchart of a training method for a card recognition model according to an embodiment of the present invention. The method includes the following steps: S102, acquire multiple sample card images from which structured information is to be extracted, input the sample card images into the card recognition model, and use the card recognition model to identify multiple key information of the sample card images; based on the multiple key information, extract all keywords and the location data of the key values ​​corresponding to the keywords in the sample card images.

[0022] Key information is used to describe the sample card image.

[0023] The card and document images include various types of documents such as ID cards, driver's licenses, business licenses, and VAT invoices. Multiple sample card and document images are used to train the card and document recognition model.

[0024] The card recognition model is used to identify the location information of keywords and key values ​​in sample card images, so as to extract the corresponding keywords and key values ​​to identify their text information.

[0025] In the keyword and its corresponding key value positional data, keywords are represented as keys, and key values ​​as values. The relationship between keywords and key values ​​is key-value, for example: <Name, Zhang San>, where the keyword is "Name" and the key value is "Zhang San". Key value positional data includes the keyword's location relative to the keyword, such as whether the key value is to the right or below the keyword.

[0026] Key information in the card recognition model consists of statements describing sample card images and defining the correspondence between keywords and key values. This allows the location data of the key values ​​corresponding to keywords to be determined based on the keywords extracted from the key information. Since the amount of text and other information in the sample card images is uncertain, the format and quantity of key information in the card recognition model do not need to be pre-defined. As an example, key information can be represented as: {key_name}, itsvalue on the {position}, where key_name can be replaced with specific text, name is a name, such as a surname, and position represents the orientation information of the key value relative to the keyword key_name. Position can be one of the following: right, down, right top, right down, etc.

[0027] Based on the key information identified by the card recognition model, the location data of keywords and key values ​​relative to keywords are identified from the key information. This yields the location data of all keywords and corresponding key values ​​for each sample card image, resulting in multiple descriptive statements describing the location data of the keywords and key values. Without needing to pre-set a text set of keywords, this key information can directly extract the location data of multiple keywords and key values ​​from sample card images. This extraction method does not require defining the structure of the sample card images to be extracted and is adaptable to extracting keywords from various types of card images.

[0028] S104: Input the location data of keywords and key values ​​into the text encoder to obtain text features, and input the sample card image into the image encoder to obtain image features.

[0029] The descriptive statement of the location data of keywords and key values ​​extracted in S102 is input into the text encoder. The text encoder can use an open-source large language model (LLM) that supports multiple languages ​​to convert the location data of the input keywords and key values ​​into a high-dimensional vector representation. Typically, the self-attention mechanism in the Transformer structure of the neural network architecture based on the self-attention mechanism is used to encode the text, and finally the text vector corresponding to the location data of keywords and key values ​​is obtained, which is the text feature.

[0030] The sample card image is input into the image encoder. The image encoder can be a Transformer-based visual model, such as the Swing Transformer, which extracts image features in a hierarchical, windowed manner. Alternatively, it can be a model that applies the Transformer architecture to computer vision tasks (Vision Transformer, VIT). There are no specific restrictions on the visual model used by the image encoder. The image encoder extracts the feature vectors of the sample card image, thus obtaining the image features of the sample card image.

[0031] Before inputting the location data of keywords and key values ​​into the text encoder, the text sentence information of the location data of keywords and key values ​​is processed by word segmentation.

[0032] S106 maps text features and image features to a multimodal space, predicts the location of keywords and key values ​​of text features in the multimodal space, and outputs the spatial location coordinates of keywords and the spatial location offset coordinates of key values.

[0033] Spatial location offset coordinates include: the offset of the key value relative to the spatial location coordinates.

[0034] Multimodal space refers to a space that can integrate information from multiple modalities, also known as a "multimodal space scene".

[0035] The card recognition model can use a multimodal pre-trained model (Contrastive Language-Image Pre-training, CLIP) as its base model. It uses text features as query vectors, keywords from image features as key vectors, and key values ​​as value vectors. By leveraging text features, it guides and extracts the visual representation of the text, forming a dynamic and dedicated feature space—the multimodal space. This multimodal space is then used to predict the positions of keywords and key values, essentially finding the matching key and value vectors within the image features using the query vector.

[0036] After mapping text and image features to the multimodal space, the spatial coordinates of the predicted keywords and the spatial offset coordinates of the key values ​​are obtained. For example, the spatial coordinates of the predicted keywords are represented as x1, y1, x2, y2, and the spatial offset coordinates of the key values ​​are represented as off_x1, off_y1, off_x2, off_y2. Then, the sample card image includes the spatial coordinates of multiple predicted keywords and the corresponding spatial offset coordinates of the key values. Here, x1, y1, x2, y2 represent the x and y coordinates of the top left and bottom right corners, respectively, and off_x1, off_y1, off_x2, off_y2 represent the spatial offset coordinates of the key values ​​corresponding to the keywords.

[0037] S108, based on the spatial coordinates of multiple keywords and the spatial offset coordinates of key values, as well as the position coordinates of keywords and key values ​​in the sample card image, update the model parameters in the card recognition model.

[0038] The spatial coordinates of multiple keywords and the spatial offset coordinates of key values ​​output by the S106 card recognition model are compared with the position coordinates of each keyword and key value in the sample card images. If the coordinates match or are highly similar to the actual position coordinates, the card recognition model is considered trained. If the similarity is low, the accuracy of the output spatial coordinates and spatial offset coordinates is improved by adjusting the model parameters. By inputting multiple sample card images into the card recognition model, the model is trained, resulting in a trained card recognition model capable of actively recognizing the spatial coordinates of keywords and the spatial offset coordinates of key values ​​in new card images.

[0039] The technical solution of this application embodiment acquires multiple sample card images from which structured information is to be extracted. These sample card images are then input into a card recognition model, which identifies multiple key pieces of information within the images. Based on this key information, the location data of all keywords and their corresponding key values ​​within the sample card images are extracted. This key information describes the sample card images. It is evident that without pre-setting the number or range of keywords, multiple key pieces of information describing the sample card images can be acquired, and the keywords within the key information and their corresponding key values ​​can be inferred and identified. Therefore, it is unnecessary to set multiple different types of card templates; even with different types of sample card images, this card recognition model can extract the key information describing the sample card images, identify the keywords within them, and the location data of their corresponding key values.

[0040] The location data of keywords and key values ​​are input into a text encoder to obtain text features, and sample card images are input into an image encoder to obtain image features. These text and image features are mapped to a multimodal space. The location of keywords and key values ​​in the text features within the multimodal space is predicted, outputting the spatial coordinates of the keywords and the spatial offset coordinates of the key values. The spatial offset coordinates include the offset of the key value relative to its spatial coordinates. Based on the spatial coordinates and spatial offset coordinates of multiple keywords and key values, as well as their location coordinates in the sample card images, the model parameters in the card recognition model are updated. It is evident that the card recognition model, through the text features of the location data of keywords and key values ​​in multiple sample card images, and the image features of the sample card images, predicts the location of keywords and key values, continuously updating the model parameters. This ensures that the latest sample card image input to the card recognition model outputs spatial coordinates of keywords and spatial offset coordinates of key values ​​that are consistent with or close to their actual locations in the sample card image. Therefore, the trained card recognition model combines the text features of keyword and key value location data with the image features of the card image, making full use of the visual features of the card image to predict the location of keywords and key values. It does not need to rely excessively on text detection and recognition technology. Subsequently, it can accurately recognize text based on the text corresponding to the location information output by the trained card recognition model, preventing errors when recognizing the entire sample card image through text detection and recognition technology. Furthermore, the trained card recognition model can adapt to various types of cards. Even if a new card is encountered that has not been used in the training phase, the location coordinates of keywords and key values ​​can be accurately extracted, improving the efficiency of card information recognition.

[0041] In one embodiment, multiple key pieces of information in a sample card image are identified using a card recognition model; based on these key pieces of information, all keywords and the location data of the key values ​​corresponding to the keywords in the sample card image are extracted (i.e., S102), which can be achieved by performing the following steps A1-A3: Step A1: The text information of the sample card image is identified by the card recognition model, and multiple key information for describing the sample card image is generated.

[0042] Text information refers to the text describing the sample card image. Based on the text information, multiple key information describing the sample card image is generated.

[0043] Step A2: Based on the key information, extract the data related to the keywords from the key information and determine them as the keywords of the sample card image.

[0044] Specifically, keyword-related data refers to extracting descriptive data identical to the keywords from the key information used to describe the sample card images. It also includes extracting descriptive data semantically similar to the keywords, such as synonyms, and randomly selecting one synonym to add to the key information. Synonyms can be obtained using intelligent tools (ChatGenerative Pre-trained Transformer, ChatGPT) to acquire synonyms for each keyword. For example, "[corporate legal person]" and "legal person" can be used as the same keyword, only the textual descriptive data differs.

[0045] Step A3: Based on the positional correspondence between keywords and key values ​​in the keywords and key information, extract the positional data of keywords and key values ​​in the sample card image. The positional data of key values ​​includes: the orientation information of key values ​​relative to keywords.

[0046] Key information also includes the positional correspondence between keywords and key values. The positional correspondence between keywords and key values ​​is defined according to the key-value positional relationship in conventional cards and certificates. The positional data of the key value, that is, the orientation information of the key value relative to the keyword, is generally to the right, upper right, or lower, etc., but in principle, it can also be extended to other positions, such as upper left, etc.

[0047] Specifically, based on the text information of the sample card image identified by the card recognition model, the generated key information can be represented as {key_name}, also called {synonyms of key_name}, and its value on the {position}. The final key information contains two descriptive statements for the keywords. Based on the content of the sample card image, n statements are generated according to these two descriptive statements. For example: {key1}, its value is on the {down}; {key2}, also called {synonyms of key2}, its value is on the {right}; {key n}, its value is on the {right / down}, etc. Here, key1 and key2 represent different keywords, and right or down indicates the position of the corresponding key value to the right or below the keyword.

[0048] Among the multiple key pieces of information in the sample certificate image, a text represents a description of the location data of a keyword and its corresponding key value. Therefore, there is no need to predefine the scope of the keyword or other information. Based on the key information extracted from the sample certificate image, the location data of the keyword and key value can be identified. Even if the structure, size and layout of the same sample certificate image are inconsistent, it will not affect the extraction of key information from the sample certificate image.

[0049] Based on key information, extract the location data of keywords and key values ​​from the card image. The location data of key values ​​includes: the location information of key values ​​relative to keywords, such as the specific location of key values ​​in relation to keywords.

[0050] It should be noted that the number of keywords in each sample card image is not fixed. The names and number of keywords can be automatically obtained from the text information of the sample card images. It is not necessary to require that the number of keywords in all sample card images be the same.

[0051] In this embodiment, the text information of sample card images is identified through a card recognition model, generating multiple key information items to describe the sample images. Keywords are determined based on this key information. The positional correspondence between key values ​​and keywords in the key information is used to extract the positional data of key values ​​in the sample card images. The acquired keywords also include synonyms; that is, synonyms are acquired along with the keywords in the sample card images, without needing to define the structure of the sample card images to be extracted. Since the key information can identify the positional data of keywords and key values, and the range of keywords is not predetermined, when keywords change, the changed keywords can be directly extracted from the key information. This eliminates the need to pre-obtain information from fixed-format sample card images using an Optical Character Recognition (OCR) engine, effectively avoiding errors caused by the OCR engine. Therefore, even using only some types of cards as training data, the trained card recognition model can adapt to other types of cards and recognize keywords from other types of cards, thereby improving the generalization ability of the card recognition model.

[0052] In one embodiment, the location data of keywords and key values ​​are input into a text encoder to obtain text features, and the sample card image is input into an image encoder to obtain image features (i.e., S104). The following steps B1-B3 can be performed: Step B1: Input the position data of keywords and key values ​​into the word segmenter, and perform word segmentation processing to obtain the position data of keywords and key values ​​after word segmentation.

[0053] In this context, each keyword after word segmentation is considered a semantic unit.

[0054] A tokenizer is used to break down text data into smaller semantic units called tokens. The tokenizer segments text into various semantic units (tokens) by word, sub-word, and character, and maps these tokens to corresponding semantic unit identification identifiers (token IDs). This converts the text descriptions corresponding to the location data of keywords and key values ​​into a unified format that can be processed by the card recognition model.

[0055] The location data of multiple keywords and their corresponding key values ​​extracted from the sample card images will be used as text data and input into the word segmenter. The word segmentation process will be performed on the location data of the keywords and key values. In order to ensure the integrity of the keywords, when the keywords are segmented, it is ensured that one keyword corresponds to one semantic unit, i.e., one token. The text descriptions related to the location data of the remaining key values ​​will be processed by conventional word segmentation.

[0056] As an example, the keyword in the sample card image is the legal representative's name. If this keyword is segmented, the result would be [legal representative, name], corresponding to two tokens, which could easily lead to ambiguity. Therefore, instead of segmenting the keyword in the sample card image from which key information is extracted, the keyword is treated as a semantic unit. Only the positional data of the keyword and its key value are processed separately, and the text corresponding to the positional data of the key value is segmented. This yields the segmented keyword and key value positional data.

[0057] Step B2: Input the positional data of keywords and key values ​​after word segmentation into the text encoder to obtain the text features of the positional data of keywords and key values.

[0058] The positional data of keywords and key values ​​after word segmentation in step B1 are input into the text encoder to convert the input positional data of keywords and key values ​​into a high-dimensional vector representation, thereby obtaining the text features of the positional data of keywords and key values.

[0059] Step B3: Input the sample card image into the image encoder to obtain the image features of the sample card image.

[0060] The sample card image is input into the image encoder, which is a model related to visual expression and can extract the image features of the sample card image.

[0061] The text encoder and image encoder can be part of the card recognition model or an encoder of an independent card recognition model, without specific limitations.

[0062] In this embodiment, a word segmenter is used to segment the positional data of keywords and key values, treating keywords as a semantic unit to distinguish them from the positional data of key values. This avoids ambiguity that might arise from segmenting keywords during the word segmentation process. Simultaneously, a text encoder extracts features from the positional data of keywords and key values ​​to obtain text features, while an image encoder extracts image features from the sample card image. This reduces the complexity of data processing and facilitates subsequent comparison between text and image representations.

[0063] In one embodiment, text features and image features are mapped to a multimodal space, and the location of keywords and key values ​​of text features in the multimodal space is predicted. The spatial location coordinates of the keywords and the spatial location offset coordinates of the key values ​​are output (i.e., S106). The following steps C1-C5 can be performed: Step C1 involves mapping the text features and image features to a multimodal space for alignment, obtaining aligned text features and image features, and predicting the spatial coordinates of the keywords.

[0064] The feature enhancement module in the card recognition model aligns the obtained text and image features, ensuring that both features describe the location data of a keyword or key value in the same position. For example, if the text feature is "name, its value on the right," the beginning of the text feature corresponds to the description of "name," and the end corresponds to the description of "right." However, in the original sample card image, "name" appears in the middle of the image, and its corresponding key value description also appears in the middle, but the keyword in the text feature is on the right. Clearly, the two features are not aligned, so alignment is necessary.

[0065] Specifically, text features and image features are mapped to a multimodal space, and each is processed through a self-attention mechanism to obtain attention features for both text and image features. Using text features as query vectors, key and value vectors are retrieved from image features. A cross-attention mechanism is then used to attach attention to the key and value vectors of the image features from the same sequence of text features, ensuring that the positional data of each keyword and key value in the text features are aligned with the corresponding positional data of the keywords and key values ​​in the image features.

[0066] Using aligned text features as query vectors and image features as key and value vectors, the visual representation of each text corresponding to the text features is extracted. In the mapped multimodal space, the location of keywords in each text feature is predicted to obtain the spatial location coordinates of the keywords. Specifically, in the multimodal space, the location information of each keyword is determined based on the x and y coordinates of the upper left corner and the x and y coordinates of the lower right corner, and the predicted spatial location coordinates of the keywords are represented as x1, y1, x2, y2.

[0067] Step C2: Determine the spatial offset coordinates of the key value based on the spatial coordinates of the keyword and the location data of the key value.

[0068] Based on the spatial coordinates of the keywords determined in step C1, the position information of the keywords in the multimodal space relative to the sample card image can be determined. Based on the correspondence between the position data of the keywords and key values, the spatial offset coordinates of the key values ​​are obtained, represented as off_x1, off_y1, off_x2, off_y2.

[0069] Among them, the maximum number of queries for text features is preset, and the number of queries can be applied automatically. Generally, it is equal to the number of text codes corresponding to the text features, which is the number of keywords that can be represented.

[0070] Step C3: Obtain the position coordinates of the keywords in the sample card image. Based on the position coordinates of the keywords in the sample card image, determine the aspect ratio of the annotation box corresponding to the keywords in the sample card image.

[0071] The annotation box refers to the text box generated by the coordinates of the keyword's position in the sample card image.

[0072] By obtaining the position coordinates of the keywords in the sample card image, the annotation box of the keywords in the sample card image is determined based on the position coordinates, and the aspect ratio of the annotation box is calculated.

[0073] Step C4: Based on the aspect ratio and key value location data, determine the weighting coefficients of spatial location coordinates and spatial location offset coordinates in the second loss function when predicting spatial location coordinates and spatial location offset coordinates.

[0074] The second loss function is generally L1 loss, which is used to represent the spatial location coordinates of the keywords and the location prediction loss of the spatial location offset coordinates of the key values.

[0075] Based on the spatial coordinates of the keywords and the positional data of the key values, the positional offset of the key values ​​relative to the keywords is determined. Based on this offset data, a second loss function is determined, corresponding to the spatial coordinates and spatial offset coordinates. This function calculates the difference between the spatial coordinates and spatial offset coordinates relative to the positional coordinates of the keywords and the positional data of the key values ​​relative to the keywords in the sample card image. Specifically, based on the positional data of the key values ​​in the text description, the direction of the key value's offset relative to the keywords is determined. Based on this offset direction information, the positional weight of the key value relative to the keywords is determined. Because the offset direction of the key values ​​relative to the keywords varies, the weight used for each coordinate in the second loss function is different when calculating the corresponding loss function. For example, when position=right, meaning the key value's positional data is to the right of the keyword, the weight coefficients of the coordinates x1, x2, off_x1, off_x2 calculated based on the top-left corner coordinates can be set to 'a', where the value of 'a' is equal to the aspect ratio of the ground truth bounding box in the sample card image. If the sample document image is a business license, and a key term in the business license is "legal person," then using the top-left corner coordinates, the bounding box for the legal person in the business license image is a 30*60 text box. The corresponding key value is to the right of the legal person, with a width of 60 and a height of 30, resulting in an aspect ratio of 2. In the L1 loss, the weight coefficients before x1, x2, off_x1, and off_x2 are 2, while the others are 1. This allows the document recognition model to focus more on the positional relationship between the key term and the key value, making the regression in the x-direction more accurate. Similarly, if the bounding box for the legal person in the business license image is calculated using the bottom-right corner coordinates, then the resulting weight coefficients are added before x2, y2, off_x2, and off_y2.

[0076] Step C5: Train the card recognition model based on the second loss function after determining the weight coefficients of the spatial location coordinates and spatial location offset coordinates.

[0077] Based on the second loss function obtained in step C4, the parameters of the card recognition model are optimized through backpropagation to make the second loss function converge. Finally, the trained card recognition model can output more accurate predicted spatial location coordinates of keywords and spatial location offset coordinates of key values. For example, the coordinates output by the card recognition model after the second loss function converges can be represented as (x1, y1, x2, y2, off_x1, off_y1, off_x2, off_y2).

[0078] In this embodiment, text features and image features are mapped to a multimodal space and aligned to ensure that keywords in the text features and image features are in the same position. Based on the text and image features in the same position in the multimodal space, the spatial coordinates and spatial offset coordinates of the keywords in the text features are predicted. Furthermore, based on the positional data of the key values ​​in the text features, the positional weights corresponding to the spatial coordinates and spatial offset coordinates in the second loss function are determined. According to the second loss function, the parameters of the card recognition model are adjusted to ensure that the trained card recognition model outputs more accurate predicted spatial coordinates of the keywords. This approach combines the visual representation of text and image features to predict the specific positional coordinates of keywords. Based on the aspect ratio of the keyword bounding boxes in the sample card images, the weight coefficients of each coordinate in the second loss function are set, resulting in more accurate spatial coordinates of the keywords and more precise predicted spatial offset coordinates of the key values.

[0079] In one embodiment, after outputting the spatial coordinates of the keywords and the spatial offset coordinates of the key values ​​(i.e., S106), the following steps D1-D6 can be performed: Step D1: Based on the spatial coordinates of each keyword and the spatial offset coordinates of the key value, take a screenshot of the sample card image to obtain a sub-image corresponding to each keyword and key value.

[0080] To make the trained card recognition model pay more attention to detailed text features, the original image used in CLIP contrastive learning was changed to a sub-image to further verify the output results of the card recognition model.

[0081] Specifically, based on the spatial coordinates of each keyword in the sample card image output by the card recognition model, and the corresponding spatial offset coordinates of the key value, a screenshot is taken of the sample card image to obtain a sub-image determined by the spatial coordinates and spatial offset coordinates output by the card recognition model. As an example, the spatial coordinates of a keyword in the sample card image, and the corresponding spatial offset coordinates of the key value, are (x1, y1, x2, y2, off_x1, off_y1, off_x2, off_y2). A screenshot is taken of the sample card image based on these coordinates to obtain the sub-image corresponding to those coordinates.

[0082] Step D2: Input the keywords and key values ​​corresponding to the keywords in the sample card image into the text encoder to obtain the text description features, and input the sub-images corresponding to the keywords into the image encoder to obtain the sub-image features.

[0083] The keywords and key values ​​corresponding to the keywords in the sample card images are input into the text encoder, including inputting the keywords and key values ​​in the real sample card images as natural language into the text encoder. The text encoder outputs text description features, which are the description features of the keywords and key values ​​corresponding to the real sample card images.

[0084] Specifically, the typical CLIP text description is usually an overview of the image, such as "a picture of {xx}", where xx is usually a noun. If this noun is a business license, the corresponding text description would be "a picture of business license". As you can see, this doesn't describe the details of the business license. Suppose the keyword to be extracted is the specific address from the business license, such as "<address: XX city XX district>". The improved CLIP model's text prompts would be as follows: "{address} in the picture" and "{XX city XX district} in the picture". This means that the text descriptions of the corresponding keywords and key values ​​from the actual sample card image are input into the text prompts.

[0085] The sub-image obtained in step D1 is input into the image encoder to obtain sub-image features, which are image features obtained based on the predicted spatial location coordinates and spatial location offset coordinates.

[0086] Step D3: Calculate the similarity between the text description features and the subgraph features to obtain the first similarity value.

[0087] Based on the text description features and subgraph features determined in step D2, calculate the similarity between the text description features and subgraph features. The similarity can be calculated using methods such as cosine similarity, without specific limitations, to obtain the first similarity value between the true text description features and subgraph features.

[0088] Step D4: Based on the subgraph features, generate the corresponding regression text features of the subgraph through autoregression, calculate the similarity between the regression text features and the text description features, and obtain the second similarity value.

[0089] Autoregressive methods refer to using the current sequence to predict future values. For example, subgraph features are input into an autoregressive model, and the model outputs the corresponding regressive text features of the subgraph.

[0090] The similarity between the predicted regressed text features derived from the autoregression of the subgraph features and the actual text description features can be calculated using methods such as cosine similarity to obtain a second similarity value.

[0091] Step D5: The mean of the first similarity value and the second similarity value is determined as the target similarity value between the sub-image feature and the text description feature.

[0092] The target similarity value represents the spatial location information of the sub-image and the similarity value of the sub-image's location information in the sample card image. The spatial location information of the sub-image includes: the spatial location coordinates and spatial location offset coordinates of the sub-image output by the card recognition model, and the location information includes: the location coordinates of the keywords and the location coordinates of the key values.

[0093] Based on the first similarity value obtained in step D4 and the second similarity value obtained in step D5, the mean of the first similarity value and the second similarity value is calculated, and the obtained mean is determined as the target similarity value between the sub-image feature and the text description feature.

[0094] Specifically, based on the spatial coordinates and offset coordinates of the sub-image output by the card recognition model, the spatial location information of the sub-image is determined, and a screenshot is taken to obtain the sub-image, thereby acquiring its features. Textual description features are determined based on the true location information of keywords and key values ​​in the sample card images. Therefore, the correctness of the spatial coordinates and offset coordinates of the sub-image output by the card recognition model can be judged based on the target similarity value between the finally determined sub-image features and textual description features.

[0095] Step D6: Based on the target similarity value, the spatial location coordinates and spatial location offset coordinates corresponding to the sub-images with a similarity value greater than the preset value are taken as difficult samples, and the weight of difficult samples is added to the first loss function to train the card recognition model.

[0096] Based on the target similarity value in step D5, when the target similarity value is greater than the preset similarity value, the spatial coordinates and spatial offset coordinates corresponding to the sub-image output by the card recognition model are reliable and belong to easy samples; when the target similarity value is less than or equal to the preset similarity value, the spatial coordinates and spatial offset coordinates corresponding to the sub-image output by the card recognition model are unreliable and belong to difficult samples.

[0097] Based on the target similarity values, easy and difficult samples are determined. The weights of the samples in the first loss function of the card recognition model are then determined. This first loss function is typically a contrastive loss function, used to characterize the loss of spatial location information of the sub-image and its position within the sample card image. Generally, a balance factor and a focus factor are added to the cross-entropy loss function, such as adjusting the imbalanced loss function (Focal Loss). The balance factor adjusts the weights of samples from different categories, while the focus factor adjusts the weights between difficult and easy samples. Difficult samples are typically those whose predicted results differ significantly from the true labels. These are prone to misclassification or underclassification by the card recognition model during training, thus requiring increased weight. Therefore, the weights of easy samples are relatively reduced, while the weights of difficult samples are relatively increased to give them more attention. By adjusting the parameters in the card recognition model to converge the first loss function, a trained card recognition model is obtained. This trained model ensures that the keywords and key values ​​in the sub-images corresponding to the output spatial location coordinates and spatial location offset coordinates are consistent with or highly similar to the keywords and key values ​​in the real sample card images.

[0098] As can be seen, the above steps are online sample mining techniques used in the training of the card recognition model. Specifically, by using auxiliary functions such as the calculated target similarity and the first loss function, the penalty for difficult samples in the sample is increased, making the information extracted by the trained card recognition model more efficient.

[0099] In this embodiment, the output results of the card recognition model are further verified during training. Based on the spatial coordinates and spatial offset coordinates output by the card recognition model, a screenshot of the sample card image is taken to obtain a sub-image. The target similarity value between the sub-image features and the actual text description features in the sample card image is determined. Based on the target similarity value, the weight of difficult samples in the first loss function of the card recognition model is adjusted. Training the card recognition model increases the weight of difficult samples with low target similarity in the first loss function, making the trained card recognition model pay more attention to difficult samples and better learn their features. This results in more accurate spatial coordinates of keywords and spatial offset coordinates of key values ​​output by the trained card recognition model, improving the accuracy of recognizing the structured information of the card.

[0100] In conclusion, Figure 2 This diagram illustrates the structure of a card recognition model according to an embodiment of the present invention, as shown below. Figure 2As shown, the card recognition model extracts the location data of keywords and key values ​​from sample card images using key information. This data is then input into a text encoder to obtain the first text feature. The sample card image is input into an image encoder to obtain the image feature. The feature enhancement module in the card recognition model performs feature enhancement processing on the first text feature and the image feature, such as mapping to a multimodal space for alignment, to obtain the second text feature corresponding to the first text feature in the multimodal space, and the third text feature corresponding to the image feature. A cross-modal decoder is then used to predict the spatial coordinates of the keywords and the spatial offset coordinates of the key values ​​based on the second and third text features. Finally, the card recognition model outputs the spatial coordinates of the keywords and the spatial offset coordinates of the key values. The card recognition model is then adjusted by comparing the loss function and the position regression loss function to obtain the trained card recognition model. The contrast loss is determined by a first loss function that combines the spatial location information and the actual location information corresponding to the keywords and key values. The spatial location information includes the spatial location coordinates and spatial location offset coordinates output by the card recognition model, and the actual location information includes the location coordinates in the sample card image. The location regression loss is determined by a second loss function that combines the predicted spatial location coordinates and spatial location offset coordinates.

[0101] Figure 3 The diagram illustrates a schematic flowchart of a method for recognizing structured information of a card according to an embodiment of the present invention. The method includes the following steps: S302, acquire the card image containing the structured information to be identified, input the card image into the trained card recognition model, and identify multiple key information of the card image through the card recognition model; based on the multiple key information, acquire the location data of all keywords and the key values ​​corresponding to the keywords in the card image, wherein the card recognition model is trained by the above-mentioned card recognition model training method.

[0102] Key information includes: data related to keywords, and the positional correspondence between keywords and key values. For example, {key1} has its value on the {down}; {key2}, also called {key2 synonym}, has its value on the {right}, etc.

[0103] Extracting the correspondence between keywords and key values ​​through key information refers to directly extracting the keywords described in natural language in the card image and the location data of the key values ​​corresponding to the keywords. The location data of the key values ​​includes the orientation information of the key values ​​relative to the keywords.

[0104] S304, extracts text features of location data of multiple keywords and key values, as well as image features of card images, through a card recognition model.

[0105] The location data of multiple keywords and key values ​​in the card image are extracted from the key information in S302. The text description of the keyword and key value location data is then segmented using a card recognition model to obtain the segmented keyword and key value location data. This segmented data is then input into the text encoder of the card recognition model to obtain text features. The card image is then input into the image encoder of the card recognition model to obtain image features.

[0106] S306 maps text features and image features into a multimodal space through a card recognition model, and predicts the location of keywords and key values ​​of text features in the multimodal space. It outputs the spatial location coordinates of keywords and the spatial location offset coordinates of key values. The spatial location offset coordinates include the offset of the key value relative to the spatial location coordinates.

[0107] The specific process of predicting the spatial coordinates of keywords and the spatial offset coordinates of key values ​​is consistent with the above description and will not be elaborated further here.

[0108] S308. Extract text from card images based on the spatial coordinates of multiple keywords and the spatial offset coordinates of key values.

[0109] Based on the spatial coordinates of the keywords and the spatial offset coordinates of the key values ​​output by the card recognition model in S306, a screenshot is taken of the corresponding position in the card image to obtain the text in the screenshot.

[0110] S310 uses a text recognition model to identify the text in the card image and obtain the text information corresponding to keywords and key values.

[0111] Text recognition models include OCR models, which are used to detect and recognize text information in images.

[0112] The text in multiple screenshots obtained by taking screenshots of the card image using the spatial location coordinates and spatial location offset coordinates output by the card recognition model is input into the text recognition model. The text recognition model recognizes the text in the screenshots and obtains the corresponding keyword text information and key value text information.

[0113] It should be noted that this application, in addition to being applied to information recognition of card and document images, can also be applied to various scenarios. For example, when performing feature extraction and analysis on collected images based on a cloud-edge collaborative data center risk identification model, the collected images can also be used to extract information by setting corresponding positional relationships, and to perform multimodal fusion without overly relying on OCR recognition technology. It can also implement multi-scene scheduling and linkage strategies for different camera shooting scenarios, calculate the number of cameras installed based on the prototype of the light source illumination area, and determine the angle and position of the installed cameras by using panoramic cameras to perform on-site scanning and modeling of the data center. In multiple scenarios, images or videos captured by cameras for analysis can be used to perform multimodal fusion based on the corresponding image, text, and video information, extract information from specific locations for recognition, and determine specific locations where cameras are advantageous for shooting.

[0114] The technical solution of this application embodiment directly extracts the location data of all keywords and corresponding key values ​​in the card and document recognition model based on the key information trained in the model. It then maps text features and image features to a multimodal space, predicts the locations of keywords and key values, and outputs the spatial coordinates of the keywords and the spatial offset coordinates of the key values. Based on these spatial coordinates and offset coordinates, the text in the card and document image is extracted. This extracted text is then input into a text recognition model for processing, enabling the identification of all keywords and key values ​​in all card and document images. It is evident that in the process of extracting information from card and document images, there is no need to use OCR recognition technology in text recognition models, and it is not affected by errors in OCR recognition technology. The trained card and document recognition model can directly obtain the keywords and key values ​​at the corresponding positions in the card and document images, and then use the text recognition model to recognize the corresponding text. This can improve the accuracy of card and document information recognition. Furthermore, by extracting key information from card and document images through the trained card and document recognition model, the location data of keywords and corresponding key values ​​in any type of card and document can be identified based on the key information. This eliminates the need to pre-set multiple sets of keywords or train corresponding models, and also improves the efficiency of card and document information recognition.

[0115] Figure 4 This is a schematic flowchart illustrating a training method for a card recognition model according to another embodiment of this application, such as... Figure 4 As shown, the method includes the following steps: S401, acquire multiple sample card images from which structured information is to be extracted, and input the sample card images into the card recognition model.

[0116] S402: Generate multiple key information to describe the sample card image through the text information of the card recognition model sample card image, extract keywords from the key information, and determine the location data of the key value corresponding to the keywords based on the keywords.

[0117] S403: Input the position data of keywords and key values ​​into the word segmenter, and perform word segmentation processing through the word segmenter to obtain the position data of keywords and key values ​​after word segmentation.

[0118] S404: Input the location data of keywords and key values ​​after word segmentation into the text encoder to obtain text features; input the sample card image into the image encoder to obtain the image features of the sample card image.

[0119] S405 maps text features and image features to a multimodal space for alignment, obtaining aligned text features and image features, and predicts the spatial coordinates of keywords and the spatial offset coordinates of key values.

[0120] S406. Based on the actual coordinates of the keywords in the sample card image, determine the aspect ratio of the annotation box corresponding to the keywords in the sample card image.

[0121] S407, based on aspect ratio and key value location data, determines the weight coefficients of spatial location coordinates and spatial location offset coordinates in the second loss function when predicting spatial location coordinates and spatial location offset coordinates, and trains the card recognition model according to the determined second loss function.

[0122] S408. Based on the spatial coordinates of each keyword and the spatial offset coordinates of the key value, the sample card image is cropped to obtain a sub-image corresponding to each keyword and key value.

[0123] S409: Input the keywords and key values ​​corresponding to the keywords in the sample card images into the text encoder to obtain text description features, and input the sub-images corresponding to the keywords into the image encoder to obtain sub-image features.

[0124] S410, calculate the similarity between the text description features and the subgraph features to obtain the first similarity value; based on the subgraph features, generate the corresponding regression text features of the subgraph through autoregression, calculate the similarity between the regression text features and the text description features to obtain the second similarity value.

[0125] S411, the mean of the first similarity value and the second similarity value is determined as the target similarity value between the sub-graph feature and the text description feature.

[0126] S412, based on the target similarity value, the spatial location coordinates and spatial location offset coordinates corresponding to the sub-images with a similarity value greater than the preset value are taken as difficult samples, and the weight of difficult samples is increased in the first loss function to further train the card recognition model, and finally the trained card recognition model is obtained.

[0127] The model parameters for training the card recognition model include relevant parameters used to predict spatial location coordinates, spatial location offset coordinates, and target similarity values.

[0128] The specific processes from S401 to S412 described above have been explained in detail in the above embodiments and will not be repeated here.

[0129] The technical solution of this application embodiment acquires multiple sample card images from which structured information is to be extracted. These sample card images are then input into a card recognition model, which identifies multiple key pieces of information within the images. Based on this key information, the location data of all keywords and their corresponding key values ​​within the sample card images are extracted. This key information describes the sample card images. It is evident that without pre-setting the number or range of keywords, multiple key pieces of information describing the sample card images can be acquired, and the keywords within the key information and their corresponding key values ​​can be inferred and identified. Therefore, it is unnecessary to set multiple different types of card templates; even with different types of sample card images, this card recognition model can extract the key information describing the sample card images, identify the keywords within them, and the location data of their corresponding key values.

[0130] The location data of keywords and key values ​​are input into a text encoder to obtain text features, and sample card images are input into an image encoder to obtain image features. These text and image features are mapped to a multimodal space. The location of keywords and key values ​​in the text features within the multimodal space is predicted, outputting the spatial coordinates of the keywords and the spatial offset coordinates of the key values. The spatial offset coordinates include the offset of the key value relative to its spatial coordinates. Based on the spatial coordinates and spatial offset coordinates of multiple keywords and key values, as well as their location coordinates in the sample card images, the model parameters in the card recognition model are updated. It is evident that the card recognition model, through the text features of the location data of keywords and key values ​​in multiple sample card images, and the image features of the sample card images, predicts the location of keywords and key values, continuously updating the model parameters. This ensures that the latest sample card image input to the card recognition model outputs spatial coordinates of keywords and spatial offset coordinates of key values ​​that are consistent with or close to their actual locations in the sample card image. Therefore, the trained card recognition model combines the text features of keyword and key value location data with the image features of the card image, making full use of the visual features of the card image to predict the location of keywords and key values. It does not need to rely excessively on text detection and recognition technology. Subsequently, it can accurately recognize text based on the text corresponding to the location information output by the trained card recognition model, preventing errors when recognizing the entire sample card image through text detection and recognition technology. Furthermore, the trained card recognition model can adapt to various types of cards. Even if a new card is encountered that has not been used in the training phase, the location coordinates of keywords and key values ​​can be accurately extracted, improving the efficiency of card information recognition.

[0131] In summary, specific embodiments of this subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing can be advantageous.

[0132] The above is a training method for a card recognition model provided by an embodiment of this application. Based on the same idea, an embodiment of this application also provides a training device for a card recognition model.

[0133] Figure 5 This is a schematic diagram of the structure of a training device for a card recognition model according to an embodiment of the present invention. Figure 5 As shown, the training device for the card recognition model includes: a first input module 51, a second input module 52, a mapping module 53, and an update module 54. The first input module 51 is used to acquire multiple sample card images for extracting structured information, input the sample card images into the card recognition model, and identify multiple key information of the sample card images through the card recognition model; based on the multiple key information, extract all keywords and the location data of the key values ​​corresponding to the keywords in the sample card images, and the key information is used to describe the sample card images. The second input module 52 is used to input the location data of keywords and key values ​​into the text encoder to obtain text features, and to input the sample card image into the image encoder to obtain image features; The mapping module 53 is used to map text features and image features to a multimodal space, predict the location of keywords and key values ​​of text features in the multimodal space, and output the spatial location coordinates of keywords and the spatial location offset coordinates of key values. The spatial location offset coordinates include the offset of the key value relative to the spatial location coordinates. The update module 54 is used to update the model parameters in the card recognition model based on the spatial location coordinates of multiple keywords and the spatial location offset coordinates of key values, as well as the location coordinates of keywords and key values ​​in the sample card image.

[0134] In one embodiment, the device further includes: The screenshot module is used to process the sample card image by taking a screenshot based on the spatial coordinates of each keyword and the spatial offset coordinates of the key value, and to obtain a sub-image corresponding to each keyword and key value. The input module is used to input the keywords and key values ​​corresponding to the keywords in the sample card images into the text encoder to obtain text description features, and to input the sub-images corresponding to the keywords into the image encoder to obtain sub-image features; The calculation module is used to calculate the similarity between text description features and subgraph features to obtain the first similarity value; The regression module is used to generate regression text features corresponding to the subgraphs through autoregression based on the subgraph features, calculate the similarity between the regression text features and the text description features, and obtain a second similarity value. The similarity module is used to determine the average of the first similarity value and the second similarity value as the target similarity value between the sub-image features and the text description features. The target similarity value represents the similarity between the spatial location information of the sub-image and the location information of the sub-image in the sample card image. The spatial location information of the sub-image includes: the spatial location coordinates and spatial location offset coordinates corresponding to the sub-image output by the card recognition model, and the location information includes: the location coordinates of the keywords and the location coordinates of the key values. The adjustment module is used to identify the spatial coordinates and spatial offset coordinates of sub-images with a similarity value greater than a preset value as difficult samples based on the target similarity value, and to add the weight of difficult samples to the first loss function in order to train the card recognition model.

[0135] In one embodiment, the first input module 51 is specifically used to identify the text information of the sample card image through the card recognition model, generate multiple key information for describing the sample card image; extract data related to keywords from the key information and determine them as keywords of the sample card image; and extract the positional data of keywords and key values ​​in the sample card image based on the positional correspondence between keywords and key values ​​in the key information, wherein the positional data of key values ​​includes: the orientation information of key values ​​relative to keywords.

[0136] In one embodiment, the second input module 52 is specifically used to input the location data of keywords and key values ​​into a word segmenter, perform word segmentation processing through the word segmenter to obtain the location data of keywords and key values ​​after word segmentation, wherein the keyword after word segmentation is a semantic unit; input the location data of keywords and key values ​​after word segmentation into a text encoder to obtain the text features of the location data of keywords and key values; and input the sample card image into an image encoder to obtain the image features of the sample card image.

[0137] In one embodiment, the mapping module 53 is specifically used to map text features and image features to a multimodal space for alignment processing, obtaining aligned text features and image features, and predicting the spatial coordinates of keywords; determining the spatial offset coordinates of key values ​​based on the spatial coordinates of keywords and the location data of key values; obtaining the location coordinates of keywords in sample card images, and determining the aspect ratio of the bounding boxes corresponding to the keywords in the sample card images based on the location coordinates of keywords in the sample card images; determining the weight coefficients of spatial coordinates and spatial offset coordinates in the second loss function when predicting spatial coordinates and spatial offset coordinates based on the aspect ratio and the location data of key values; and training the card recognition model based on the second loss function after determining the weight coefficients of spatial coordinates and spatial offset coordinates.

[0138] The technical solution of this application embodiment acquires multiple sample card images from which structured information is to be extracted. These sample card images are then input into a card recognition model, which identifies multiple key pieces of information within the images. Based on this key information, all keywords and their corresponding key values ​​are extracted from the sample card images. This key information describes the sample card images. It is evident that without pre-setting the number or range of keywords, multiple key pieces of information describing the sample card images can be acquired, and the keywords and their corresponding key values ​​can be inferred and identified. Therefore, it is unnecessary to set multiple different types of card templates; even with different types of sample card images, this card recognition model can extract the key information describing the sample card images, identify the keywords within them, and determine the location data of their corresponding key values.

[0139] The location data of keywords and key values ​​are input into a text encoder to obtain text features, and sample card images are input into an image encoder to obtain image features. These text and image features are mapped to a multimodal space. The location of keywords and key values ​​in the text features within the multimodal space is predicted, outputting the spatial coordinates of the keywords and the spatial offset coordinates of the key values. The spatial offset coordinates include the offset of the key value relative to its spatial coordinates. Based on the spatial coordinates and spatial offset coordinates of multiple keywords and key values, as well as their location coordinates in the sample card images, the model parameters in the card recognition model are updated. It is evident that the card recognition model, through the text features of the location data of keywords and key values ​​in multiple sample card images, and the image features of the sample card images, predicts the location of keywords and key values, continuously updating the model parameters. This ensures that the latest sample card image input to the card recognition model outputs spatial coordinates of keywords and spatial offset coordinates of key values ​​that are consistent with or close to their actual locations in the sample card image. Therefore, the trained card recognition model combines the text features of keyword and key value location data with the image features of the card image, making full use of the visual features of the card image to predict the location of keywords and key values. It does not need to rely excessively on text detection and recognition technology. Subsequently, it can accurately recognize text based on the text corresponding to the location information output by the trained card recognition model, preventing errors when recognizing the entire sample card image through text detection and recognition technology. Furthermore, the trained card recognition model can adapt to various types of cards. Even if a new card is encountered that has not been used in the training phase, the location coordinates of keywords and key values ​​can be accurately extracted, improving the efficiency of card information recognition.

[0140] Figure 6 This is a schematic diagram of a card structure information recognition device according to an embodiment of the present invention. Figure 6 As shown, the training device for the card recognition model includes: an acquisition module 61, a first extraction module 62, a prediction module 63, a second extraction module 64, and a recognition module 65. The acquisition module 61 is used to acquire a card image containing structured information to be identified, input the card image into a trained card recognition model, and identify multiple key information of the card image through the card recognition model; based on the multiple key information, acquire the location data of all keywords and the key values ​​corresponding to the keywords in the card image, wherein the card recognition model is trained by the card recognition model training method described above. The first extraction module 62 is used to extract text features of location data of multiple keywords and key values, as well as image features of card images, through the card recognition model; The prediction module 63 is used to map text features and image features into a multimodal space through the card recognition model, and to predict the location of keywords and key values ​​of text features in the multimodal space. It outputs the spatial location coordinates of keywords and the spatial location offset coordinates of key values. The spatial location offset coordinates include the offset of the key value relative to the spatial location coordinates. The second extraction module 64 is used to extract text from the card image based on the spatial coordinates of multiple keywords and the spatial offset coordinates of key values. The recognition module 65 is used to recognize the text in the card image using a text recognition model, and obtain the text information corresponding to keywords and key values.

[0141] The technical solution of this application embodiment directly extracts the location data of all keywords and corresponding key values ​​in the card and document recognition model based on the key information in the trained model. It then maps text features and image features to a multimodal space, predicts the positions of keywords and key values, and outputs the spatial coordinates of the keywords and the spatial offset coordinates of the key values. Based on the spatial coordinates and spatial offset coordinates, the text in the card and document image is extracted. The obtained text in the card and document image is input into a text recognition model for text recognition processing, which can recognize the text information of all keywords and key values ​​in all card and document images. It is evident that in the process of extracting information from card and document images, there is no need to use OCR recognition technology in the text recognition model, and it is not affected by OCR recognition errors. The trained card and document recognition model can directly obtain the keywords and key values ​​at corresponding positions in the card and document image, and then use the text recognition model to recognize the corresponding text. This improves the accuracy of card and document information recognition and, through the key information of the trained model, can recognize the location data of keywords and corresponding key values ​​in any type of card and document without the need for pre-setting multiple sets of keywords or training corresponding models, thus improving the efficiency of card and document information recognition.

[0142] Those skilled in the art will understand that Figure 5 The training device for the card recognition model can be used to implement the training method for the card recognition model described above. The details should be similar to those described in the method section above. To avoid being too complicated, they will not be repeated here. Figure 6 The card structure information recognition device can be used to implement the card structure information recognition method described above. The detailed description should be similar to the method description above, and will not be repeated here to avoid being cumbersome.

[0143] Based on the same technical concept, embodiments of this application also provide an electronic device for performing the above-described method for recognizing structured card information. Figure 7 This is a schematic diagram of the structure of an electronic device to implement various embodiments of this application. The electronic device can vary significantly due to differences in configuration or performance, and may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740. The processor 710, communications interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call a computer program stored in the memory 730 and executable on the processor 710 to perform the following steps: Obtain a card image containing structured information to be identified, input the card image into a trained card recognition model, and use the card recognition model to identify multiple key pieces of information in the card image; based on the multiple key pieces of information, obtain the location data of all keywords and the key values ​​corresponding to the keywords in the card image, wherein the card recognition model is trained by the card recognition model training method described above. Text features of location data for multiple keywords and key values, as well as image features of card images, are extracted using a card recognition model. The card recognition model maps text features and image features into a multimodal space, and predicts the location of keywords and key values ​​of text features in the multimodal space. It outputs the spatial location coordinates of keywords and the spatial location offset coordinates of key values. The spatial location offset coordinates include the offset of the key value relative to the spatial location coordinates. Extract text from the card image based on the spatial coordinates of multiple keywords and the spatial offset coordinates of key values; By using an optical character recognition (OCR) model to identify the text in a card image, the text information corresponding to keywords and key values ​​can be obtained.

[0144] The technical solution of this application embodiment directly extracts the location data of all keywords and corresponding key values ​​in the card and document recognition model based on the key information in the trained model. It then maps text features and image features to a multimodal space, predicts the positions of keywords and key values, and outputs the spatial coordinates of the keywords and the spatial offset coordinates of the key values. Based on the spatial coordinates and spatial offset coordinates, the text in the card and document image is extracted. The obtained text in the card and document image is input into a text recognition model for text recognition processing, which can recognize the text information of all keywords and key values ​​in all card and document images. It is evident that in the process of extracting information from card and document images, there is no need to use OCR recognition technology in the text recognition model, and it is not affected by OCR recognition errors. The trained card and document recognition model can directly obtain the keywords and key values ​​at corresponding positions in the card and document image, and then use the text recognition model to recognize the corresponding text. This improves the accuracy of card and document information recognition and, through the key information of the trained model, can recognize the location data of keywords and corresponding key values ​​in any type of card and document without the need for pre-setting multiple sets of keywords or training corresponding models, thus improving the efficiency of card and document information recognition.

[0145] The specific execution steps can be found in the various steps of the above-described method embodiment for recognizing structured information of cards and training card recognition models, and can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0146] It should be noted that the electronic devices in the embodiments of this application include: servers, terminals, or other devices besides terminals.

[0147] The above electronic device structure does not constitute a limitation on the electronic device. An electronic device may include more or fewer components than illustrated, or combine certain components, or arrange them differently. For example, an input unit may include a Graphics Processing Unit (GPU) and a microphone, and a display unit may use a liquid crystal display (LCD), organic light-emitting diode (OLED), or other similar display panels. User input units include at least one of a touch panel and other input devices. A touch panel is also called a touchscreen. Other input devices may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be elaborated further here.

[0148] Memory can be used to store software programs and various data. Memory can primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area can store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, memory can include volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (Synchlink DRAM, SLDRAM), and direct memory bus RAM (DRRAM).

[0149] The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly handles operations related to the operating system, user interface, and applications, while the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into the processor.

[0150] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described card recognition model training method embodiment, or implement the various processes of the above-described card structured information recognition method embodiment, and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0151] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0152] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described card recognition model training method embodiment, or to implement the various processes of the above-described card structured information recognition method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0153] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0154] This application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the processor is used to run the program or instructions to implement the various processes of the above-mentioned product recommended method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0155] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0156] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0157] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A training method for a card recognition model, characterized in that, The method includes: Multiple sample card images for which structured information is to be extracted are obtained. The sample card images are input into a card recognition model, and multiple key information of the sample card images are identified by the card recognition model. Based on the multiple key information, the location data of all keywords and the key values ​​corresponding to the keywords in the sample card images are extracted. The key information is used to describe the sample card images. The location data of the keywords and key values ​​are input into the text encoder to obtain text features, and the sample card image is input into the image encoder to obtain image features. The text features and the image features are mapped to a multimodal space. The location of the keywords and key values ​​of the text features in the multimodal space is predicted, and the spatial location coordinates of the keywords and the spatial location offset coordinates of the key values ​​are output. The spatial location offset coordinates include the offset of the key value relative to the spatial location coordinates. Based on the spatial coordinates of the keywords and the spatial offset coordinates of the key values, as well as the position coordinates of the keywords and the position coordinates of the key values ​​in the sample card image, the model parameters in the card recognition model are updated.

2. The method according to claim 1, characterized in that, After outputting the spatial coordinates of the keyword and the spatial offset coordinates of the key value, the method further includes: Based on the spatial coordinates of each keyword and the spatial offset coordinates of the key value, the sample card image is cropped to obtain a sub-image corresponding to each keyword and key value; The keyword and key value corresponding to the keyword in the sample card image are input into the text encoder to obtain text description features, and the sub-image corresponding to the keyword is input into the image encoder to obtain sub-image features; Calculate the similarity between the text description features and the sub-graph features to obtain a first similarity value; Based on the subgraph features, regression text features corresponding to the subgraph are generated through autoregression, and the similarity between the regression text features and the text description features is calculated to obtain a second similarity value. The mean of the first similarity value and the second similarity value is determined as the target similarity value between the sub-image feature and the text description feature; wherein, the target similarity value represents the similarity value between the spatial location information of the sub-image and the location information of the sub-image in the sample card image; the spatial location information of the sub-image includes: the spatial location coordinates and the spatial location offset coordinates corresponding to the sub-image output by the card recognition model, and the location information includes: the location coordinates of the keyword and the location coordinates of the key value; Based on the target similarity value, the spatial location coordinates and spatial location offset coordinates corresponding to the sub-image with a similarity value greater than the preset similarity value are taken as difficult samples, and the weight of the difficult samples is added to the first loss function to train the card recognition model.

3. The method according to claim 1, characterized in that, The process involves identifying multiple key pieces of information from the sample card image using the card recognition model; and based on this key information, extracting the location data of all keywords and their corresponding key values ​​from the sample card image, including: The card recognition model identifies the text information of the sample card image and generates multiple key information items to describe the sample card image. Based on the key information, extract the data related to the keywords from the key information and determine them as the keywords of the sample card image; Based on the positional correspondence between the keywords and key values ​​in the keywords and key information, the positional data of the keywords and key values ​​in the sample card image are extracted. The positional data of the key values ​​includes the orientation information of the key values ​​relative to the keywords.

4. The method according to claim 1, characterized in that, The process of inputting the location data of the keywords and key values ​​into a text encoder to obtain text features, and inputting the sample card image into an image encoder to obtain image features, includes: The location data of the keywords and key values ​​are input into the word segmenter, and the word segmenter performs word segmentation processing to obtain the location data of the keywords and key values ​​after word segmentation processing, wherein the keyword after word segmentation processing is a semantic unit; The location data of the keywords and key values ​​after word segmentation are input into the text encoder to obtain the text features of the location data of the keywords and key values; The sample card image is input into the image encoder to obtain the image features of the sample card image.

5. The method according to claim 1, characterized in that, The step of mapping the text features and image features to a multimodal space, predicting the location of the keywords and key values ​​of the text features in the multimodal space, and outputting the spatial location coordinates of the keywords and the spatial location offset coordinates of the key values ​​includes: The text features and the image features are mapped to the multimodal space for alignment processing to obtain the aligned text features and the image features, and the spatial location coordinates of the keyword are predicted. Based on the spatial coordinates of the keyword and the location data of the key value, determine the spatial offset coordinates of the key value; Obtain the position coordinates of the keyword in the sample card image, and determine the aspect ratio of the annotation box corresponding to the keyword in the sample card image based on the position coordinates of the keyword in the sample card image; Based on the aspect ratio and the location data of the key values, determine the weight coefficients of the spatial location coordinates and the spatial location offset coordinates in the second loss function when predicting the spatial location coordinates and the spatial location offset coordinates; The card recognition model is trained based on the second loss function after determining the weight coefficients of the spatial location coordinates and the spatial location offset coordinates.

6. A method for recognizing structured information of cards and certificates, characterized in that, The method includes: A card image containing structured information to be identified is obtained; the card image is input into a trained card recognition model; multiple key pieces of information in the card image are identified by the card recognition model; based on the multiple key pieces of information, the location data of all keywords in the card image and the key values ​​corresponding to the keywords are obtained; wherein the card recognition model is trained by the training method of the card recognition model according to any one of claims 1-5. The card recognition model extracts text features of the location data of multiple keywords and key values, as well as image features of the card image; The card recognition model maps the text features and image features into a multimodal space, and predicts the positions of the keywords and key values ​​of the text features in the multimodal space, outputting the spatial position coordinates of the keywords and the spatial position offset coordinates of the key values. The spatial position offset coordinates include the offset of the key value relative to the spatial position coordinates. Based on the spatial coordinates of multiple keywords and the spatial offset coordinates of the key values, extract the text from the card image; The text in the card image is identified using a text recognition model to obtain the text information corresponding to the keywords and key values.

7. A device for recognizing structured information of cards and certificates, characterized in that, include: The acquisition module is used to acquire a card image containing structured information to be identified, input the card image into a trained card recognition model, identify multiple key information of the card image through the card recognition model, and acquire the location data of all keywords and the key values ​​corresponding to the keywords in the card image based on the multiple key information, wherein the card recognition model is trained by the training method of the card recognition model according to any one of claims 1-5; The first extraction module is used to extract text features of the location data of multiple keywords and key values, as well as image features of the card image, through the card recognition model; The prediction module is used to map the text features and the image features into a multimodal space through the card recognition model, and to predict the position of the keywords and key values ​​of the text features in the multimodal space, and output the spatial position coordinates of the keywords and the spatial position offset coordinates of the key values, wherein the spatial position offset coordinates include the offset of the key value relative to the spatial position coordinates; The second extraction module is used to extract text from the card image based on the spatial coordinates of the multiple keywords and the spatial offset coordinates of the key values; The recognition module is used to recognize the text in the card image using a text recognition model, and obtain the text information corresponding to the keywords and key values.

8. An electronic device, characterized in that, The device includes a processor and a memory electrically connected to the processor. The memory stores a computer program, and the processor is used to call and execute the computer program from the memory to implement a training method for a card recognition model as described in claims 1-5, or to implement a method for recognizing structured information of a card as described in claim 6.

9. A computer-readable storage medium, characterized in that, The storage medium is used to store a computer program that can be executed by a processor to implement a training method for a card recognition model as described in claims 1-5, or a method for recognizing structured information of a card as described in claim 6.

10. A computer program product, characterized in that, The system includes a computer program that, when executed by a processor, implements a training method for a card recognition model as described in claims 1-5, or a method for recognizing structured information of a card as described in claim 6.

Images