A port service confirmation letter identification method, system, device and medium

Abstract

The application relates to the technical field of file identification, in particular to a port business confirmation letter identification method, system, device and medium, which comprises the following steps: obtaining a business confirmation letter file; converting the business confirmation letter file into business confirmation letter image data; performing primary extraction on the business confirmation letter image data to obtain primary extraction information; analyzing the primary extraction information to determine whether the seal part content and the table part content are successfully identified; if both are successfully identified, taking the primary extraction information as a business confirmation letter identification result; if there is a part of the content that is not successfully identified, performing secondary processing on the part to obtain a business confirmation letter identification result; and encapsulating the business confirmation letter identification result into JSON structured data. Through the primary extraction and secondary processing cooperative mechanism, the dynamic identification result credibility evaluation and the structured data encapsulation technology, the application realizes the full-element accurate extraction of the guarantee letter key information, the automatic closed-loop analysis and the efficient interaction of the business system.

Description

Technical Field

[0001] This application relates to the field of document recognition technology, specifically to a method, system, equipment, and medium for recognizing port business confirmation letters. Background Technology

[0002] With the accelerated digital transformation of the port logistics industry, the need for automated identification of port business confirmation letters, as core credentials for port transportation operations, is becoming increasingly urgent. The port business confirmation letter process includes the following steps: First, business personnel need to understand the port and terminal's rate information; second, they need to create the port business confirmation letter according to business requirements; finally, the port business confirmation letter is sent to the port and terminal via email, explaining the business details and copying the billing center. In this process, the identification and processing of port business confirmation letters still rely on manual operation, resulting in a low level of digitalization, cumbersome operations, and low efficiency.

[0003] In existing technologies, to improve the recognition efficiency and automation of port business confirmation letters, a phased processing flow can be adopted: first, the text content of the port business confirmation letter is extracted using OCR technology; then, the seal area is located and text recognition is performed using image segmentation algorithms; finally, the result is output as structured data. This method, by processing text and seal information step by step, initially achieves automated parsing of the content of port business confirmation letters.

[0004] However, existing technologies still have the following drawbacks: a single recognition process cannot dynamically assess the completeness of key information extraction, resulting in missed recognition content directly entering the output stage, requiring manual completion; there is a lack of an automated compensation mechanism for unsuccessfully recognized parts, and the system cannot automatically correct errors when the stamp area is blurry or the table structure is abnormal; the output data format is loose and has not formed a standardized interface with the business system, increasing the cost of subsequent data processing. Summary of the Invention

[0005] To address the technical problems of existing port business confirmation letter identification methods, such as the lack of dynamic integrity judgment and secondary processing mechanisms, the inability to automatically repair missing content when the seal is blurred or the form is abnormal, and the loose output data format making it difficult to apply directly, this application provides a sea-rail intermodal transport business confirmation letter identification method, system, equipment, and medium. Through the collaborative mechanism of initial extraction and secondary processing, dynamic recognition result credibility assessment, and structured data encapsulation technology, it achieves accurate extraction of all key information elements of the guarantee letter, automated closed-loop parsing, and efficient interaction with the business system.

[0006] Firstly, this application provides a method for identifying port business confirmation letters, comprising the following steps:

[0007] S1. Obtain the business confirmation letter file, which is a single-page port business confirmation letter PDF file containing a title, tables, seals and other content;

[0008] S2. Convert the business confirmation letter file into business confirmation letter image data;

[0009] S3. Perform initial extraction on the image data of the business confirmation letter, including key information extraction, seal extraction, and seal content recognition, to obtain initial extracted information;

[0010] Among them, key information extraction involves inputting the business confirmation letter image data into a pre-trained business confirmation letter key information extraction model and outputting key information extraction results, including all text fragments in the business confirmation letter image data, the category label corresponding to each text fragment, and the key name-key value pair relationship label of the question-answer;

[0011] Seal extraction includes locating the seal area, extracting the circular text image and the linearly arranged text image within the seal area, and preprocessing the circular text image.

[0012] Seal content recognition includes recognizing the content of preprocessed circular text images and linearly arranged text images, and outputting the seal content recognition results;

[0013] S4. Determine whether the initial information extraction successfully identified the seal portion and the table portion;

[0014] If all are successfully identified, the initial extracted information will be used as the identification result of the business confirmation letter.

[0015] If there are parts of content that were not successfully identified, then the parts that were not successfully identified will be processed again to obtain the business confirmation letter identification result;

[0016] The secondary processing includes secondary processing of the table part and secondary processing of the seal part. The secondary processing of the table part is to generate the missing key name-key value pair relationship labels in the initial extracted information and supplement them into the key information extraction results. The secondary processing of the seal part is to match the seal area image with the seal template library and use the seal content information corresponding to the matched seal template image to replace the original seal content recognition result in the initial extracted information.

[0017] S5. Encapsulate the business confirmation letter recognition results into JSON structured data.

[0018] It should be further noted that step S2 performs a three-channel pixel conversion on the business confirmation letter file to generate business confirmation letter image data, which is in JPG, JPEG, or PNG format.

[0019] It should be further explained that step S3, which involves extracting key information using a pre-trained business confirmation letter key information extraction model, includes the following steps:

[0020] Extract all text fragments from the image data of the business confirmation letter;

[0021] The text fragments are categorized into categories including questions, answers, titles, and other content;

[0022] Perform key-value matching on questions and answers, where questions correspond to key names and answers correspond to key values;

[0023] Output the key information extraction results, including all text fragments in the business confirmation letter image data, the category label corresponding to each text fragment, and the key name-key value pair relationship label of the question-answer.

[0024] It should be further noted that in step S3, the seal extraction includes:

[0025] The pre-trained seal location detection model is used to locate the seal region and outputs the coordinates of the seal region bounding box and the seal region image. The seal location detection model is a deep learning model.

[0026] A pre-trained text region segmentation model is used to extract circular text images and linearly arranged text images within the stamp region image. The text region segmentation model is a deep learning model.

[0027] A polar coordinate transformation is performed on the circular text image to obtain a straightened circular text image.

[0028] It should be further noted that in step S3, a pre-trained seal content recognition model is used to recognize the seal content. This seal content recognition model is a deep learning model, including:

[0029] Input the linearly arranged text image and the straightened circular text image into the seal content recognition model, and output the seal content recognition result.

[0030] It should be further noted that the key information extraction model for business confirmation letters is built based on the VI-LayoutXLM algorithm, and the training steps include:

[0031] Obtain a sample business confirmation letter file and convert it into image data of the business confirmation letter file.

[0032] The text fragments in the sample image data of the business confirmation letter are manually classified, and key-value matching is performed on the questions and answers to obtain the actual key information of the sample image data of the business confirmation letter, including the category label corresponding to each text fragment and the key name-key value pair relationship label of the question-answer. Each group of sample image data of the business confirmation letter and its actual key information are used as a sample to construct a sample key information dataset of the business confirmation letter.

[0033] The business confirmation letter key information extraction model was trained in two stages using a sample key information dataset of business confirmation letters:

[0034] The first stage trains the text segment category recognition capability of the business confirmation letter key information extraction model based on the category label corresponding to each text segment, and uses the cross-entropy loss function to optimize the category label recognition result;

[0035] In the second stage, based on the fixed category labels corresponding to each text segment, the key-value matching ability of the business confirmation letter key information extraction model is trained based on the key-value pair relationship labels of question-answer. The similarity comparison loss function is used to optimize the key-value pair matching relationship.

[0036] It should be further noted that the seal position detection model is built based on the YOLO algorithm, and the training steps include:

[0037] Obtain a sample business confirmation letter file and convert it into image data of the business confirmation letter file.

[0038] In the sample image data of business confirmation letters, the coordinates of the bounding box of the seal area and the category label are manually annotated using the annotation tool. Each set of sample image data of business confirmation letters and its seal area bounding box coordinates and category label are used as a sample to construct a sample seal location dataset of business confirmation letters.

[0039] The seal location detection model was trained using a sample seal location dataset from business confirmation letters. The positioning accuracy of the seal region bounding box was optimized by using a multi-scale feature map prediction mechanism and the intersection-union loss function.

[0040] The text region segmentation model is built based on the Mask R-CNN algorithm, and the training steps include:

[0041] Obtain stamp sample images, and manually use annotation tools to annotate the circular text region segmentation mask and the straight text region segmentation mask in the stamp sample images. Use each set of stamp sample images and the corresponding circular text region segmentation mask and straight text region segmentation mask as a sample to construct a text region segmentation dataset.

[0042] A text region segmentation model was trained using a text region segmentation dataset. A pixel-level segmentation network was employed, and the text region segmentation results were optimized using a binary cross-entropy loss function.

[0043] It should be further noted that the seal content recognition model is built based on the PPOCRv4 algorithm, and the training steps include:

[0044] Obtain a sample image of a seal, extract a sample circular text image and a sample linear text image from the sample image, perform polar coordinate transformation on the sample circular text image to obtain a straightened sample circular text image;

[0045] Manually identify the text content of straightened sample circular text images and sample linear text images, and construct a stamp content dataset by taking each straightened sample circular text image and its text content as a sample and each linear text image and its text content as a sample.

[0046] A seal content recognition model was trained using a seal content dataset, and the seal content recognition results were optimized using a sequence transcription loss function.

[0047] It should be further noted that step S6 is also included: adding the business confirmation letter image data and the corresponding business confirmation letter recognition results to the training dataset, and using the training dataset to iteratively update the business confirmation letter key information extraction model, seal position detection model, text region segmentation model and seal content recognition model through incremental learning.

[0048] It should be further explained that the rules for determining whether the stamp and table portions have been successfully recognized in step S4 include:

[0049] Extract the company name from the seal content recognition results and the company name from the title text fragment in the key information extraction results, and then calculate the similarity between the two. If the similarity is lower than the preset threshold, it is determined that the seal content has not been successfully recognized.

[0050] The key-value matching relationship of all questions and answers in the key information extraction results is verified. If a key name does not match a key value, it is determined that the table part content has not been successfully identified.

[0051] It should be further noted that the secondary processing steps for the table portion in step S4 include:

[0052] S401. Set up a key name library, which stores key names and their corresponding key-value regular expression rules;

[0053] S402. When the content of the table portion is not successfully recognized, perform OCR recognition on the table portion, extract all text fragments of the table portion, and record the spatial coordinates of the key name of each unmatched key value in the image;

[0054] S403. Match the key name of each unmatched key value with the key name in the key name library. If a match is successful, extract the text fragments that match the regular expression corresponding to the key name from all text fragments as candidate key values.

[0055] S404. Record the spatial coordinates of the candidate key values ​​in the image. Based on the spatial coordinates, select the candidate key value that is spatially closest to the unmatched key value as the matching key value of that key name, form the key name-key value pair relationship label of that key name, and add it to the key information extraction results.

[0056] The secondary processing of the seal includes:

[0057] S411. Set up a seal template library. The seal template library stores the mapping relationship between company names and corresponding seal template image groups. Each seal template image group contains at least one seal template image corresponding to the company. Each seal template image is associated with pre-stored seal content information.

[0058] S412. Extract the company name from the title text fragment in the extraction results. Based on the extracted company name, query the seal template library and select the seal template image group corresponding to the company name as the candidate seal template image group.

[0059] S413. Perform ORB feature point matching between the seal region image and each seal template image in the candidate seal template image group, calculate the image matching similarity score, and take the seal template image with the highest image matching similarity score and higher than the preset threshold as the target seal template image.

[0060] S414. Use the seal content information associated with the target seal template image as the actual seal content recognition result, replacing the original seal content recognition result in the initial extracted information.

[0061] It should be further noted that in step S403, if there is no key name in the key name library that does not have a matching key value, the matching fails and the key value corresponding to the key name is defined as blank.

[0062] If a match is successful, and there is no text fragment in the entire text that matches the regular expression corresponding to the key name, then the key value corresponding to that key name is defined as blank.

[0063] Secondly, this application provides a port business confirmation letter identification system for implementing the aforementioned business confirmation letter identification method, including:

[0064] The preprocessing module is used to acquire the business confirmation letter file and convert it into business confirmation letter image data.

[0065] The initial extraction module is used to initially extract the image data of the business confirmation letter to obtain initial extraction information.

[0066] The analysis and judgment module is used to analyze the initially extracted information and determine whether the seal part and the table part have been successfully recognized.

[0067] The secondary processing module is used to perform secondary processing on the parts of the content that were not successfully recognized, so as to obtain the recognition result of the business confirmation letter.

[0068] The structured processing module is used to encapsulate the business confirmation letter recognition results into JSON structured data.

[0069] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described business confirmation letter identification method.

[0070] Fourthly, this application provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described business confirmation letter identification method.

[0071] As can be seen from the above technical solutions, this application has the following advantages:

[0072] 1. This application solves the problem of missed recognition caused by blurred seals or complex forms in existing technologies by using a collaborative mechanism of initial extraction and secondary processing. It achieves complete extraction of key information and seal content in business confirmation letters, and significantly improves the robustness of recognition in complex scenarios.

[0073] 2. This application solves the problem that existing technologies cannot automatically assess the credibility of recognition by dynamically judging the completeness of the initial extraction results, realizes targeted secondary processing of unrecognized content, reduces the need for manual intervention, and improves processing efficiency.

[0074] 3. This application solves the problem of loose output format and difficulty in direct application of existing technologies by using structured data encapsulation technology, and realizes standardized JSON output of business confirmation letter information, which facilitates seamless integration with business systems and improves data flow efficiency. Attached Figure Description

[0075] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0076] Figure 1This is a flowchart of a business confirmation letter identification method in one embodiment of this application.

[0077] Figure 2 This is a schematic block diagram of a business confirmation letter identification system in one embodiment of this application.

[0078] Figure 3 This is a schematic diagram of the hardware structure of an electronic device in one embodiment of this application. Detailed Implementation

[0079] To make the purpose, features, and advantages of this application more apparent and understandable, specific embodiments and accompanying drawings will be used to clearly and completely describe the technical solution protected by this application. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this patent, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this patent.

[0080] The business confirmation letter recognition method involved in this application mainly targets the field of document recognition technology. Through a collaborative mechanism of initial extraction and secondary processing, it solves the problem of missed recognition caused by blurred seals or complex tables in existing technologies, and achieves complete extraction of key information and seal content of business confirmation letters, significantly improving the robustness of recognition in complex scenarios. By dynamically judging the completeness of the initial extraction results, it solves the defect of existing technologies that cannot automatically assess the recognition credibility, and realizes targeted secondary processing of unrecognized content, reducing the need for manual intervention and improving processing efficiency. Through structured data encapsulation technology, it solves the problem of loose output format and difficulty in direct application of existing technologies, and realizes standardized JSON output of business confirmation letter information, which facilitates seamless integration with business systems and improves data flow efficiency.

[0081] The business confirmation letter recognition method involved in this application mainly addresses the technical problems of existing port business confirmation letter recognition methods, such as the lack of dynamic integrity judgment and secondary processing mechanism, the inability to automatically repair missing content when the seal is blurry or the form is abnormal, and the loose output data format that is difficult to apply directly.

[0082] The following describes in detail the business confirmation letter identification method involved in this application. Specific details such as particular system structures and technologies are presented for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details.

[0083] In the business confirmation letter identification method involved in this application, the term "comprising" indicates the presence of the described feature, whole, step, operation, element, and / or component, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or collections thereof. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0084] To facilitate a clear description of the technical solutions of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" do not necessarily imply that they are different.

[0085] The terms "one embodiment" or "some embodiments" used in this application mean that one or more embodiments of this application include the specific features, structures, or characteristics described in that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this application do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0086] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0087] The business confirmation letter recognition method provided in this application embodiment is executed by a computer device, and correspondingly, the business confirmation letter recognition system runs in the computer device.

[0088] The following is a definition of some terms used in this plan to facilitate a better understanding of the plan:

[0089] A port business confirmation letter is an electronic confirmation document issued by the business initiator to the port operator in port logistics service scenarios. This document records core information about a specific business in a structured table format, including business elements such as the cargo transportation acceptance number, container specifications / number, operation time window, fee details, and settlement terms. It must also bear a registered electronic seal or digital signature of the enterprise. Its characteristics include: the document content follows a standard port business template; key fields (such as container number and fee item code) conform to port data specifications; and the electronic seal information is linked to the registration database. The port business confirmation letter is a key document in the digitalization process of port operations, enabling automated billing, operational accountability, and compliance auditing.

[0090] Figure 1 This is a flowchart of a business confirmation letter identification method according to an embodiment of this application. Figure 1 The executing entity can be a port business confirmation letter identification system. Depending on different requirements, the order of steps in this flowchart can be changed, and some steps can be omitted.

[0091] like Figure 1 As shown, the method for identifying this business confirmation letter includes:

[0092] Step S1: Obtain the business confirmation letter file. The business confirmation letter file is a single-page port business confirmation letter PDF file containing a title, table, seal and other content.

[0093] By limiting the business confirmation letter to a single-page PDF format and including a title, tables, and seal, the uniformity and integrity of the input data were ensured, providing a standardized data source for subsequent information extraction and structured processing.

[0094] Step S2: Convert the business confirmation letter file into business confirmation letter image data.

[0095] By converting PDF files into image data, the complexity of parsing the native PDF format is resolved, providing a unified image input interface for subsequent computer vision-based text and seal extraction.

[0096] In some specific embodiments, the business confirmation letter file is subjected to three-channel pixel conversion to generate business confirmation letter image data, which is in JPG, JPEG or PNG format.

[0097] By generating JPG / JPEG / PNG format images through three-channel pixel conversion, the problem of missing color channels in PDF conversion is solved, ensuring the complete preservation of color features and format adaptation during subsequent model processing.

[0098] Step S3: Perform initial extraction on the business confirmation letter image data, including key information extraction, seal extraction, and seal content recognition, to obtain initial extracted information;

[0099] Among them, key information extraction involves inputting the business confirmation letter image data into a pre-trained business confirmation letter key information extraction model and outputting key information extraction results, including all text fragments in the business confirmation letter image data, the category label corresponding to each text fragment, and the key name-key value pair relationship label of the question-answer;

[0100] Seal extraction includes locating the seal area, extracting the circular text image and the linearly arranged text image within the seal area, and preprocessing the circular text image.

[0101] Seal content recognition includes recognizing the content of preprocessed circular text images and linearly arranged text images, and outputting the seal content recognition results.

[0102] By performing key information extraction, seal extraction, and seal content recognition, parallel processing of text structure parsing and seal semantic understanding is achieved, improving overall recognition efficiency and reducing module coupling.

[0103] In some specific embodiments, the steps for extracting key information using a pre-trained business confirmation letter key information extraction model include:

[0104] Extract all text fragments from the image data of the business confirmation letter;

[0105] The text fragments are categorized into categories including questions, answers, titles, and other content;

[0106] Perform key-value matching on questions and answers, where questions correspond to key names and answers correspond to key values;

[0107] Output the key information extraction results, including all text fragments in the business confirmation letter image data, the category label corresponding to each text fragment, and the key name-key value pair relationship label of the question-answer.

[0108] By using a pre-trained model to classify text fragments and match question-answer key values, semantic relationship parsing of non-fixed format documents is achieved, avoiding the limitations of traditional rule-based methods that are heavily dependent on layout.

[0109] In some specific embodiments, seal extraction includes:

[0110] The pre-trained seal location detection model is used to locate the seal region and outputs the coordinates of the seal region bounding box and the seal region image. The seal location detection model is a deep learning model.

[0111] A pre-trained text region segmentation model is used to extract circular text images and linearly arranged text images within the stamp region image. The text region segmentation model is a deep learning model.

[0112] A polar coordinate transformation is performed on the circular text image to obtain a straightened circular text image.

[0113] By cascading the processing of seal region localization and text region segmentation models, the independent extraction of circular and linear text within the seal is achieved, providing a suitable input format for subsequent content recognition.

[0114] In some specific embodiments, a pre-trained seal content recognition model is used to recognize the seal content. This seal content recognition model is a deep learning model, including:

[0115] Input the linearly arranged text image and the straightened circular text image into the seal content recognition model, and output the seal content recognition result.

[0116] By unifying the recognition models for straightened circular text and linearly arranged text, the blind spots of traditional OCR in recognizing specially formatted text are solved, improving the completeness and accuracy of seal content analysis.

[0117] In some specific embodiments, the key information extraction model for the business confirmation letter is built based on the VI-LayoutXLM algorithm, and the training steps include:

[0118] Obtain a sample business confirmation letter file and convert it into image data of the business confirmation letter file.

[0119] The text fragments in the sample image data of the business confirmation letter are manually classified, and key-value matching is performed on the questions and answers to obtain the actual key information of the sample image data of the business confirmation letter, including the category label corresponding to each text fragment and the key name-key value pair relationship label of the question-answer. Each group of sample image data of the business confirmation letter and its actual key information are used as a sample to construct a sample key information dataset of the business confirmation letter.

[0120] The business confirmation letter key information extraction model was trained in two stages using a sample key information dataset of business confirmation letters:

[0121] The first stage trains the text segment category recognition capability of the business confirmation letter key information extraction model based on the category label corresponding to each text segment, and uses the cross-entropy loss function to optimize the category label recognition result;

[0122] In the second stage, based on the fixed category labels corresponding to each text segment, the key-value matching ability of the business confirmation letter key information extraction model is trained based on the key-value pair relationship labels of question-answer. The similarity comparison loss function is used to optimize the key-value pair matching relationship.

[0123] By optimizing text classification and key-value matching tasks through a phased training strategy, the problem of parameter conflict in multi-task models was solved, and the classification accuracy and relation mapping ability of the key information extraction model were improved.

[0124] In some specific embodiments, the seal position detection model is built based on the YOLO algorithm, and the training steps include:

[0125] Obtain a sample business confirmation letter file and convert it into image data of the business confirmation letter file.

[0126] In the sample image data of business confirmation letters, the coordinates of the bounding box of the seal area and the category label are manually annotated using the annotation tool. Each set of sample image data of business confirmation letters and its seal area bounding box coordinates and category label are used as a sample to construct a sample seal location dataset of business confirmation letters.

[0127] The seal location detection model was trained using a sample seal location dataset from business confirmation letters. The positioning accuracy of the seal region bounding box was optimized by using a multi-scale feature map prediction mechanism and the intersection-union loss function.

[0128] The text region segmentation model is built based on the Mask R-CNN algorithm, and the training steps include:

[0129] Obtain stamp sample images, and manually use annotation tools to annotate the circular text region segmentation mask and the straight text region segmentation mask in the stamp sample images. Use each set of stamp sample images and the corresponding circular text region segmentation mask and straight text region segmentation mask as a sample to construct a text region segmentation dataset.

[0130] A text region segmentation model was trained using a text region segmentation dataset. A pixel-level segmentation network was employed, and the text region segmentation results were optimized using a binary cross-entropy loss function.

[0131] By using multi-scale feature prediction and pixel-level segmentation networks, high-precision localization of the seal region bounding box and fine segmentation of the text region were achieved, providing reliable spatial feature support for subsequent processing.

[0132] In some specific embodiments, the seal content recognition model is built based on the PPOCRv4 algorithm, and the training steps include:

[0133] Obtain a sample image of a seal, extract a sample circular text image and a sample linear text image from the sample image, perform polar coordinate transformation on the sample circular text image to obtain a straightened sample circular text image;

[0134] Manually identify the text content of straightened sample circular text images and sample linear text images, and construct a stamp content dataset by taking each straightened sample circular text image and its text content as a sample and each linear text image and its text content as a sample.

[0135] A seal content recognition model was trained using a seal content dataset, and the seal content recognition results were optimized using a sequence transcription loss function.

[0136] By using polar coordinate transformation preprocessing and sequence transcription loss function, the problem of recognition difficulties caused by the deformation of circular characters is solved, and the end-to-end recognition efficiency of circular characters in seals is significantly improved.

[0137] Step S4: Analyze the initially extracted information to determine whether the seal portion and the table portion have been successfully identified.

[0138] If all are successfully identified, the initial extracted information will be used as the identification result of the business confirmation letter.

[0139] If there are parts of content that were not successfully identified, then the parts of content that were not successfully identified will be processed again to obtain the business confirmation letter identification result;

[0140] The secondary processing includes secondary processing of the table part and secondary processing of the seal part. The secondary processing of the table part is to generate the missing key name-key value pair relationship labels in the initial extracted information and supplement them into the key information extraction results. The secondary processing of the seal part is to match the seal area image with the seal template library and use the seal content information corresponding to the matched seal template image to replace the original seal content recognition result in the initial extracted information.

[0141] By setting branch judgment logic for the initial extraction results, targeted secondary processing of unrecognized content is achieved, avoiding the termination of the entire process due to partial recognition failure and enhancing the system's fault tolerance.

[0142] In some specific embodiments, the rules for determining whether the stamp portion and the table portion have been successfully identified include:

[0143] Extract the company name from the seal content recognition results and the company name from the title text fragment in the key information extraction results, and then calculate the similarity between the two. If the similarity is lower than the preset threshold, it is determined that the seal content has not been successfully recognized.

[0144] The key-value matching relationship of all questions and answers in the key information extraction results is verified. If a key name does not match a key value, it is determined that the table part content has not been successfully identified.

[0145] By using similarity threshold determination and key integrity verification rules, automated quality assessment of the recognition results is achieved, ensuring that the output data meets business logic constraints and consistency requirements.

[0146] In some specific embodiments, the secondary processing steps for the table portion include:

[0147] S401. Set up a key name library, which stores key names and their corresponding key-value regular expression rules;

[0148] S402. When the content of the table portion is not successfully recognized, perform OCR recognition on the table portion, extract all text fragments of the table portion, and record the spatial coordinates of the key name of each unmatched key value in the image;

[0149] S403. Match the key name of each unmatched key value with the key name in the key name library. If a match is successful, extract the text fragments that match the regular expression corresponding to the key name from all text fragments as candidate key values.

[0150] S404. Record the spatial coordinates of the candidate key values ​​in the image. Based on the spatial coordinates, select the candidate key value that is spatially closest to the unmatched key value as the matching key value of that key name, form the key name-key value pair relationship label of that key name, and add it to the key information extraction results.

[0151] The secondary processing of the seal includes:

[0152] S411. Set up a seal template library. The seal template library stores the mapping relationship between company names and corresponding seal template image groups. Each seal template image group contains at least one seal template image corresponding to the company. Each seal template image is associated with pre-stored seal content information.

[0153] S412. Extract the company name from the title text fragment in the extraction results. Based on the extracted company name, query the seal template library and select the seal template image group corresponding to the company name as the candidate seal template image group.

[0154] S413. Perform ORB feature point matching between the seal region image and each seal template image in the candidate seal template image group, calculate the image matching similarity score, and take the seal template image with the highest image matching similarity score and higher than the preset threshold as the target seal template image.

[0155] S414. Use the seal content information associated with the target seal template image as the actual seal content recognition result, replacing the original seal content recognition result in the initial extracted information.

[0156] By using a key name library regular expression matching and spatial coordinate association strategy, the problem of dynamically supplementing unrecognized key values ​​is solved, and the fault tolerance and repair capabilities in complex scenarios are improved by combining a seal template matching mechanism.

[0157] In some specific embodiments, in step S403, if there is no key name in the key name library that does not have a matching key value, the matching fails, and the key value corresponding to the key name is defined as blank;

[0158] If a match is successful, and there is no text fragment in the entire text that matches the regular expression corresponding to the key name, then the key value corresponding to that key name is defined as blank.

[0159] By defining key-value blanks and handling rules for failed regular expression matching, the output format of abnormal data is standardized, avoiding parsing errors or process interruptions in downstream systems due to missing content.

[0160] In some specific embodiments, in step S412, if the extracted company name does not exist in the seal template library, the current process ends and an error message is output.

[0161] In step S413, if there is no stamp template image with an image matching similarity score higher than the preset threshold, the current process ends and an error message is output.

[0162] Step S5: Encapsulate the business confirmation letter recognition result into JSON structured data.

[0163] By encapsulating the recognition results into JSON format, a standardized conversion from unstructured documents to machine-readable data is achieved, meeting the compatibility requirements of business systems for data interfaces.

[0164] In some specific embodiments, step S6 is also included: adding the business confirmation letter image data and the corresponding business confirmation letter recognition result to the training dataset, and using the training dataset to iteratively update the business confirmation letter key information extraction model, the seal position detection model, the text region segmentation model and the seal content recognition model through incremental learning.

[0165] By dynamically updating model parameters through incremental learning, the problem of model performance degradation caused by changes in business data distribution is solved, ensuring the stability and adaptability of the recognition system in long-term operation.

[0166] In one specific embodiment, the steps of the business confirmation letter identification method include:

[0167] Step S1: Obtain the business confirmation letter file. The business confirmation letter file is a single-page port business confirmation letter PDF file containing a title, table, seal and other content.

[0168] Step S2: Perform three-channel pixel conversion on the business confirmation letter file to generate business confirmation letter image data. The business confirmation letter image data is in JPG, JPEG or PNG format.

[0169] Step S3: Perform initial extraction on the business confirmation letter image data, including key information extraction, seal extraction, and seal content recognition, to obtain initial extracted information;

[0170] The key information extraction process, which utilizes a pre-trained business confirmation letter key information extraction model, includes the following steps:

[0171] Extract all text fragments from the image data of the business confirmation letter;

[0172] The text fragments are categorized into categories including questions, answers, titles, and other content;

[0173] Perform key-value matching on questions and answers, where questions correspond to key names and answers correspond to key values;

[0174] Output the key information extraction results, including all text fragments in the business confirmation letter image data, the category label corresponding to each text fragment, and the key name-key value pair relationship label of the question-answer;

[0175] Seal extraction includes:

[0176] The pre-trained seal location detection model is used to locate the seal region and outputs the coordinates of the seal region bounding box and the seal region image. The seal location detection model is a deep learning model.

[0177] A pre-trained text region segmentation model is used to extract circular text images and linearly arranged text images within the stamp region image. The text region segmentation model is a deep learning model.

[0178] Perform polar coordinate transformation on the circular text image to obtain a straightened circular text image;

[0179] Seal content recognition is performed using a pre-trained deep learning model, which includes:

[0180] Input the image of text arranged in straight lines and the image of text arranged in a straight ring into the seal content recognition model, and output the seal content recognition result;

[0181] The key information extraction model for business confirmation letters is built based on the VI-LayoutXLM algorithm. The training steps include:

[0182] Obtain a sample business confirmation letter file and convert it into image data of the business confirmation letter file.

[0183] The text fragments in the sample image data of the business confirmation letter are manually classified, and key-value matching is performed on the questions and answers to obtain the actual key information of the sample image data of the business confirmation letter, including the category label corresponding to each text fragment and the key name-key value pair relationship label of the question-answer. Each group of sample image data of the business confirmation letter and its actual key information are used as a sample to construct a sample key information dataset of the business confirmation letter.

[0184] The business confirmation letter key information extraction model was trained in two stages using a sample key information dataset of business confirmation letters:

[0185] The first stage trains the text segment category recognition capability of the business confirmation letter key information extraction model based on the category label corresponding to each text segment, and uses the cross-entropy loss function to optimize the category label recognition result;

[0186] In the second stage, based on the fixed category labels corresponding to each text segment, the key-value matching ability of the business confirmation letter key information extraction model is trained based on the key-value pair relationship labels of question-answer. The similarity comparison loss function is used to optimize the key-value pair matching relationship.

[0187] The seal location detection model is built based on the YOLO algorithm, and the training steps include:

[0188] Obtain a sample business confirmation letter file and convert it into image data of the business confirmation letter file.

[0189] In the sample image data of business confirmation letters, the coordinates of the bounding box of the seal area and the category label are manually annotated using the annotation tool. Each set of sample image data of business confirmation letters and its seal area bounding box coordinates and category label are used as a sample to construct a sample seal location dataset of business confirmation letters.

[0190] The seal location detection model was trained using a sample seal location dataset from business confirmation letters. The positioning accuracy of the seal region bounding box was optimized by using a multi-scale feature map prediction mechanism and the intersection-union loss function.

[0191] The text region segmentation model is built based on the Mask R-CNN algorithm, and the training steps include:

[0192] Obtain stamp sample images, and manually use annotation tools to annotate the circular text region segmentation mask and the straight text region segmentation mask in the stamp sample images. Use each set of stamp sample images and the corresponding circular text region segmentation mask and straight text region segmentation mask as a sample to construct a text region segmentation dataset.

[0193] A text region segmentation model was trained using a text region segmentation dataset. A pixel-level segmentation network was used, and the text region segmentation results were optimized using a binary cross-entropy loss function.

[0194] The seal content recognition model is built based on the PPOCRv4 algorithm, and the training steps include:

[0195] Obtain a sample image of a seal, extract a sample circular text image and a sample linear text image from the sample image, perform polar coordinate transformation on the sample circular text image to obtain a straightened sample circular text image;

[0196] Manually identify the text content of straightened sample circular text images and sample linear text images, and construct a stamp content dataset by taking each straightened sample circular text image and its text content as a sample and each linear text image and its text content as a sample.

[0197] A seal content recognition model was trained using a seal content dataset, and the seal content recognition results were optimized using a sequence transcription loss function.

[0198] Step S4: Analyze the initially extracted information to determine whether the seal portion and the table portion have been successfully identified;

[0199] If all are successfully identified, the initial extracted information will be used as the identification result of the business confirmation letter.

[0200] If there are parts of content that were not successfully identified, then the parts of content that were not successfully identified will be processed again to obtain the business confirmation letter identification result;

[0201] The rules for determining whether the stamp and table portions have been successfully recognized include:

[0202] Extract the company name from the seal content recognition results and the company name from the title text fragment in the key information extraction results, and then calculate the similarity between the two. If the similarity is lower than the preset threshold, it is determined that the seal content has not been successfully recognized.

[0203] The key-value matching relationship of all questions and answers in the key information extraction results is verified. If a key name does not match a key value, it is determined that the table part content has not been successfully identified.

[0204] The secondary processing steps for the table section include:

[0205] S401. Set up a key name library, which stores key names and their corresponding key-value regular expression rules;

[0206] S402. When the content of the table portion is not successfully recognized, perform OCR recognition on the table portion, extract all text fragments of the table portion, and record the spatial coordinates of the key name of each unmatched key value in the image;

[0207] S403. Match the key name of each unmatched key value with the key name in the key name library. If a match is successful, extract the text fragments that match the regular expression corresponding to the key name from all text fragments as candidate key values.

[0208] S404. Record the spatial coordinates of the candidate key values ​​in the image. Based on the spatial coordinates, select the candidate key value that is spatially closest to the unmatched key value as the matching key value of that key name, form the key name-key value pair relationship label of that key name, and add it to the key information extraction results.

[0209] The secondary processing of the seal includes:

[0210] S411. Set up a seal template library. The seal template library stores the mapping relationship between company names and corresponding seal template image groups. Each seal template image group contains at least one seal template image corresponding to the company. Each seal template image is associated with pre-stored seal content information.

[0211] S412. Extract the company name from the title text fragment in the extraction results. Based on the extracted company name, query the seal template library and select the seal template image group corresponding to the company name as the candidate seal template image group.

[0212] S413. Perform ORB feature point matching between the seal region image and each seal template image in the candidate seal template image group, calculate the image matching similarity score, and take the seal template image with the highest image matching similarity score and higher than the preset threshold as the target seal template image.

[0213] S414. Use the seal content information associated with the target seal template image as the actual seal content recognition result, replacing the original seal content recognition result in the initial extracted information.

[0214] Step S5: Encapsulate the business confirmation letter recognition result into JSON structured data.

[0215] Step S6: Add the business confirmation letter image data and the corresponding business confirmation letter recognition results to the training dataset. Use the training dataset to iteratively update the business confirmation letter key information extraction model, seal position detection model, text region segmentation model, and seal content recognition model through incremental learning.

[0216] The following are embodiments of the business confirmation letter recognition system provided in this application. This business confirmation letter recognition system and the business confirmation letter recognition methods in the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the business confirmation letter recognition system, please refer to the embodiments of the above business confirmation letter recognition methods.

[0217] like Figure 2 As shown, the business confirmation letter identification system includes:

[0218] The preprocessing module is used to acquire the business confirmation letter file and convert it into business confirmation letter image data.

[0219] The initial extraction module is used to initially extract the image data of the business confirmation letter to obtain initial extraction information.

[0220] The analysis and judgment module is used to analyze the initially extracted information and determine whether the seal part and the table part have been successfully recognized.

[0221] The secondary processing module is used to perform secondary processing on the parts of the content that were not successfully recognized, so as to obtain the recognition result of the business confirmation letter.

[0222] The structured processing module is used to encapsulate the business confirmation letter recognition results into JSON structured data.

[0223] The business confirmation letter recognition system in this embodiment is used to implement the business confirmation letter recognition method, the steps of which include:

[0224] S1. Obtain the business confirmation letter document. The business confirmation letter document is a single-page port business confirmation letter PDF file containing a title, table, seal and other content;

[0225] S2. Convert the business confirmation letter file into business confirmation letter image data;

[0226] S3. Perform initial extraction on the image data of the business confirmation letter, including key information extraction, seal extraction, and seal content recognition, to obtain the initial extracted information;

[0227] Among them, key information extraction involves inputting the business confirmation letter image data into a pre-trained business confirmation letter key information extraction model and outputting key information extraction results, including all text fragments in the business confirmation letter image data, the category label corresponding to each text fragment, and the key name-key value pair relationship label of the question-answer;

[0228] Seal extraction includes locating the seal area, extracting the circular text image and the linearly arranged text image within the seal area, and preprocessing the circular text image.

[0229] Seal content recognition includes recognizing the content of preprocessed circular text images and linearly arranged text images, and outputting the seal content recognition results;

[0230] S4. Determine whether the initial information extraction successfully identified the seal portion and the table portion;

[0231] If all are successfully identified, the initial extracted information will be used as the identification result of the business confirmation letter.

[0232] If there are parts of content that were not successfully identified, then the parts of content that were not successfully identified will be processed again to obtain the business confirmation letter identification result;

[0233] The secondary processing includes secondary processing of the table part and secondary processing of the seal part. The secondary processing of the table part is to generate the missing key name-key value pair relationship labels in the initial extracted information and supplement them into the key information extraction results. The secondary processing of the seal part is to match the seal area image with the seal template library and use the seal content information corresponding to the matched seal template image to replace the original seal content recognition result in the initial extracted information.

[0234] S5. Encapsulate the business confirmation letter recognition results into JSON structured data.

[0235] This application also provides an electronic device for implementing the various embodiments of this application. Figure 3 To illustrate the hardware structure of an electronic device according to various embodiments of this application, as shown in the following diagram... Figure 3 As shown, the electronic device includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor.

[0236] Those skilled in the art will understand that the electronic device structure involved in the embodiments of this application does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0237] In embodiments of this application, electronic devices include, but are not limited to, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described and / or claimed herein.

[0238] In this application embodiment, the processor can be implemented using at least one of an Application-Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a processor, a controller, a microcontroller, a microprocessor, or an electronic unit designed to perform the functions described herein. In some cases, such implementations can be implemented within a controller. For software implementations, implementations such as processes or functions can be implemented with separate software modules that allow the performance of at least one function or operation. The software code can be implemented by a software application (or program) written in any suitable programming language, and the software code can be stored in memory and executed by the controller.

[0239] In addition, the electronic device includes some functional modules not shown, which will not be described in detail here.

[0240] Those skilled in the art will understand that the various aspects of the electronic device provided in this application can be implemented as a system, method, or program product. Therefore, the various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0241] This application also provides a storage medium storing a program product capable of implementing the business confirmation letter identification method. In some possible implementations, various aspects of this application can also be implemented as a program product comprising program code that, when run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this application.

[0242] The storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0243] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for identifying port business confirmation letters, characterized in that, include: S1. Obtain the business confirmation letter file, which is a single-page port business confirmation letter PDF file containing a title, tables, seals and other content; S2. Convert the business confirmation letter file into business confirmation letter image data; S3. Perform initial extraction on the image data of the business confirmation letter, including key information extraction, seal extraction, and seal content recognition, to obtain initial extracted information; Among them, key information extraction involves inputting the business confirmation letter image data into a pre-trained business confirmation letter key information extraction model and outputting key information extraction results, including all text fragments in the business confirmation letter image data, the category label corresponding to each text fragment, and the key name-key value pair relationship label of the question-answer; Seal extraction includes: The pre-trained seal location detection model is used to locate the seal region and outputs the coordinates of the seal region bounding box and the seal region image. The seal location detection model is a deep learning model. A pre-trained text region segmentation model is used to extract circular text images and linearly arranged text images within the stamp region image. The text region segmentation model is a deep learning model. Perform polar coordinate transformation on the circular text image to obtain a straightened circular text image; Seal content recognition is performed using a pre-trained seal content recognition model, which is a deep learning model and includes: Input the image of text arranged in straight lines and the image of text arranged in a straight ring into the seal content recognition model, and output the seal content recognition result; S4. Determine whether the initial information extraction successfully identified the seal portion and the table portion; If all are successfully identified, the initial extracted information will be used as the identification result of the business confirmation letter. If there are parts of content that were not successfully identified, then the parts that were not successfully identified will be processed again to obtain the business confirmation letter identification result; The rules for determining whether the seal and table portions have been successfully recognized include: Extract the company name from the seal content recognition results and the company name from the title text fragment in the key information extraction results, and then calculate the similarity between the two. If the similarity is lower than the preset threshold, it is determined that the seal content has not been successfully recognized. The key-value matching relationship of all questions and answers in the key information extraction results is verified. If a key name does not match a key value, it is determined that the table part content has not been successfully identified. The secondary processing includes secondary processing of the table part and secondary processing of the seal part. The secondary processing of the table part is to generate the missing key name-key value pair relationship labels in the initial extracted information and supplement them into the key information extraction results. The secondary processing of the seal part is to match the seal area image with the seal template library and use the seal content information corresponding to the matched seal template image to replace the original seal content recognition result in the initial extracted information. The secondary processing steps for the table section include: S401. Set up a key name library, which stores key names and their corresponding key-value regular expression rules; S402. When the content of the table portion is not successfully recognized, perform OCR recognition on the table portion, extract all text fragments of the table portion, and record the spatial coordinates of the key name of each unmatched key value in the image; S403. Match the key name of each unmatched key value with the key name in the key name library. If a match is successful, extract the text fragments that match the regular expression corresponding to the key name from all text fragments as candidate key values. S404. Record the spatial coordinates of the candidate key values ​​in the image. Based on the spatial coordinates, select the candidate key value that is spatially closest to the unmatched key value as the matching key value of that key name, form the key name-key value pair relationship label of that key name, and add it to the key information extraction results. The secondary processing of the seal includes: S411. Set up a seal template library. The seal template library stores the mapping relationship between company names and corresponding seal template image groups. Each seal template image group contains at least one seal template image corresponding to the company. Each seal template image is associated with pre-stored seal content information. S412. Extract the company name from the title text fragment in the extraction results. Based on the extracted company name, query the seal template library and select the seal template image group corresponding to the company name as the candidate seal template image group. S413. Perform ORB feature point matching between the seal region image and each seal template image in the candidate seal template image group, calculate the image matching similarity score, and take the seal template image with the highest image matching similarity score and higher than the preset threshold as the target seal template image. S414. Use the seal content information associated with the target seal template image as the actual seal content recognition result, replacing the original seal content recognition result in the initial extracted information. S5. Encapsulate the business confirmation letter recognition results into JSON structured data.

2. The business confirmation letter identification method as described in claim 1, characterized in that, Step S3, which involves extracting key information using a pre-trained business confirmation letter key information extraction model, includes the following steps: Extract all text fragments from the image data of the business confirmation letter; The text fragments are categorized into categories including questions, answers, titles, and other content; Perform key-value matching on questions and answers, where questions correspond to key names and answers correspond to key values; Output the key information extraction results.

3. The business confirmation letter identification method as described in claim 2, characterized in that, It also includes step S6: adding the business confirmation letter image data and the corresponding business confirmation letter recognition results to the training dataset, and using the training dataset to iteratively update the business confirmation letter key information extraction model, seal position detection model, text region segmentation model and seal content recognition model through incremental learning.

4. A port business confirmation letter identification system, characterized in that, To implement the business confirmation letter identification method as described in any one of claims 1-3, the method includes: The preprocessing module is used to acquire the business confirmation letter file and convert it into business confirmation letter image data. The initial extraction module is used to initially extract the image data of the business confirmation letter to obtain initial extraction information. The analysis and judgment module is used to analyze the initially extracted information and determine whether the seal part and the table part have been successfully recognized. The secondary processing module is used to perform secondary processing on the parts of the content that were not successfully recognized, so as to obtain the recognition result of the business confirmation letter. The structured processing module is used to encapsulate the business confirmation letter recognition results into JSON structured data.

5. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the business confirmation letter identification method as described in any one of claims 1-3.

6. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the business confirmation letter identification method as described in any one of claims 1-3.

Images