A multimedia data retrieval method based on deep learning character recognition

Abstract

The application discloses a kind of multimedia data retrieval methods based on deep learning character recognition, it is related to mail attachment content analysis, identification, business marking field, it will mass mail content decompression and according to file format hierarchical analysis and business marking in turn classified retrieval, for user quickly find out business label lay certain foundation, also provide a self-identification solution for attachment content automatic marking, by obtaining uploaded compressed package, decompression into corresponding file content, according to file format classification analysis, synchronous based on business understanding, under the condition of artificial arrangement, obtain the training data required by business marking model, construct mail multimedia text extraction model and business label classification model according to training data characteristics, output graphic content and business label, find attachment information of business level attention from mass mail data, improve retrieval efficiency, realize import and mark, actively recommend the attachment information of attention, to this release manpower.

Description

Technical Field

[0001] This invention relates to the fields of email attachment content parsing, recognition, and business tagging, and in particular to a multimedia data retrieval method based on deep learning text recognition. Background Technology

[0002] Current situation:

[0003] Smuggling, as an international illegal trade activity, is often accompanied by communication between domestic and foreign entities, generating massive amounts of email data during trade exchanges. These emails and their attachments frequently contain a wealth of information, including orders, product weight, product models, and prices. Identifying key information elements relevant to anti-smuggling investigations as evidence is a crucial method for qualitatively and quantitatively determining the nature of illegal smuggling activities.

[0004] Generally, there are two approaches to carrying out this work:

[0005] Import email data into software such as Foxmail and Outlook, and manually check the email information and attachments one by one to find relevant orders, contracts, customs declarations and attachments.

[0006] We have customized and developed full-text search capabilities based on the open-source distributed search engine ElasticSearch. After importing email data, you can quickly retrieve the desired attachment information using keywords.

[0007] Problems and drawbacks:

[0008] While manual screening is accurate, it is time-consuming and labor-intensive, and often leaves one at a loss when faced with a large amount of email data.

[0009] While using a storage search engine can quickly retrieve information from attachments, the keyword matching accuracy is insufficient, and the search results cannot meet the requirements of anti-smuggling police officers.

[0010] Email attachments include various data structures such as images and PDFs, which cannot be parsed, matched, and identified by search engines within the anti-smuggling industry at the operational level, thus failing to meet users' daily needs.

[0011] In conclusion, another feasible solution is needed to parse and identify useful information from massive amounts of email attachments. Summary of the Invention

[0012] The technical problem to be solved by this invention is to propose a multimedia data retrieval method based on deep learning text recognition, which decompresses massive amounts of email content and performs hierarchical parsing and business tagging according to file format, thereby classifying and retrieving the data. This lays a foundation for users to quickly find business tags and also provides a self-recognition solution for automatic tagging of attachment content.

[0013] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0014] A multimedia data retrieval method based on deep learning text recognition involves obtaining uploaded compressed packages, decompressing them into corresponding file contents, classifying and parsing them according to file format, and simultaneously obtaining training data for a business tagging model based on business understanding and manual processing. A multimedia text extraction model and a business tag classification model are then constructed based on the features of the training data, outputting the text and image content and business tags. The method specifically includes the following steps:

[0015] Step 1: Upload the compressed email file and create an import task;

[0016] Step 2: According to the import task, use the Linux decompression command to decompress the compressed file, call the parsing template and JavaMail to obtain the basic information of the email, and batch input the basic information of the email and attachments into the database. After inputting into the database, update the task information to the parsing success status.

[0017] Step 3: Build and train an email multimedia text extraction model based on manually compiled email image data;

[0018] Step 4: Based on the manually sorted email text categories, construct and train a business tag classification model;

[0019] Step 5: Determine the attachment type in the email. For image attachments, call the multimedia text extraction model and output editable formatted text content.

[0020] Step 6: Call the business tag classification model on the text attachments of the email and the text attachments output by the email multimedia text extraction model, and output the business tags;

[0021] Step 7: Store the business tags and attachment relationships output by the business tag classification model into the business library, and then present them through the interface;

[0022] In step 3, a multimedia text extraction model for emails is built and trained based on manually compiled email image data. This includes the following steps:

[0023] Step 3.1: Through training, an OCR detection model, an OCR orientation classifier model, and an OCR recognition model are obtained. The sample to be recognized is processed by the OCR detection model to output the four coordinates P of the text box in the image. i The coordinates of the four points are arranged clockwise, starting from the bottom left corner.

[0024] Step 3.2: Based on the four-point coordinates P and the rich text image, perform four-point image cropping to obtain a quadrilateral text line image, and then classify it using an OCR orientation classifier model to obtain the angle θ. i , where θ i ∈[0,360];

[0025] Step 3.3, using angle θ i and the coordinates of the four points P i Calculate the hybrid transformation matrix MaT_M, and perform a matrix transformation on the four-point text line image based on the hybrid transformation matrix MaT_M to transform it into a positive definite text line rectangle image;

[0026] Specifically, correcting a quadrilateral image into a positive definite text line rectangle image mainly involves two steps: angle transformation and perspective distortion transformation. Among them, perspective distortion transformation can improve calculation accuracy, speed up inference, and reduce pixel loss. The specific derivation process is as follows: Assume that the coordinates of the quadrilateral vertices are P=[(x1,y1),(x2,y2),(x3,y3),(x4,y4)], and its true angle is θ.

[0027] Step 1: Calculate the angle transformation matrix MaT_θ, as follows:

[0028]

[0029] Step 2, calculate the coordinates P′ of the four points after the angle transformation, as follows:

[0030]

[0031] P′=[(x′1,y′1),(x′2,y′2),(x′3,y′3),(x′4,y′4)]

[0032] Step 3, calculate the perspective transformation matrix MaT_T, the specific calculation is as follows:

[0033] w=max(x′1,x′2,x′3,x′4)-min(x′1,x′2,x′3,x′4)

[0034] h=max(y′1,y′2,y′3,y′4)-min(y′1,y′2,y′3,y′4)

[0035] Pm =[(0,0),(w,0),(w,h),(0,h)]

[0036] MaT_T=PT(P′,P m )

[0037] Where w and h represent the width and height of the quadrilateral, respectively, and P m PT represents the coordinates of the quadrilateral after the affine transformation, and PT represents the calculation of the affine matrix.

[0038] Step 4, calculate the hybrid transformation matrix MaT_M, the specific calculation is as follows:

[0039] MaT_M=MaT_θ×MaT_T

[0040] To correct a quadrilateral image into a positive definite text line rectangle image, an affine transformation is first performed using MaT_θ, followed by a perspective transformation using MaT_T. The quadrilateral image can be calculated by matrix multiplication with MaT_M.

[0041] Step 3.4: Use the OCR recognition model to obtain the text field string, and at the same time, label and test the massive amount of randomly selected email image data that was not used in the training.

[0042] As a further preferred embodiment of the multimedia data retrieval method based on deep learning text recognition of the present invention, in step 3, the construction of the email multimedia text extraction model includes an attachment parsing module, an OCR detection model, an OCR direction classifier, and an OCR recognition model.

[0043] The attachment parsing module is used to parse the multimedia files extracted from the email into three-channel images.

[0044] The OCR detection model is used to detect multiple text foreground regions from a three-channel image and crop the multiple text foreground regions into multiple quadrilateral text line images through various connected component algorithms.

[0045] An OCR orientation classifier is used to classify the orientation of quadrilateral text line images, calculate the true rotation angle of the quadrilateral text line images, and transform the quadrilateral text line images into positive definite text line rectangle images through perspective transformation.

[0046] The OCR recognition model is used to perform OCR recognition on rectangular images of text lines. It also combines the TopK of business corpus to obtain the text content. Furthermore, it combines the foreground region coordinates and text content to extract text from multimedia image files and output editable and formatted text content.

[0047] Among them, OCR stands for Optical Character Recognition, which generally refers to the recognition of text in images; TopK is a professional interface term, which generally refers to the top k most likely categories that the neural network considers.

[0048] As a further preferred embodiment of the multimedia data retrieval method based on deep learning character recognition of the present invention, the file categories include invoices, documents, contracts, and transfer records, and the file formats include JPEG, PNG, BMP, GIP, and PDF.

[0049] As a further preferred embodiment of the multimedia data retrieval method based on deep learning text recognition of the present invention, the construction of the business tag classification model includes a text parsing module and a text classification module;

[0050] The file parsing module is used to automatically read the text content of a file based on its filename.

[0051] The text classification model is used to classify parsed text using TextCNN, and to correct the category by combining business data to output business labels; TextCNN is short for Text Convolutional Neural Networks, which means a convolutional neural network used for text classification.

[0052] As a further preferred embodiment of the multimedia data retrieval method based on deep learning character recognition of the present invention, the file parsing module supports the following file types: CSV, TXT, XLS, XLSX, DOC, DOCX, PPTX, EML, and PDF.

[0053] As a further preferred embodiment of the multimedia data retrieval method based on deep learning text recognition of the present invention, in step 4, a business tag classification model is constructed and trained based on the manually sorted email text categories, specifically including the following steps:

[0054] Step 4.1: Select multiple email attachments containing text as training samples. For different file formats, use different parsing tools. Specifically, PDF files are parsed using the Python extension package pymupdf, docx and pptx files are parsed using the Python extension packages docx and pptx, xls and xlsx files are parsed using the Python extension package xlrd, and html and eml files are parsed using the Python extension package bs4.

[0055] Step 4.2: Perform word segmentation and cleaning on the text obtained from the file parsing;

[0056] The word segmentation uses the jieba word segmentation tool; jieba is a Python Chinese word segmentation component that can perform word segmentation, part-of-speech tagging, keyword extraction, and other functions on Chinese text, and also supports custom dictionaries.

[0057] Step 4.3: After word segmentation, use the stop word list from Harbin Institute of Technology for filtering to obtain the final text. Use the final text as the dataset to train the TextCNN text classification model.

[0058] Step 4.4 involves labeling and testing randomly selected file data that was not used in the training.

[0059] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0060] 1. This invention provides a multimedia data retrieval method based on deep learning text recognition. It decompresses massive amounts of email content and performs hierarchical parsing and business tagging according to file format, thereby classifying and retrieving the data. This lays a foundation for users to quickly find business tags and also provides a self-recognition solution for automatic tagging of attachment content. By obtaining the uploaded compressed package, decompressing it into corresponding file content, classifying and parsing it according to file format, and simultaneously based on business understanding, the training data required for the business tagging model is obtained under manual processing. Based on the features of the training data, an email multimedia text extraction model and a business tag classification model are constructed, and the image and text content and business tags are output.

[0061] 2. This invention finds attachment information of business interest from massive email data, improves retrieval efficiency, enables tagging upon import, and proactively recommends attachment information of interest, thereby freeing up manpower;

[0062] 3. This invention is compatible with image and PDF parsing capabilities. Using this invention, these two types of data can be parsed and business tags can be applied simultaneously, maximizing coverage of various email attachment formats. Attached Figure Description

[0063] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0064] Figure 1 This is a schematic diagram of the business process call of the present invention;

[0065] Figure 2 This is a flowchart of the multimedia text extraction process of the present invention;

[0066] Figure 3 This is a flowchart of the business tag classification process of this invention;

[0067] Figure 4 This is a schematic diagram of the OCR detection model of the present invention;

[0068] Figure 5 This is a network diagram of the OCR directional classifier structure of the present invention;

[0069] Figure 6 This is a schematic diagram of the OCR recognition model of the present invention;

[0070] Figure 7 This is the network diagram of the text extraction model structure of this invention. Detailed Implementation

[0071] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings:

[0072] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention. Those skilled in the art should understand that various substitutions and modifications made to the present invention based on basic technical knowledge and means in the art without departing from the above-described technical concept should be included within the protection scope of the present invention.

[0073] The purpose of this patent is to decompress massive amounts of email content, perform hierarchical parsing based on file format, and then classify and retrieve it by business tagging. This lays a foundation for users to quickly find business tags and also provides a self-recognition solution for automatic tagging of attachments. By acquiring the uploaded compressed package, decompressing it into corresponding file content, classifying and parsing it according to file format, and simultaneously obtaining the training data required for the business tagging model based on business understanding and manual processing, a multimedia text extraction model and a business tag classification model are constructed based on the features of the training data, outputting image and text content and business tags.

[0074] The specific implementation steps are as follows:

[0075] Step 1: Upload the compressed email file and create an import task;

[0076] Step 2: According to the import task, use the Linux decompression command to decompress the compressed file, call the parsing template and JavaMail to obtain the basic information of the email, and batch input the basic information of the email and attachments into the database. After inputting into the database, update the task information to the parsing success status.

[0077] Step 3: Build and train an email multimedia text extraction model based on manually compiled email image data;

[0078] Step 4: Based on the manually sorted email text categories, construct and train a business tag classification model;

[0079] Step 5: Determine the attachment type in the email. For image attachments, call the multimedia text extraction model and output editable formatted text content.

[0080] Step 6: Call the business tag classification model on the text attachments of the email and the text attachments output by the email multimedia text extraction model, and output the business tags;

[0081] Step 7: Store the business tags and attachment relationships output by the business tag classification model into the business library, and then present them through the interface.

[0082] The following are the attached figures and brief descriptions:

[0083] For each email compressed file, the model call and result output will be completed according to the business process call diagram, as shown in the attached diagram. Figure 1 As shown:

[0084] The image parsing stage, or multimedia text extraction model, is mainly composed of four modules: an attachment parsing module, an OCR detection model, an OCR direction classifier, and an OCR recognition model. OCR, short for Optical Character Recognition, generally refers to the recognition of text within images.

[0085] First, the attachment parsing module parses the multimedia files extracted from the email into three-channel images. The file types mainly include invoices, documents, contracts, and transfer records, and the file formats mainly include JPEG, PNG, BMP, GIP, and PDF.

[0086] Secondly, multiple text foreground regions are detected from the three-channel image using an OCR detection model. These regions are then cropped into multiple quadrilateral text line images using various connected component algorithms. An OCR orientation classifier is then used to classify the quadrilateral text line images by orientation, calculate the true rotation angle of the quadrilateral text line images, and finally transform the quadrilateral text line images into positive definite text line rectangle images through perspective transformation.

[0087] Finally, an OCR recognition model is used to perform OCR recognition on the aforementioned text line rectangle image. Simultaneously, combined with the TopK business corpus, the text content is obtained. Then, by combining the foreground region coordinates and the text content, text extraction is performed on the multimedia image file, outputting editable, formatted text content. Here, TopK refers to the professional interface vocabulary, generally referring to the k most likely categories that the neural network considers most probable.

[0088] like Figure 2 The flowchart shown is for multimedia text extraction.

[0089] The business tagging process, i.e., the construction of the business tag classification model, consists of a text parsing module and a text classification module. Its main flowchart is as follows: Figure 3 :

[0090] The file parsing module automatically reads the text content of the file based on the file name. The attachment parsing module supports the following file types: CSV, TXT, XLS, XLSX, DOC, DOCX, PPTX, EML, and PDF. Then, the parsed text is classified by the pre-trained text classification model TextCNN, and the category is corrected by combining business data to output the final business label.

[0091] TextCNN stands for Text Convolutional Neural Networks, which means a convolutional neural network used for text classification.

[0092] This invention provides a multimedia tagging and retrieval solution based on deep learning text recognition. The solution consists of three main stages: email upload, image parsing, and business tagging. An example from a practical application will be used to illustrate this solution.

[0093] Example 1:

[0094] This example illustrates how users can utilize email analytics products in real-world business scenarios. For instance, given a 1GB zip archive of emails, the goal is to quickly identify and retrieve attachments such as contracts, shipping containers, pay slips, and bank transfers.

[0095] like Figure 1 As shown, the uploaded email package is obtained, decompressed, and its attachment format is judged, and the corresponding model is called accordingly.

[0096] like Figure 2 As shown, image files are obtained through the email attachment parsing module, and the OCR detection model is then built, as detailed below. Figure 4 As shown;

[0097] Text lines are obtained using an OCR detection model, and an OCR orientation classifier is constructed. The specific structure and network diagram are as follows. Figure 5 As shown;

[0098] The structure and parameters of the orientation classifier are shown in Table 1.

[0099] Table 1

[0100]

[0101]

[0102] A rectangular image is obtained through an OCR orientation classifier, an OCR recognition model is built, and finally, editable text content is output, such as... Figure 6 As shown in Table 2, the structure and parameters of its recognition model are as follows.

[0103] Table 2

[0104] Identification Model Structure parameter enter Cropped text line image Convolutional layer Kernel=3*3, stride=1*1, channel=64 BN layer Relu Convolutional layer Kernel=3*3, stride=2*2, channel=64 BN layer Relu residual block Kernel=3*3, stride=1*1, channel=128 residual block Kernel=3*3, stride=2*2, channel=256 residual block Kernel=3*3, stride=2*1, channel=512 residual block Kernel=3*3, stride=2*1, channel=1024 AveragePooling Kernel = 1 * 2, stride = 1 * 2 Fully connected layer Channel = 6000 softmax

[0105] Using manually collected rich text image data as an application scenario, the real-world performance of the email multimedia text extraction model was tested. Multiple (8866) labeled rich text images from emails were selected as training samples. Through training, an OCR detection model, an OCR orientation classifier model, and an OCR recognition model were obtained. The sample to be recognized was processed by the OCR detection model, which outputs the four coordinates P of the image text box. i The four coordinates are arranged clockwise from the bottom left corner. Based on the four coordinates P and the rich text image, a four-point image is cropped to obtain a quadrilateral text line image. The angle θ is then determined by classifying the text using an OCR orientation classifier model. i , where θ i ∈[0,360], and finally through the angle θ i and the coordinates of the four points P i Calculate the hybrid transformation matrix MaT_M, and then perform a matrix transformation on the four-point text line image based on MaT_M to convert it into a positive definite text line rectangle image.

[0106] Specifically, correcting a quadrilateral image into a positive definite text line rectangle image mainly involves two steps: angle transformation and perspective distortion transformation. Among them, perspective distortion transformation can improve calculation accuracy, speed up inference, and reduce pixel loss. The specific derivation process is as follows: Assume that the coordinates of the quadrilateral vertices are P=[(x1,y1),(x2,y2),(x3,y3),(x4,y4)], and its true angle is θ.

[0107] Step 1: Calculate the angle transformation matrix MaT_θ, as follows:

[0108]

[0109] Step 2, calculate the coordinates P′ of the four points after the angle transformation, as follows:

[0110]

[0111] P′=[(x′1,y′1),(x′2,y′2),(x′3,y′3),(x′4,y′4)]

[0112] Step 3, calculate the perspective transformation matrix MaT_T, the specific calculation is as follows:

[0113] w=max(x′1,x′2,x′3,x′4)-min(x′1,x′2,x′3,x′4)

[0114] h=max(y′1,y′2,y′3,y′4)-min(y′1,y′2,y′3,y′4)

[0115] P m =[(0,0),(w,0),(w,h),(0,h)]

[0116] MaT_T=PT(P′,P m )

[0117] Where w and h represent the width and height of the quadrilateral, respectively, and P m PT represents the coordinates of the quadrilateral after the affine transformation, and PT represents the calculation of the affine matrix.

[0118] Step 4, calculate the hybrid transformation matrix MaT_M, the specific calculation is as follows:

[0119] MaT_M=MaT_θ×MaT_T

[0120] To correct a quadrilateral image into a positive definite text line rectangle image, an affine transformation is first performed using MaT_θ, followed by a perspective transformation using MaT_T. The quadrilateral image can be calculated by matrix multiplication with MaT_M.

[0121] Then, an OCR recognition model was used to obtain the text field string. Simultaneously, a massive dataset of 193 randomly selected, untrained rich text email images was labeled and tested. The statistical analysis of this test was divided into two main categories: line-by-line and character-by-character. The test results show that the email multimedia text extraction model can completely replace manual verification of email information and attachments. Table 3 below shows the specific test results:

[0122] Table 3

[0123]

[0124] Based on the email attachment format and the text content output by the email media text extraction model, a text classification model, TextCNN, is constructed. The network structure is shown in the diagram below. Figure 7 As shown in Table 4, its parameters and values ​​are as follows.

[0125] Table 4

[0126] parameter value learning rate 0.001 kernel size (2,3,4) Number of convolution kernels 256 Word vector dimension 100

[0127] Using email text data as an application scenario, the real-world performance of the text labeling model was tested. Multiple (5537) email attachments containing text were selected as training samples. Different parsing tools were used for different file formats: PDF files were parsed using the Python extension package pymupdf, docx and pptx files using the Python extension packages docx and pptx, xls and xlsx files using the Python extension package xlrd, and html and eml files using the Python extension package bs4. The parsed text was then segmented and cleaned. Jieba was used for segmentation, and the segmented text was filtered using the stop word list from Harbin Institute of Technology to obtain the final text. This final text was then used as the dataset to train the TextXNN text classification model. The model was then labeled and tested using 500 randomly selected files that were not part of the training dataset. The test results shown in Table 9 below demonstrate that the text labeling model can completely replace manual file classification.

[0128] Table 5

[0129]

[0130]

[0131] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0132] The above embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of this invention. The embodiments of the present invention have been described in detail above, but the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A multimedia data retrieval method based on deep learning character recognition, characterized in that: By obtaining the uploaded compressed package, decompressing it into corresponding file contents, classifying and parsing them according to file format, and simultaneously obtaining the training data required for the business tagging model based on business understanding and manual processing, a multimedia text extraction model and a business tag classification model are constructed based on the features of the training data, outputting image and text content and business tags; specifically including the following steps: Step 1: Upload the compressed email file and create an import task; Step 2: According to the import task, use Linux decompression commands to decompress the compressed file package and call... Parse the template and JavaMail to obtain basic information about the email, then batch store the basic information of the email and attachments into the database at once, and update the task information to the parsing success status after storage. Step 3: Build and train an email multimedia text extraction model based on manually compiled email image data; Step 4: Based on the manually sorted email text categories, construct and train a business tag classification model; Step 5: Determine the attachment type in the email. For image attachments, call and extract the multimedia text model, and output the result. Editable formatted text content; Step 6: Call the business logic to retrieve email text attachments and text attachments output by the email multimedia text extraction model. A tag classification model is developed, and business tags are output. Step 7: Store the business tags and attachment associations output by the business tag classification model into the business database. This is then presented through the interface; In step 3, a multimedia text extraction model for emails is built and trained based on manually compiled email image data. This includes the following steps: Step 3.1: Through training, an OCR detection model, an OCR orientation classifier model, and an OCR recognition model are obtained. The sample to be recognized is processed by the OCR detection model to output the four coordinates of the text box in the image. The coordinates of the four points are arranged clockwise, starting from the bottom left corner. Step 3.2: Based on the four-point coordinates P and the rich text image, perform four-point image cropping to obtain a quadrilateral text line image. Then, classify the image using an OCR orientation classifier model to obtain the angle. ,in ; Step 3.3, by angle and four coordinates Calculate the hybrid transformation matrix According to the hybrid transformation matrix Transform a four-point text line image into a positive definite text line rectangle image using matrix transformation; Specifically, correcting a quadrilateral image into a rectangular image containing positive text lines involves two steps: angle transformation and perspective distortion transformation. Perspective distortion transformation improves computational accuracy, speeds up inference, and reduces pixel loss. The specific derivation process is as follows: Assume the coordinates of the vertices of the quadrilateral are respectively... Its true angle is θ; Step 1, Calculate the angle transformation matrix The specific calculations are as follows: ； Step 2: Calculate the coordinates of the four points after the angle transformation. The specific calculations are as follows: ； ； Step 3, calculate the perspective transformation matrix. The specific calculations are as follows: ； in, , Representing the width and height of the quadrilateral respectively. Represents the coordinates of the quadrilateral after the affine transformation. Represents affine matrix calculation; Step 4: Calculate the hybrid transformation matrix The specific calculations are as follows: = ； To correct a quadrilateral image into a rectangular image for positive definite text lines, you first need to... Perform an affine transformation, and then through... To perform perspective transformation, the quadrilateral image only needs to be transformed with... By performing matrix multiplication, the corresponding positive definite text line rectangle image can be calculated; Step 3.4: Use the OCR recognition model to obtain the text field string, and at the same time, label and test the massive amount of randomly selected email image data that was not used in the training.

2. The multimedia data retrieval method based on deep learning character recognition according to claim 1, characterized in that: In step 3, the construction of the email multimedia text extraction model includes an attachment parsing module and an OCR module. Detection model, OCR orientation classifier, OCR recognition model; The attachment parsing module is used to parse the multimedia files extracted from the email into three-channel images. The OCR detection model is used to detect multiple text foreground regions from a three-channel image and crop the multiple text foreground regions into multiple quadrilateral text line images through various connected component algorithms. An OCR orientation classifier is used to classify the orientation of quadrilateral text line images, calculate the true rotation angle of the quadrilateral text line images, and transform the quadrilateral text line images into positive definite text line rectangle images through perspective transformation. The OCR recognition model is used to perform OCR recognition on rectangular images of text lines. It also combines the TopK of business corpus to obtain the text content. Furthermore, it combines the foreground region coordinates and text content to extract text from multimedia image files and output editable and formatted text content. Among them, OCR stands for Optical Character Recognition, which in Chinese means optical character recognition and refers to the recognition of text in images; TopK is a professional interface term that refers to the k most likely categories that the neural network considers to be.

3. The multimedia data retrieval method based on deep learning character recognition according to claim 2, characterized in that: The file categories include invoices, receipts, contracts, and transfer records, and their file formats include JPEG, PNG, BMP, GIP, and PDF.

4. The multimedia data retrieval method based on deep learning character recognition according to claim 1, characterized in that: The construction of the business tag classification model includes a text parsing module and a text classification module; The file parsing module is used to automatically read the text content of a file based on its filename. The text classification model is used to classify parsed text using TextCNN, and to correct the category by combining business data to output business labels; TextCNN stands for Text Convolutional Neural Networks, which means a convolutional neural network used for text classification.

5. A multimedia data retrieval method based on deep learning character recognition according to claim 4, characterized in that: The file parsing module supports the following file types: CSV, TXT, XLS, XLSX, DOC, DOCX, PPTX, EML, and PDF.

6. The multimedia data retrieval method based on deep learning character recognition according to claim 1, characterized in that: In step 4, a training business tag classification model is constructed based on the manually sorted email text categories. The process specifically includes the following steps: Step 4.1: Select multiple email attachments containing text as training samples. Use different parsing tools for different file formats. For PDF files, use the Python extension package pymupdf for parsing. docx and pptx format files are parsed using the Python extension packages docx and pptx, as well as xls and xlsx formats. The files are parsed using the Python xlrd extension package, while HTML and EML files are parsed using the Python bs4 extension package. Step 4.2: Perform word segmentation and cleaning on the text obtained from the file parsing; The word segmentation uses the jieba word segmentation tool; jieba is a Python Chinese word segmentation component that performs word segmentation, part-of-speech tagging, and keyword extraction on Chinese text, and also supports custom dictionaries. Step 4.3: After word segmentation, use the stop word list from Harbin Institute of Technology for filtering to obtain the final text. Use the final text as the dataset to train the TextCNN text classification model. Step 4.4 involves labeling and testing randomly selected file data that was not used in the training.

Images