Unstructured data feature multi-mode extraction and identification method

Abstract

The application provides a non-structured data feature multi-mode extraction and identification method, a data type judgment module is arranged in the system, when non-structured data accesses the system, firstly, data is analyzed preliminarily, the data type is judged according to the format and content feature of the data, the recognition method based on content, natural language processing and semantic analysis is used for pure text; for pictures, the content recognition method based on machine learning is used. The application realizes accurate identification of data with various formats and ambiguous semantics by designing a multi-mode feature extraction strategy, enhances the capturing ability of the context semantic information of the text, improves the processing speed and precision of complex background and low-resolution pictures, and constructs a flexible technical combination scheme to meet the feature extraction requirements of different types of non-structured data, thereby improving the efficiency and accuracy of overall data processing, and effectively coping with the increasing non-structured data processing requirements in the financial, telecommunication, medical and other industries.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing technology, specifically to a method for multi-modal extraction and recognition of unstructured data features. Background Technology

[0002] In today's digital age, industries such as finance, telecommunications, healthcare, and the internet generate massive amounts of unstructured data, including documents, text, and images. This data contains a wealth of important information, such as personal information and sensitive data. Accurate feature extraction and content recognition are crucial for data security risk detection and information classification management. Currently, unstructured data feature extraction techniques mainly include content-based feature extraction methods and natural language processing-based content recognition methods. In content-based extraction, existing technologies often employ regular expressions and keyword matching, such as using regular expressions to match ID card numbers in personal information recognition. However, this approach is poorly adapted to complex data scenarios, easily leading to missed or false detections when faced with diverse formats and semantically ambiguous data. While natural language processing-based methods can process text content to some extent, traditional bag-of-words or vector space models ignore textual context and cannot effectively capture semantic information, resulting in low accuracy in recognizing Chinese entity names and texts with similar features. In image data processing, existing image classification and OCR technologies suffer from slow recognition speed and low accuracy when handling complex backgrounds and low-resolution images, making it difficult to meet the needs of real-time processing of large-scale unstructured data. Furthermore, most existing technologies employ a single extraction method, lacking the effective integration of multiple technologies. They are unable to flexibly switch and combine appropriate extraction strategies for different types of unstructured data, resulting in overall data processing efficiency and accuracy falling short of ideal levels and failing to meet the industry's growing data processing needs. Summary of the Invention

[0003] This invention addresses the problems of poor adaptability, weak semantic capture ability, low image processing efficiency and accuracy, and ineffective integration of multiple extraction methods in existing unstructured data feature extraction technologies. It provides a multi-modal feature extraction and recognition method for unstructured data. By designing a multi-modal feature extraction strategy, it achieves accurate recognition of data with diverse formats and ambiguous semantics, enhances the ability to capture semantic information from textual context, improves the processing speed and accuracy of complex backgrounds and low-resolution images, and constructs flexible technology combinations to meet the feature extraction needs of different types of unstructured data. This improves the overall efficiency and accuracy of data processing and effectively addresses the growing demand for unstructured data processing in industries such as finance, telecommunications, and healthcare.

[0004] This invention provides a method for multi-modal extraction and recognition of unstructured data features, comprising the following steps: S1. The data access module receives unstructured data, which can be any of the following: documents, plain text files, and image files. Unstructured data may include general personal information and / or sensitive personal information and / or property information and / or telecommunications industry information. S2. The data access module determines the type of unstructured data based on the data format. When it is determined to be a document, plain text feature extraction proceeds to step S3, and image feature extraction proceeds to step S6. When the file is determined to be a plain text file, feature extraction proceeds to step S3; When the file is identified as an image file, feature extraction proceeds to step S6. S3, the multi-modal feature extraction module performs content-based primary feature extraction, and determines whether it includes personal digital information and / or personal general information that matches keywords through content matching or document attribute detection, and then performs feature extraction and outputs it to the data processing and analysis module; S4, the multi-modal feature extraction module performs secondary content recognition based on natural language processing for Chinese entity naming and similar feature files. It uses the text representation method of vector space model to select feature terms and calculate feature weights to obtain text vector representations that include semantic relationships and contextual information between words. It extracts features of general personal information, sensitive personal information, property information, and telecommunications industry information and outputs them to the data processing and analysis module. S5. The multi-modal feature extraction module uses the BERT model to perform final semantic analysis and identification of personal sensitive information and / or property information and / or communication industry information, determine the information category, and output it to the data processing and analysis module, then proceed to step S7. S6. The image model in the multi-modal feature extraction module classifies the image. When it includes sensitive information, the text detection algorithm is used to detect personal information, property information and communication industry information in the text area of ​​the image, output the coordinate information of the text area, and then convert the text area image into a computer-processable format for recognition, proofreading and correction to obtain the extracted features, and proceed to step S7. S7. The data processing and analysis module integrates and analyzes the extracted features, and the result output module outputs the final recognition results in the form of a visual interface or data interface, thus completing a multi-modal extraction and recognition method for unstructured data features.

[0005] The present invention discloses a method for multimodal extraction and recognition of unstructured data features. In a preferred embodiment, the multimodal feature extraction module is a multimodal fusion module including a deep learning network. First, a knowledge base labeled with personal general information and / or personal sensitive information and / or property information and / or communication industry information is used for pre-training. Then, a low-rank LoRA adapter is combined with the LoRA adapter for optimization training. Finally, the trained LoRA adapter is merged with the pre-trained model, the effect is verified, and iterative optimization is performed.

[0006] In the unstructured data feature multimodal extraction and recognition method of the present invention, as a preferred embodiment, in step S2, the document format includes Word and PDF, and the image format includes JPEG and PNG; In step S3, personal digital information includes any one or more of the following: ID card number, bank card number, IP address, MAC address, and QQ number; Content matching methods include regular expression matching and rule matching, which can be performed using regular expression matching libraries and Intel SIMD commands. The rule-based matching method is as follows: matching is performed based on a set of keywords, including ethnicity information, nationality information, marital history information, educational background information, and degree information; In step S4, the document attributes are detected and recorded as data features. The document attributes include type, size, and name. Chinese entity naming includes names, addresses, and employer names; similar characteristic files include construction plans, medical records, test reports, and operation logs. In step S3, when there is doubt about the recognition result of the content-based feature extraction method, step S4 is automatically invoked to perform auxiliary verification using the word segmentation recognition method based on natural language processing.

[0007] The unstructured data feature multimodal extraction and recognition method of the present invention, as a preferred embodiment, includes the following steps in step S4: S41. The Chinese word segmentation model of the multi-modal feature extraction module segments Chinese entity names and continuous text in similar feature files into word sequences, marks key information including personal names and place names, and removes stop words to obtain preprocessed words. S42. Use TF-IDF for feature selection and weight calculation: For each preprocessed word, calculate the word frequency in the current text and the inverse document frequency in the entire corpus, and use the product of the word frequency and the inverse document frequency as the weight of the preprocessed word. The words are sorted according to their weights, and the top N words with higher weights are selected as features. The remaining words are filtered out to obtain the preprocessed text. S43. Representing the preprocessed text as vectors using the Doc2Vec model and outputting the result to the data processing and analysis module.

[0008] In the multi-modal extraction and recognition method for unstructured data features according to the present invention, as a preferred embodiment, in step S41, the method for removing stop words is: establishing a stop word list, traversing the word sequence after word segmentation, deleting the words in the stop word list, and retaining the words with practical meanings; Stop words are words without practical semantics, including "de", "le", and "zai".

[0009] In the multi-modal extraction and recognition method for unstructured data features according to the present invention, as a preferred embodiment, step S5 includes the following steps: S51. BERT model training: collecting text data including personal health physiological information, property information and documents in the telecommunications field, cleaning and labeling the data, labeling key information including disease names, amounts and telephone numbers, and dividing the labeled data into a training set and a test set according to a ratio of 8:2; Using the pre-trained BERT model as the base model, in the training phase, inputting the training set data into the BERT model, adding a fully connected layer as the output layer, setting output categories, training the model using the cross-entropy loss function and the Adam optimizer, and adjusting model parameters to gradually reduce the loss value of the model on the training set; S52. Inputting the test set data into the pre-trained BERT model, and performing fine-tuning and optimization of model parameters in combination with the LoRA low-rank adaptation model; S53. Using the BERT model to perform semantic analysis and recognition on plain text files including personal health physiological information, property information and document information in the telecommunications field, determining the category of the information, outputting the result to the data processing and analysis module, and proceeding to step S7.

[0010] In the multi-modal extraction and recognition method for unstructured data features according to the present invention, as a preferred embodiment, in step S51, an additional output layer is added to the pre-trained BERT model to adapt to different downstream NLP tasks.

[0011] In the multi-modal extraction and recognition method for unstructured data features according to the present invention, as a preferred embodiment, step S6 includes the following steps: S61. Classifying images using the trained deep learning architecture; S62. When sensitive information is included, detecting the text region in the image using a deep learning-based text detection algorithm, and outputting the coordinate information of the text region; S63. Cropping the text region according to its coordinates, and then performing noise reduction, binarization, segmentation and normalization to convert the text image into a computer-processable format. S64. Use the OCR engine to recognize the processed text image and convert the text in the image into a computer-readable text format. S65. Perform proofreading and correction, remove incorrectly identified characters, obtain the extracted features, and proceed to step S7.

[0012] In the unstructured data feature multimodal extraction and recognition method of the present invention, as a preferred embodiment, the deep learning architecture in step S61 includes MobileNet and MobileVit. In step S62, the deep learning-based text detection algorithms include EAST and CTPN; In step S63, the noise reduction method is median filtering, and the binarization method is Otsu's algorithm. In step S64, the OCR engine is either the TesseractOCR engine or a CRNN model.

[0013] The present invention provides a method for multi-modal extraction and recognition of unstructured data features. In a preferred embodiment, the data access module, the multi-modal feature extraction module, the data processing and analysis module, and the result output module are sequentially connected to obtain a multi-modal extraction system for unstructured data features.

[0014] This invention focuses on the efficient and accurate feature extraction and content recognition of various types of unstructured data, such as documents and texts containing personal information, image files carrying sensitive information, and documents in the telecommunications field, to meet the needs of data security risk detection, information classification and management, and other aspects. It is widely used in many industry scenarios such as finance, telecommunications, medical care, and the Internet that require the processing and analysis of large amounts of unstructured data.

[0015] The present invention has the following advantages: (1) Improve the accuracy of data identification More accurate content matching: Traditional content extraction techniques based on single regular expressions or keyword matching are prone to missed or false detections when faced with data that is diverse in format and semantically ambiguous. This technical solution comprehensively utilizes multiple methods, including regular expressions, keyword matching, keyword set matching, and document attribute detection, combined with the high-performance Hyperscan matching library, to more comprehensively and accurately describe and identify data category features. For example, in personal information recognition, it can more accurately match common information such as ID card numbers and bank card numbers, as well as rule set information such as ethnicity and nationality, effectively reducing recognition errors.

[0016] Deeper Semantic Understanding: Traditional natural language processing methods, such as the bag-of-words model and vector space model, ignore the contextual relationships within the text and cannot effectively capture semantic information. This solution adopts a semantic analysis method based on the BERT model. Through pre-training and fine-tuning, it can fully utilize a large-scale corpus to capture semantic information at the word and sentence levels, improving the accuracy of recognizing textual content such as personal health and physiological information and financial information, and better understanding the true meaning behind the data.

[0017] More Precise Image Processing: Traditional image classification and OCR technologies suffer from accuracy issues when handling complex backgrounds and low-resolution images. This solution employs advanced deep learning architectures such as MobileNet and MobileVit for image classification, combined with an optimized OCR workflow, including text detection, image processing, and post-processing steps. This enables more accurate identification and extraction of sensitive information in images, effectively improving the precision of image data processing.

[0018] (2) Enhance data adaptability Addressing diverse data formats: Traditional technologies often only process data of specific formats or types, exhibiting poor adaptability to diverse unstructured data. This solution encompasses multiple feature extraction methods, including content-based, natural language processing, semantic analysis, and machine learning, enabling it to adapt to different types of unstructured data, including documents, text, images, and various complex data formats and scenarios.

[0019] Flexible handling of data from different sectors: Unstructured data from different industries has different characteristics and requirements, and previous technologies have been unable to meet the needs of applications across multiple sectors. This solution, through a combination of multi-modal fusion mechanisms and various extraction methods, can flexibly address the data processing needs of different industries such as finance, telecommunications, healthcare, and the internet, providing a more universal and effective data feature extraction solution for various industries.

[0020] (3) Expanding application scenarios Supports multi-task processing: The pre-trained BERT model can be fine-tuned to adapt to different downstream NLP tasks, such as text classification, named entity recognition, and question answering systems. This allows the technical solution to not only be used for data feature extraction and recognition, but also to be extended to more natural language processing application scenarios, providing users with broader functional support.

[0021] Meeting diverse industry needs: Combining image classification and OCR technology, it can identify and analyze sensitive information in images, meeting the image data security detection needs of industries such as finance and telecommunications. Simultaneously, the effective processing of various data types, including personal information and telecommunications data, provides a more comprehensive solution for data security management and risk detection across various industries. Attached Figure Description

[0022] Figure 1 This is a flowchart of a multi-modal feature extraction and recognition method for unstructured data. Detailed Implementation

[0023] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Example 1

[0024] A multi-modal extraction and recognition method for unstructured data features is proposed to address the problems of poor adaptability, weak semantic capture ability, low image processing efficiency and accuracy, and insufficient fusion of multiple methods in existing unstructured data feature extraction methods, thereby improving the efficiency and accuracy of data processing.

[0025] A data type detection module is set up in the system. When unstructured data is input into the system, the data is first preliminarily analyzed. The data type is determined based on the data format (such as file extension to determine document type, image format to determine image type) and content characteristics (such as whether the text contains specific keywords to determine whether it is sensitive text).

[0026] For text data containing obvious features (such as ID card numbers, bank card numbers, etc.), content-based feature extraction methods are preferred; for Chinese entity names or text with similar features, content recognition methods based on natural language processing are used; for text such as personal health and physiological information or property information, recognition methods based on semantic analysis are used; and for image files, content recognition methods based on machine learning are used.

[0027] During processing, the combination of extraction methods and parameter settings are dynamically adjusted based on the accuracy of the recognition results and the characteristics of the data. For example, when there are doubts about the recognition results of content-based feature extraction methods, natural language processing methods are automatically invoked for auxiliary verification to ensure the accuracy and reliability of the final recognition results.

[0028] The technical solution of this invention is based on an integrated unstructured data feature extraction system. This system mainly includes a data access module, a multi-modal feature extraction module, a data processing and analysis module, and a result output module. The data access module supports input of various unstructured data, such as documents (Word, PDF, etc.), plain text files, and image files (JPEG, PNG, etc.). The multi-modal feature extraction module integrates multiple technologies such as content-based feature extraction, natural language processing, semantic analysis, and machine learning image processing. The data processing and analysis module integrates and analyzes the extracted features. The result output module outputs the final recognition results in the form of a visual interface or data interface.

[0029] like Figure 1 As shown, this embodiment provides a method for multi-modal extraction and recognition of unstructured data features, which mainly includes the following steps: S1. The data access module receives unstructured data, which can be any of the following: documents, plain text files, and image files; S2, Data Analysis The data access module determines the data type. If it is document data, it uses the appropriate document parsing library (such as python-docx for Word documents and PyPDF2 for PDF documents) to parse it into plain text. When the document is identified, plain text feature extraction proceeds to step S3, and image file feature extraction proceeds to step S6. When the file is determined to be a plain text file, feature extraction proceeds to step S3; When the file is identified as an image file, feature extraction proceeds to step S6. Multi-strategy content feature extraction: S3. For data with clear and specific features, such as personal information, conventional detection techniques such as regular expressions, keyword matching, and keyword set matching are comprehensively used to describe and identify the data's category features. Simultaneously, document attribute detection is employed to detect document type, size, and name. Specifically, regular expression matching is used for common personal information such as ID card numbers, bank card numbers, IP addresses, MAC addresses, and QQ numbers; rule set matching is used for information such as ethnicity, nationality, marital history, education, and degrees. The system uses Intel's Hyperscan high-performance regular expression matching library, combining its specific syntax and working modes, and utilizing numerous efficient algorithms and Intel SIMD instructions to achieve high-performance content matching.

[0030] The specific method is as follows: During system initialization, the Hyperscan regular expression matching library is configured according to the recognition requirements. For ID card numbers, the regular expression is set to (^\d{15}$)|(^\d{18}$)|(^\d{17}(\d|X|x)$); for bank card numbers, it can be set to ^\d{16,19}$. For keyword sets (such as ethnicity and nationality information), relevant keywords (such as "Han nationality" and "China") are compiled into a set and imported into the system.

[0031] Content matching and recognition The Hyperscan library is used to iterate and match the parsed text data. When content matching a regular expression or keyword set is detected, its location, data type, and other information are recorded. For example, if a string matching the regular expression for "ID card number" is detected, it is marked as "ID card number" and recorded in the recognition results list. Simultaneously, document attributes (type, size, name) are detected, and this attribute information is also recorded as part of the data features.

[0032] Hyperscan achieves high-performance matching through the use of numerous efficient algorithms and IntelSIMD* instructions. It supports detection based on any combination of data columns within a specific database. For example, if the policy requests a combination of "name + ID number + mobile phone number," then "Zhang San" + "13333333333" + "110001198107011533" will trigger this policy. However, even if "Li Si" is also in the same database, "Li Si" + "13333333333" + "110001198107011533" will not trigger this policy. Similar logic is also supported to reduce potential false positives. For free-format text processed during detection, the character count of each data point in a single feature column must be within the configurable range to be considered a match. For example, in the detected email body text, the number of characters in "Zhang San" + "13333333333" and "110001198107011533" must each fall within the selected range for a match to appear. For text containing tabular data (such as Excel spreadsheets), all data in a single feature column must be located on the same row of the tabular text to be considered a match, in order to reduce overall false positives.

[0033] For data with specific features already identified through feature matching, including data matching specific features using regular expressions, to improve the accuracy of data detection, it is necessary to define specific verification algorithms for some data items with verification algorithms. Simultaneously, for data classes with finite sets, feature libraries are used to assist in defining data features. This mainly includes: verification of the ID card itself: this involves rule-based verification of the six-digit address, birth date code, and the last check digit. This verification can be performed directly according to the ID card authenticity verification algorithm. It also verifies the contextual relationship of the matched specific feature data, specifically the verification of preceding and following characters. For example, the ID card number cannot be preceded or followed by numbers; otherwise, it cannot be recognized as an ID card number string. To improve data accuracy, the ability to exclude invalid data is added. Invalid scenarios are excluded using keyword rules: this mainly excludes invalid data from the perspective of actual needs and scenarios. This method has relatively poor accuracy and can be improved through other methods such as natural language processing and machine learning.

[0034] S4. Multi-stage analysis in natural language processing includes the following steps: S41. Word segmentation and preprocessing For Chinese entity names (such as names, addresses, employers, etc.) and similar characteristic texts (such as construction plans, medical records, test reports, operation logs, etc.), a word segmentation method based on natural language processing is used for identification. The system has a built-in personal information classification and extracts personal information in various scenarios through Chinese word segmentation technology. In the text preprocessing stage, text segmentation and stop word removal are performed, using word granularity as feature granularity to improve the effectiveness of text feature extraction.

[0035] S42. Feature Extraction and Text Representation Feature extraction is performed using a vector space model for text representation, including feature selection and feature weight calculation. The original feature items (words) are scored and ranked according to evaluation metrics, and the highest-scoring feature item is selected, while the remaining feature items are filtered out. The text representation employs an improved approach to overcome the shortcomings of traditional bag-of-words (BOW) and vector space models, which ignore textual context and fail to represent semantic information.

[0036] The specific methods for content recognition based on natural language processing are as follows: Word segmentation technology implementation The system has a built-in deep learning-based Chinese word segmentation model (such as THULAC or Jieba word segmentation tool). After receiving text data containing personal information (name, address, employer, etc.), the text is input into the word segmentation model, which splits the text into individual words according to the trained lexicon and algorithm. For example, for the text "Zhang San works at a technology company in Haidian District, Beijing", the word segmentation results are "Zhang San", "in", "Beijing City", "Haidian District", "a", "technology company", "work", and key information such as person names and place names are marked out.

[0037] Implementation of Text Preprocessing Word Segmentation Processing: The selected word segmentation tool is used to segment the text, converting the continuous text into a word sequence.

[0038] Stop Word Removal: A stop word list (such as words with no actual semantic meaning like "de", "le", "zai", etc.) is established, the segmented word sequence is traversed, the words in the stop word list are deleted, and the words with actual meanings are retained.

[0039] Implementation of Feature Extraction The TF-IDF (Term Frequency-Inverse Document Frequency) algorithm is used for feature item selection and weight calculation. For each word, its term frequency (TF) in the current text and its inverse document frequency (IDF) in the entire corpus are calculated, and the product of the two is taken as the weight of the word. The words are sorted according to the weight, the top N words with higher weights are selected as feature items, and the remaining words are filtered out.

[0040] K-means clustering; extracting information such as web page text, time, and title through text label path ratio; string similarity solving algorithm based on improved edit distance; recognition of complete objects in text content; phrase extraction based on mutual information and left-right information entropy; C-value, D-value algorithms, etc.

[0041] Implementation of Text Representation The Doc2Vec model is used to represent the preprocessed text. The word vectors in the text are trained through the Doc2Vec model to obtain the vector representation of the text. This vector contains the semantic relationship and context information between words, which solves the deficiencies of the traditional bag-of-words model and vector space model.

[0042] S5. Implementation Mode of Recognition Based on Semantic Analysis For personal health and physiological information, financial information, and telecommunications documents, the BERT model is used for sample training to form an inference and recognition model. Through the pre-training task of the BERT model, a large-scale corpus is fully utilized to capture semantic information at the word and sentence levels. An additional output layer is added to the pre-trained BERT model for fine-tuning to adapt to different downstream NLP tasks, such as text classification and named entity recognition. The specific methods are as follows: S51, Data Preparation We collected a large amount of text data containing personal health and physiological information, financial information, and telecommunications documents. The data was cleaned and labeled, highlighting key information (such as disease names, amounts, and phone numbers). The labeled data was then divided into training and testing sets in an 8:2 ratio.

[0043] BERT model training A pre-trained BERT model (such as the BERT-Base, Chinese released by Google) is used as the base model. During the training phase, the training set data is input into the BERT model, and a fully connected layer is added as the output layer, setting the output category (such as "personal health information," "financial information," etc.). The model is trained using the cross-entropy loss function and the Adam optimizer, adjusting the model parameters to gradually reduce the loss value of the model on the training set.

[0044] Model fine-tuning and application The test set data is input into the trained model for fine-tuning, further optimizing the model parameters.

[0045] First, a knowledge base labeled with general personal information and / or sensitive personal information and / or property information and / or telecommunications industry information is used for pre-training. Then, the LoRA low-rank adaptation model is combined for optimization training. Finally, the trained LoRA adapter is merged with the pre-trained model, the effect is verified, and iterative optimization is performed.

[0046] S52, Semantic Analysis and Recognition The fine-tuned model is used in a real production environment to perform semantic analysis on the input text data, output recognition results, and determine the information category to which the text data belongs.

[0047] S6, Machine Learning Image Processing For image files containing personal information, telecommunications information, or other sensitive information, risk identification and analysis are performed using image classification technology from machine learning and OCR text content extraction technology.

[0048] Image Categories We employ deep learning architectures such as MobileNet and MobileVit, which are suitable for computer vision tasks such as image classification and object detection, to achieve efficient image classification.

[0049] OCR extraction OCR technology includes steps such as text detection, text recognition, and post-processing. First, the text region is detected to separate the text from the background; then, image processing techniques are used to denoise, binarize, segment, and normalize the text image, converting it into a computer-processable format; finally, the OCR engine is used to recognize the text and convert it into a computer-readable text format.

[0050] The specific methods for content recognition based on machine learning are as follows: S61. Implementation of Image Classification Dataset Construction: Collect image data containing personal information, telecommunications information, or other sensitive information, label the images (such as "ID card image", "bank card image", etc.), and construct training and testing datasets.

[0051] Model Selection and Training: Choose either MobileNet or MobileVit and input the training dataset into the model for training. During training, set appropriate hyperparameters (such as learning rate, batch size, etc.), and use the cross-entropy loss function and stochastic gradient descent optimization algorithm to update the model parameters, enabling the model to accurately classify images.

[0052] Model application: Input the image to be processed into the trained model, and the model outputs the image's category information to determine whether the image contains sensitive information.

[0053] S62. Text Detection: Use deep learning-based text detection algorithms (such as EAST and CTPN) to detect text regions in the image and output the coordinate information of the text regions.

[0054] S63. Image Processing: Based on the coordinates of the text region, crop the text region, and then perform noise reduction (such as median filtering), binarization (such as Otsu algorithm), segmentation and normalization to convert the text image into a clear and standardized format.

[0055] S64. Text Recognition: Using the Tesseract OCR engine or a deep learning-based OCR model (such as CRNN) to recognize the processed text images and convert the text in the images into computer-readable text format.

[0056] S65. Post-processing: Proofread and correct the recognized text, remove incorrectly recognized characters, and improve the accuracy of text recognition.

[0057] S7. The data processing and analysis module integrates and analyzes the extracted features, and the result output module outputs the final recognition results in the form of a visual interface or data interface. A multi-modal extraction and recognition method for unstructured data features is completed.

[0058] This embodiment also constructs a multi-modal fusion mechanism to automatically select and combine appropriate feature extraction methods based on different types of unstructured data. Through preliminary analysis and feature judgment of the data, the use of various methods such as content-based, natural language processing, semantic analysis, and machine learning is dynamically adjusted to achieve efficient and accurate feature extraction of unstructured data.

[0059] Through the above technical solutions, this invention can effectively improve the accuracy and efficiency of feature extraction from unstructured data, adapt to different types and formats of data, and has been successfully applied to the processing of unstructured data in industries such as finance, telecommunications, healthcare, and the Internet.

[0060] All unstructured data mentioned above was obtained with user authorization.

[0061] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for multi-modal extraction and recognition of unstructured data features, characterized in that: Includes the following steps: S1. The data access module receives unstructured data, which can be any of the following: documents, plain text files, and image files. Unstructured data may include general personal information and / or sensitive personal information and / or property information and / or telecommunications industry information. S2. The data access module determines the type of unstructured data based on the data format. When it is determined to be a document, plain text feature extraction proceeds to step S3, and image feature extraction proceeds to step S6. When the file is determined to be a plain text file, feature extraction proceeds to step S3; When the file is identified as an image file, feature extraction proceeds to step S6. S3, the multi-modal feature extraction module performs content-based primary feature extraction, and determines whether it includes personal digital information and / or personal general information that matches keywords through content matching or document attribute detection, and then performs feature extraction and outputs it to the data processing and analysis module; The multimodal feature extraction module is a multimodal fusion module that includes a deep learning network; When there is doubt about the recognition result of the content-based feature extraction method, step S4 is automatically invoked to perform auxiliary verification using the word segmentation and recognition method based on natural language processing. S4, the multi-modal feature extraction module performs secondary content recognition based on natural language processing for Chinese entity naming and similar feature files. It uses the text representation method of vector space model to select feature terms and calculate feature weights to obtain text vector representations that include semantic relationships and contextual information between words. It extracts features of general personal information, sensitive personal information, property information, and telecommunications industry information and outputs them to the data processing and analysis module. S5. The multi-modal feature extraction module uses the BERT model to perform final semantic analysis and identification of personal sensitive information and / or property information and / or communication industry information, determine the information category, and output it to the data processing and analysis module, then proceed to step S7. Step S5 includes the following steps: S51, BERT model training: Collect text data including personal health and physiological information, property information and telecommunications documents, clean and label the data, label key information including disease name, amount, and phone number, and divide the labeled data into training set and test set in an 8:2 ratio; Using a pre-trained BERT model as the base model, during the training phase, the training set data is input into the BERT model, a fully connected layer is added as the output layer, the output class is set, and the model is trained using the cross-entropy loss function and the Adam optimizer. The model parameters are adjusted so that the loss value of the model on the training set gradually decreases. S52. Input the test set data into the pre-trained BERT model, and fine-tune and optimize the model parameters by combining it with the low-rank adaptation model of Lora. S53. Use the BERT model to perform semantic analysis and identification on plain text files including personal health and physiological information, property information and telecommunications document information, determine the information category and output it to the data processing and analysis module, then proceed to step S7. S6. The image model in the multimodal feature extraction module classifies the image. When sensitive information is included, a text detection algorithm is used to detect personal information, property information, and communications industry information in the text region of the image, output the coordinate information of the text region, then convert the text region image into a computer-processable format for recognition, proofreading and correction to obtain extracted features, and proceed to step S7; S7. The data processing and analysis module integrates and analyzes the extracted features, and the result output module outputs the final recognition result in the form of a visual interface or a data interface, so that the unstructured data feature multimodal extraction and recognition method is completed.

2. The method for multi-modal extraction and recognition of unstructured data features according to claim 1, characterized in that: In step S2, the document formats include Word and PDF, and the image formats include JPEG and PNG; In step S3, the personal digital information includes any one or more of the following: ID number, bank card number, IP address, MAC address and QQ number; The content matching method includes regular matching and rule matching, and regular expression matching and rule set matching are performed through a regular expression matching library and Intel SIMD instructions; The rule matching method is: perform matching according to a keyword set, where the keywords include ethnic information, nationality information, marriage history information, education background information and degree information; In step S3, document attributes are detected and recorded as data features, and the document attributes include type, size, and name; Chinese named entities include names, addresses, and employing entity names, and the homogeneous feature files include construction schemes, medical records, test reports and operation logs.

3. The method for multi-modal extraction and recognition of unstructured data features according to claim 1, characterized in that: Step S4 includes the following steps: S41. The Chinese word segmentation model of the multimodal feature extraction module performs word segmentation on continuous text in the Chinese named entities and the homogeneous feature files, converts the continuous text into a word sequence, marks key information including person names and place names, and then removes stop words to obtain preprocessed words; S42. Use TF-IDF for feature item selection and weight calculation: for each preprocessed word, calculate the term frequency in the current text and the inverse document frequency in the entire corpus, and take the product of the term frequency and the inverse document frequency as the weight of the preprocessed word; Sort the words according to the weight, select the top N words with higher weights as feature items, filter out the remaining words, and obtain the preprocessed text; S43. Use the Doc2Vec model to represent the preprocessed text as a vector and output the vector to the data processing and analysis module.

4. The method for multi-modal extraction and recognition of unstructured data features according to claim 3, characterized in that: In step S41, the method for removing stop words is: establishing a stop word list, traversing the word sequence after word segmentation, deleting the words in the stop word list, and retaining words with actual meanings; The stop words are words with no actual semantic meaning, including "de", "le", and "zai".

5. The method for multi-modal extraction and recognition of unstructured data features according to claim 1, characterized in that: In step S51, an additional output layer is added to the pre-trained BERT model to adapt to different downstream NLP tasks.

6. The method for multi-modal extraction and recognition of unstructured data features according to claim 1, characterized in that: Step S6 includes the following steps: S61. Classify the image using the trained deep learning architecture; S62. When sensitive information is included, a deep learning-based text detection algorithm is used to detect the text region in the image, and output the coordinate information of the text region; S63. Cropping the text region according to its coordinates, and then performing noise reduction, binarization, segmentation and normalization to convert the text image into a computer-processable format. S64. Use the OCR engine to recognize the processed text image and convert the text in the image into a computer-readable text format. S65. Perform proofreading and correction, remove incorrectly identified characters, obtain the extracted features, and proceed to step S7.

7. The method for multi-modal extraction and recognition of unstructured data features according to claim 6, characterized in that: In step S61, the deep learning architecture includes MobileNet and MobileVit; In step S62, the deep learning-based text detection algorithms include EAST and CTPN; In step S63, the noise reduction method is median filtering, and the binarization method is Otsu's algorithm. In step S64, the OCR engine is either the Tesseract OCR engine or a CRNN model.

8. The method for multi-modal extraction and recognition of unstructured data features according to claim 1, characterized in that: The data access module, the multi-modal feature extraction module, the data processing and analysis module, and the result output module are connected in sequence to obtain an unstructured data feature multi-modal extraction system.

Images