Unstructured data feature multimode extraction and identification method

Abstract

The invention provides an unstructured data feature multimode extraction and identification method, which comprises the following steps of: setting a data type judgment module in a system, after unstructured data is accessed to the system, firstly carrying out preliminary analysis on the data, judging the data type according to the format and content features of the data, and judging the data type according to the content features of the data; carrying out an identification method based on content, natural language processing and semantic analysis on the plain text; and for the picture, a content identification method based on machine learning is adopted. According to the method, by designing a multimode feature extraction strategy, accurate recognition of data with various formats and semantic fuzziness is achieved, the capability of capturing text context semantic information is enhanced, the processing speed and precision of complex background and low-resolution pictures are improved, a flexible technical combination scheme is constructed, and the method is suitable for large-scale popularization and application. The feature extraction requirements of different types of unstructured data are met, so that the overall data processing efficiency and accuracy are improved, and the ever-increasing unstructured data processing requirements in the industries of finance, telecommunication, medical treatment and the like are effectively met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electric digital data processing, in particular to a non-structured data feature multi-mode extraction and identification method. BACKGROUND

[0002] In today's digital era, the financial, telecommunications, medical, Internet and other industries have generated a large amount of unstructured data, including documents, texts, pictures, etc. These data contain a large amount of important content such as personal information and sensitive data. Accurate feature extraction and content identification are crucial for data security risk detection, information classification management, etc. Currently, the non-structured data feature extraction technology mainly includes content-based feature extraction method, content identification method based on natural language processing, etc. In terms of content-based extraction, the existing technology often uses regular expressions, keyword matching, etc. For example, in personal information identification, regular matching of ID numbers is used, but this method has poor adaptability to complex data scenarios. When faced with data with various formats and ambiguous semantics, it is easy to miss or misidentify. The method based on natural language processing can process text content to some extent, but the traditional bag-of-words model or vector space model ignores the context relationship of the text and cannot effectively capture semantic information, resulting in low accuracy in identifying Chinese entity naming and similar feature texts. In terms of picture data processing, the existing picture classification and OCR technology has slow recognition speed and low accuracy in processing complex background and low-resolution pictures, which cannot meet the real-time processing needs of large-scale unstructured data. In addition, the existing technology mostly uses a single extraction method, lacks effective integration of multiple technologies, and cannot flexibly switch and combine appropriate extraction strategies for different types of unstructured data, resulting in low overall data processing efficiency and accuracy, which cannot meet the growing data processing needs of the industry. SUMMARY

[0003] The present application is to solve the problems of poor adaptability, weak semantic capture ability, low picture processing efficiency and accuracy, and ineffective integration of multiple extraction methods in existing non-structured data feature extraction technology. A non-structured data feature multi-mode extraction and identification method is provided, which realizes accurate identification of data with various formats and ambiguous semantics, enhances the capture ability of text context semantic information, improves the processing speed and accuracy of complex background and low-resolution pictures, and constructs a flexible technical combination scheme to meet the feature extraction needs of different types of unstructured data, thereby improving the efficiency and accuracy of overall data processing and effectively meeting the growing non-structured data processing needs of the financial, telecommunications, medical and other industries.

[0004] The application provides a non-structured data feature multi-mode extraction and identification method, comprising the following steps: S1, a data access module receives non-structured data, the non-structured data is any one of the following: a document, a pure text file and a picture file, and the non-structured data can include personal general information and / or personal sensitive information and / or property information and / or communication industry information; S2, the data access module judges the type of the non-structured data according to the data format, when the non-structured data is judged as a document, the pure text feature extraction enters step S3, and the picture feature extraction enters step S6; when the non-structured data is judged as a pure text file, the feature extraction enters step S3; when the non-structured data is judged as a picture file, the feature extraction enters step S6; S3, a multi-mode feature extraction module performs primary feature extraction based on content, judges whether the personal digital information and / or the personal general information meeting the keywords are included through content matching or document attribute detection, and performs feature extraction and outputs to a data processing and analysis module; S4, the multi-mode feature extraction module performs secondary content identification based on natural language processing on Chinese entity naming and similar feature files, performs feature item selection and feature weight calculation by using a text representation method of a vector space model to obtain a text vector representation including semantic relationships and context information between words, extracts the features of the personal general information, the personal sensitive information, the property information and the communication industry information, and outputs to the data processing and analysis module; S5, the multi-mode feature extraction module uses a BERT model to perform ultimate semantic analysis and identification of the personal sensitive information and / or the property information and / or the communication industry information, judges the information categories, and outputs to the data processing and analysis module, and enters step S7; S6, a picture model in the multi-mode feature extraction module classifies the pictures, when the pictures include sensitive information, uses a text detection algorithm to detect the personal information, the property information and the communication industry information in the text area of the pictures, outputs the coordinate information of the text area, converts the text area image into a format that can be processed by a computer, and performs identification, proofreading and correction to obtain the extracted features, and enters step S7; S7, the data processing and analysis module integrates and analyzes the extracted features, a result output module outputs the final identification result in the form of a visual interface or a data interface, and a non-structured data feature multi-mode extraction and identification method is completed.

[0005] The present invention discloses a method for multimodal extraction and recognition of unstructured data features. In a preferred embodiment, the multimodal feature extraction module is a multimodal fusion module including a deep learning network. First, a knowledge base labeled with personal general information and / or personal sensitive information and / or property information and / or communication industry information is used for pre-training. Then, a low-rank LoRA adapter is combined with the LoRA adapter for optimization training. Finally, the trained LoRA adapter is merged with the pre-trained model, the effect is verified, and iterative optimization is performed.

[0006] In the unstructured data feature multimodal extraction and recognition method of the present invention, as a preferred embodiment, in step S2, the document format includes Word and PDF, and the image format includes JPEG and PNG; In step S3, personal digital information includes any one or more of the following: ID card number, bank card number, IP address, MAC address, and QQ number; Content matching methods include regular expression matching and rule matching, which can be performed using regular expression matching libraries and Intel SIMD commands. The rule-based matching method is as follows: matching is performed based on a set of keywords, including ethnicity information, nationality information, marital history information, educational background information, and degree information; In step S4, the document attributes are detected and recorded as data features. The document attributes include type, size, and name. Chinese entity naming includes names, addresses, and employer names; similar characteristic files include construction plans, medical records, test reports, and operation logs. In step S3, when there is doubt about the recognition result of the content-based feature extraction method, step S4 is automatically invoked to perform auxiliary verification using the word segmentation recognition method based on natural language processing.

[0007] The unstructured data feature multimodal extraction and recognition method of the present invention, as a preferred embodiment, includes the following steps in step S4: S41. The Chinese word segmentation model of the multi-modal feature extraction module segments Chinese entity names and continuous text in similar feature files into word sequences, marks key information including personal names and place names, and removes stop words to obtain preprocessed words. S42. Use TF-IDF for feature selection and weight calculation: For each preprocessed word, calculate the word frequency in the current text and the inverse document frequency in the entire corpus, and use the product of the word frequency and the inverse document frequency as the weight of the preprocessed word. The words are sorted according to their weights, and the top N words with higher weights are selected as features. The remaining words are filtered out to obtain the preprocessed text. S43. Use the Doc2Vec model to represent the preprocessed text as vectors and output them to the data processing and analysis module.

[0008] In a preferred embodiment of the multi-modal extraction and recognition method for unstructured data features according to the present invention, in step S41, the method for removing stop words is as follows: establish a stop word list, traverse the sequence of words after word segmentation, delete the words in the stop word list, and retain the words with actual meanings. Stop words are words without actual semantic meanings, including "de", "le", "zai".

[0009] In a preferred embodiment of the multi-modal extraction and recognition method for unstructured data features according to the present invention, step S5 includes the following steps: S51. BERT model training: Collect text data including personal health physiological information, property information, and telecommunications field documents, clean and annotate the data, annotate key information including disease names, amounts, and phone numbers, and divide the annotated data into a training set and a test set according to a ratio of 8:2. Use a pre-trained BERT model as the base model. In the training stage, input the training set data into the BERT model, add a fully connected layer as the output layer, set the output categories, use the cross-entropy loss function and the Adam optimizer to train the model, and adjust the model parameters to gradually reduce the loss value of the model on the training set. S52. Input the test set data into the pre-trained BERT model, and fine-tune and optimize the model parameters in combination with the low-rank adaptation model of Lora. S53. Use the BERT model to perform semantic analysis and recognition on plain text files including personal health physiological information, property information, and telecommunications field document information, determine the category of the information, and output it to the data processing and analysis module, and enter step S7.

[0010] In a preferred embodiment of the multi-modal extraction and recognition method for unstructured data features according to the present invention, in step S51, an additional output layer is added to the pre-trained BERT model to adapt to different downstream NLP tasks.

[0011] In a preferred embodiment of the multi-modal extraction and recognition method for unstructured data features according to the present invention, step S6 includes the following steps: S61. Use the trained deep learning architecture to classify the pictures. S62. When sensitive information is included, use a deep learning-based text detection algorithm to detect the text area in the picture and output the coordinate information of the text area. S63. Cropping the text region according to its coordinates, and then performing noise reduction, binarization, segmentation and normalization to convert the text image into a computer-processable format. S64. Use the OCR engine to recognize the processed text image and convert the text in the image into a computer-readable text format. S65. Perform proofreading and correction, remove incorrectly identified characters, obtain the extracted features, and proceed to step S7.

[0012] The unstructured data feature multimodal extraction and recognition method of the present invention, as a preferred embodiment, includes a deep learning architecture including MobileNet and MobileVit in step S61; In step S62, the deep learning-based text detection algorithms include EAST and CTPN; In step S63, the noise reduction method is median filtering, and the binarization method is Otsu's algorithm. In step S64, the OCR engine is either the TesseractOCR engine or a CRNN model.

[0013] The present invention provides a method for multi-modal extraction and recognition of unstructured data features. In a preferred embodiment, the data access module, the multi-modal feature extraction module, the data processing and analysis module, and the result output module are sequentially connected to obtain a multi-modal extraction system for unstructured data features.

[0014] This invention focuses on the efficient and accurate feature extraction and content recognition of various types of unstructured data, such as documents and texts containing personal information, image files carrying sensitive information, and documents in the telecommunications field, to meet the needs of data security risk detection, information classification and management, and other aspects. It is widely used in many industry scenarios such as finance, telecommunications, medical care, and the Internet that require the processing and analysis of large amounts of unstructured data.

[0015] The present invention has the following advantages: (1) Improve the accuracy of data identification More accurate content matching: Traditional content extraction techniques based on single regular expressions or keyword matching are prone to missed or false detections when faced with data that is diverse in format and semantically ambiguous. This technical solution comprehensively utilizes multiple methods, including regular expressions, keyword matching, keyword set matching, and document attribute detection, combined with the high-performance Hyperscan matching library, to more comprehensively and accurately describe and identify data category features. For example, in personal information recognition, it can more accurately match common information such as ID card numbers and bank card numbers, as well as rule set information such as ethnicity and nationality, effectively reducing recognition errors.

[0016] Deeper Semantic Understanding: Traditional natural language processing methods, such as the bag-of-words model and vector space model, ignore the contextual relationships within the text and cannot effectively capture semantic information. This solution adopts a semantic analysis method based on the BERT model. Through pre-training and fine-tuning, it can fully utilize a large-scale corpus to capture semantic information at the word and sentence levels, improving the accuracy of recognizing textual content such as personal health and physiological information and financial information, and better understanding the true meaning behind the data.

[0017] More Precise Image Processing: Traditional image classification and OCR technologies suffer from accuracy issues when handling complex backgrounds and low-resolution images. This solution employs advanced deep learning architectures such as MobileNet and MobileVit for image classification, combined with an optimized OCR workflow, including text detection, image processing, and post-processing steps. This enables more accurate identification and extraction of sensitive information in images, effectively improving the precision of image data processing.

[0018] (2) Enhance data adaptability Addressing diverse data formats: Traditional technologies often only process data of specific formats or types, exhibiting poor adaptability to diverse unstructured data. This solution encompasses multiple feature extraction methods, including content-based, natural language processing, semantic analysis, and machine learning, enabling it to adapt to different types of unstructured data, including documents, text, images, and various complex data formats and scenarios.

[0019] Flexible handling of data from different sectors: Unstructured data from different industries has different characteristics and requirements, and previous technologies have been unable to meet the needs of applications across multiple sectors. This solution, through a combination of multi-modal fusion mechanisms and various extraction methods, can flexibly address the data processing needs of different industries such as finance, telecommunications, healthcare, and the internet, providing a more universal and effective data feature extraction solution for various industries.

[0020] (3) Expanding application scenarios Supports multi-task processing: The pre-trained BERT model can be fine-tuned to adapt to different downstream NLP tasks, such as text classification, named entity recognition, and question answering systems. This allows the technical solution to not only be used for data feature extraction and recognition, but also to be extended to more natural language processing application scenarios, providing users with broader functional support.

[0021] Meeting diverse industry needs: Combining image classification and OCR technology, it can identify and analyze sensitive information in images, meeting the image data security detection needs of industries such as finance and telecommunications. Simultaneously, the effective processing of various data types, including personal information and telecommunications data, provides a more comprehensive solution for data security management and risk detection across various industries. Attached Figure Description

[0022] Figure 1 This is a flowchart of a multi-modal feature extraction and recognition method for unstructured data. Detailed Implementation

[0023] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Example 1

[0024] A multi-modal extraction and recognition method for unstructured data features is proposed to address the problems of poor adaptability, weak semantic capture ability, low image processing efficiency and accuracy, and insufficient fusion of multiple methods in existing unstructured data feature extraction methods, thereby improving the efficiency and accuracy of data processing.

[0025] A data type detection module is set up in the system. When unstructured data is input into the system, it first performs preliminary analysis. The data type is determined based on the data format (e.g., file extension to determine document type, image format to determine image type) and content characteristics (e.g., whether the text contains specific keywords to determine whether it is sensitive text).

[0026] For text data containing obvious features (such as ID card numbers, bank card numbers, etc.), content-based feature extraction methods are preferred; for Chinese entity names or text with similar features, content recognition methods based on natural language processing are used; for text such as personal health and physiological information or property information, recognition methods based on semantic analysis are used; and for image files, content recognition methods based on machine learning are used.

[0027] During processing, the combination of extraction methods and parameter settings are dynamically adjusted based on the accuracy of the recognition results and the characteristics of the data. For example, when there are doubts about the recognition results of content-based feature extraction methods, natural language processing methods are automatically invoked for auxiliary verification to ensure the accuracy and reliability of the final recognition results.

[0028] The technical solution of this invention is based on an integrated unstructured data feature extraction system. This system mainly includes a data access module, a multi-modal feature extraction module, a data processing and analysis module, and a result output module. The data access module supports input of various unstructured data, such as documents (Word, PDF, etc.), plain text files, and image files (JPEG, PNG, etc.). The multi-modal feature extraction module integrates multiple technologies such as content-based feature extraction, natural language processing, semantic analysis, and machine learning image processing. The data processing and analysis module integrates and analyzes the extracted features. The result output module outputs the final recognition results in the form of a visual interface or data interface.

[0029] like Figure 1 As shown in the figure, this embodiment provides a method for multi-modal extraction and recognition of unstructured data features, which mainly includes the following steps: S1. The data access module receives unstructured data, which can be any of the following: documents, plain text files, and image files; S2, Data Analysis The data access module determines the data type. If it is document data, it uses the appropriate document parsing library (such as python-docx for Word documents and PyPDF2 for PDF documents) to parse it into plain text. When the document is identified, plain text feature extraction proceeds to step S3, and image file feature extraction proceeds to step S6. When the file is determined to be a plain text file, feature extraction proceeds to step S3; When the file is identified as an image file, feature extraction proceeds to step S6. Multi-strategy content feature extraction: S3. For data with clear and specific features, such as personal information, conventional detection techniques such as regular expressions, keyword matching, and keyword set matching are comprehensively used to describe and identify the data's category features. Simultaneously, document attribute detection is employed to detect document type, size, and name. Specifically, regular expression matching is used for common personal information such as ID card numbers, bank card numbers, IP addresses, MAC addresses, and QQ numbers; rule set matching is used for information such as ethnicity, nationality, marital history, education, and degrees. The system uses Intel's Hyperscan high-performance regular expression matching library, combining its specific syntax and working modes, and utilizing numerous efficient algorithms and Intel SIMD instructions to achieve high-performance content matching.

[0030] The specific method is as follows: During system initialization, the Hyperscan regular expression matching library is configured according to the recognition requirements. For ID card numbers, the regular expression is set to (^\d{15}$)|(^\d{18}$)|(^\d{17}(\d|X|x)$); for bank card numbers, it can be set to ^\d{16,19}$. For keyword sets (such as ethnicity and nationality information), relevant keywords (such as "Han nationality" and "China") are compiled into a set and imported into the system.

[0031] Content matching and recognition The Hyperscan library is used to iterate and match the parsed text data. When content matching a regular expression or keyword set is detected, its location, data type, and other information are recorded. For example, if a string matching the regular expression for "ID card number" is detected, it is marked as "ID card number" and recorded in the recognition results list. Simultaneously, document attributes (type, size, name) are detected, and this attribute information is also recorded as part of the data features.

[0032] Hyperscan achieves high-performance matching through the use of numerous efficient algorithms and IntelSIMD* instructions. It supports detection based on any combination of data columns within a specific database. For example, if the policy requests a combination of "name + ID number + mobile phone number," then "Zhang San" + "13333333333" + "110001198107011533" will trigger this policy. However, even if "Li Si" is also in the same database, "Li Si" + "13333333333" + "110001198107011533" will not trigger this policy. Similar logic is also supported to reduce potential false positives. For free-format text processed during detection, the character count of each data point in a single feature column must be within the configurable range to be considered a match. For example, in the detected email body text, the number of characters in "Zhang San" + "13333333333" and "110001198107011533" must each fall within the selected range for a match to appear. For text containing tabular data (such as Excel spreadsheets), all data in a single feature column must be located on the same row of the tabular text to be considered a match, in order to reduce overall false positives.

[0033] For data with specific features already identified through feature matching, including data matching specific features using regular expressions, to improve the accuracy of data detection, it is necessary to define specific verification algorithms for some data items with verification algorithms. Simultaneously, for data classes with finite sets, feature libraries are used to assist in defining data features. This mainly includes: verification of the ID card itself: this involves rule-based verification of the six-digit address, birth date code, and the last check digit. This verification can be performed directly according to the ID card authenticity verification algorithm. It also verifies the contextual relationship of the matched specific feature data, specifically the verification of preceding and following characters. For example, the ID card number cannot be preceded or followed by numbers; otherwise, it cannot be recognized as an ID card number string. To improve data accuracy, the ability to exclude invalid data is added. Invalid scenarios are excluded using keyword rules: this mainly excludes invalid data from the perspective of actual needs and scenarios. This method has relatively poor accuracy and can be improved through other methods such as natural language processing and machine learning.

[0034] S4. Multi-stage analysis in natural language processing includes the following steps: S41. Word segmentation and preprocessing For Chinese entity names (such as names, addresses, employers, etc.) and similar characteristic texts (such as construction plans, medical records, test reports, operation logs, etc.), a word segmentation method based on natural language processing is used for identification. The system has a built-in personal information classification and extracts personal information in various scenarios through Chinese word segmentation technology. In the text preprocessing stage, text segmentation and stop word removal are performed, using word granularity as feature granularity to improve the effectiveness of text feature extraction.

[0035] S42. Feature Extraction and Text Representation Feature extraction is performed using a vector space model for text representation, including feature selection and feature weight calculation. The original feature items (words) are scored and ranked according to evaluation metrics, and the highest-scoring feature item is selected, while the remaining feature items are filtered out. The text representation employs an improved approach to overcome the shortcomings of traditional bag-of-words (BOW) and vector space models, which ignore textual context and fail to represent semantic information.

[0036] The specific methods for content recognition based on natural language processing are as follows: Word segmentation technology implementation The system is built-in with a Chinese word segmentation model based on deep learning (such as THULAC or Jieba word segmentation tool). After receiving text data containing personal information (name, address, employer, etc.), the text is input into the word segmentation model, and the model splits the text into individual words according to the trained word library and algorithms. For example, for the text "Zhang San works in a certain technology company in Haidian District, Beijing", after word segmentation, we get "Zhang San", "in", "Beijing City", "Haidian District", "a certain", "technology company", "works", and mark key information such as personal names and place names.

[0037] Implementation of text preprocessing Word segmentation processing: Use the selected word segmentation tool to segment the text and convert the continuous text into a sequence of words.

[0038] Removing stop words: Establish a stop word list (such as words without actual semantic meaning like "de", "le", "zai", etc.), traverse the sequence of words after word segmentation, delete the words in the stop word list, and retain the words with actual meaning.

[0039] Implementation of feature extraction Adopt the TF-IDF (Term Frequency - Inverse Document Frequency) algorithm for feature term selection and weight calculation. For each word, calculate its term frequency (TF) in the current text and its inverse document frequency (IDF) in the entire corpus, and the product of the two is used as the weight of the word. Sort the words according to the weight size, and select the top N words with higher weights as feature terms, filtering out the remaining words.

[0040] Kmeans clustering; Extracting information such as web page body text, time, title, etc. for the text label path ratio; String similarity solving algorithm based on improved edit distance; Identification of integrity objects in text content; Phrase extraction based on mutual information and left - right information entropy; C - value, D - value algorithms, etc.

[0041] Implementation of text representation Adopt the Doc2Vec model to represent the preprocessed text. Train the word vectors in the text through the Doc2Vec model to obtain the vector representation of the text, which contains the semantic relationships and context information between words, and solves the deficiencies of traditional bag - of - words models and vector space models.

[0042] S5. Implementation method for semantic analysis and recognition For personal health and physiological information, financial information, and telecommunications documents, the BERT model is used for sample training to form an inference and recognition model. Through the pre-training task of the BERT model, a large-scale corpus is fully utilized to capture semantic information at the word and sentence levels. An additional output layer is added to the pre-trained BERT model for fine-tuning to adapt to different downstream NLP tasks, such as text classification and named entity recognition. The specific methods are as follows: S51, Data Preparation We collected a large amount of text data containing personal health and physiological information, financial information, and telecommunications documents. The data was cleaned and labeled, highlighting key information (such as disease names, amounts, and phone numbers). The labeled data was then divided into training and testing sets in an 8:2 ratio.

[0043] BERT model training A pre-trained BERT model (such as the BERT-Base, Chinese released by Google) is used as the base model. During the training phase, the training set data is input into the BERT model, and a fully connected layer is added as the output layer, setting the output category (such as "personal health information," "financial information," etc.). The model is trained using the cross-entropy loss function and the Adam optimizer, adjusting the model parameters to gradually reduce the loss value of the model on the training set.

[0044] Model fine-tuning and application The test set data is input into the trained model for fine-tuning, further optimizing the model parameters.

[0045] First, a knowledge base labeled with general personal information and / or sensitive personal information and / or property information and / or telecommunications industry information is used for pre-training. Then, the LoRA low-rank adaptation model is combined for optimization training. Finally, the trained LoRA adapter is merged with the pre-trained model, the effect is verified, and iterative optimization is performed.

[0046] S52, Semantic Analysis and Recognition The fine-tuned model is used in a real production environment to perform semantic analysis on the input text data, output recognition results, and determine the information category to which the text data belongs.

[0047] S6, Machine Learning Image Processing For image files containing personal information, telecommunications information, or other sensitive information, risk identification and analysis are performed using image classification technology from machine learning and OCR text content extraction technology.

[0048] Image Categories We employ deep learning architectures such as MobileNet and MobileVit, which are suitable for computer vision tasks such as image classification and object detection, to achieve efficient image classification.

[0049] OCR extraction OCR technology includes steps such as text detection, text recognition, and post-processing. First, the text region is detected to separate the text from the background; then, image processing techniques are used to denoise, binarize, segment, and normalize the text image, converting it into a computer-processable format; finally, the OCR engine is used to recognize the text and convert it into a computer-readable text format.

[0050] The specific methods for content recognition based on machine learning are as follows: S61. Implementation of Image Classification Dataset Construction: Collect image data containing personal information, telecommunications information, or other sensitive information, label the images (such as "ID card image", "bank card image", etc.), and construct training and testing datasets.

[0051] Model Selection and Training: Choose either MobileNet or MobileVit and input the training dataset into the model for training. During training, set appropriate hyperparameters (such as learning rate, batch size, etc.), and use the cross-entropy loss function and stochastic gradient descent optimization algorithm to update the model parameters, enabling the model to accurately classify images.

[0052] Model application: Input the image to be processed into the trained model, and the model outputs the image's category information to determine whether the image contains sensitive information.

[0053] S62. Text Detection: Use deep learning-based text detection algorithms (such as EAST and CTPN) to detect text regions in the image and output the coordinate information of the text regions.

[0054] S63. Image Processing: Based on the coordinates of the text region, crop the text region, and then perform noise reduction (such as median filtering), binarization (such as Otsu algorithm), segmentation and normalization to convert the text image into a clear and standardized format.

[0055] S64. Text Recognition: Using the Tesseract OCR engine or a deep learning-based OCR model (such as CRNN) to recognize the processed text images and convert the text in the images into computer-readable text format.

[0056] S65. Post-processing: Proofread and correct the recognized text, remove incorrectly recognized characters, and improve the accuracy of text recognition.

[0057] S7. The data processing and analysis module integrates and analyzes the extracted features, and the result output module outputs the final recognition results in the form of a visual interface or data interface. A multi-modal extraction and recognition method for unstructured data features is completed.

[0058] This embodiment also constructs a multi-modal fusion mechanism to automatically select and combine appropriate feature extraction methods based on different types of unstructured data. Through preliminary analysis and feature judgment of the data, the use of various methods such as content-based, natural language processing, semantic analysis, and machine learning is dynamically adjusted to achieve efficient and accurate feature extraction of unstructured data.

[0059] Through the above technical solutions, this invention can effectively improve the accuracy and efficiency of feature extraction from unstructured data, adapt to different types and formats of data, and has been successfully applied to the processing of unstructured data in industries such as finance, telecommunications, healthcare, and the Internet.

[0060] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for multi-modal extraction and recognition of unstructured data features, characterized in that: Includes the following steps: S1. The data access module receives unstructured data, which can be any of the following: documents, plain text files, and image files. Unstructured data may include general personal information and / or sensitive personal information and / or property information and / or telecommunications industry information. S2. The data access module determines the type of unstructured data based on the data format. When it is determined to be a document, plain text feature extraction proceeds to step S3, and image feature extraction proceeds to step S6. When the file is determined to be a plain text file, feature extraction proceeds to step S3; When the file is identified as an image file, feature extraction proceeds to step S6. S3, the multi-modal feature extraction module performs content-based primary feature extraction, and determines whether it includes personal digital information and / or personal general information that matches keywords through content matching or document attribute detection, and then performs feature extraction and outputs it to the data processing and analysis module; S4, the multi-modal feature extraction module performs secondary content recognition based on natural language processing for Chinese entity naming and similar feature files. It uses the text representation method of vector space model to select feature terms and calculate feature weights to obtain text vector representations that include semantic relationships and contextual information between words. It extracts features of general personal information, sensitive personal information, property information, and telecommunications industry information and outputs them to the data processing and analysis module. S5. The multi-modal feature extraction module uses the BERT model to perform final semantic analysis and identification of personal sensitive information and / or property information and / or communication industry information, determine the information category, and output it to the data processing and analysis module, then proceed to step S7. S6. The image model in the multi-modal feature extraction module classifies the image. When it includes sensitive information, the text detection algorithm is used to detect personal information, property information and communication industry information in the text area of ​​the image, output the coordinate information of the text area, and then convert the text area image into a computer-processable format for recognition, proofreading and correction to obtain the extracted features, and proceed to step S7. S7. The data processing and analysis module integrates and analyzes the extracted features, and the result output module outputs the final recognition results in the form of a visual interface or data interface, thus completing a multi-modal extraction and recognition method for unstructured data features.

2. The method for multi-modal extraction and recognition of unstructured data features according to claim 1, characterized in that: The multimodal feature extraction module is a multimodal fusion module including a deep learning network. First, it is pre-trained using a knowledge base that annotates general personal information and / or sensitive personal information and / or property information and / or communication industry information. Then, it is optimized by combining the LoRA low-rank adaptation model. Finally, the trained LoRA adapter is merged with the pre-trained model, the effect is verified, and iterative optimization is performed.

3. The method for multi-modal extraction and recognition of unstructured data features according to claim 1, characterized in that: In step S2, document formats include Word and PDF, and image formats include JPEG and PNG; In step S3, the personal digital information includes any one or more of the following: ID card number, bank card number, IP address, MAC address, and QQ number; Content matching methods include regular expression matching and rule matching, which can be performed using regular expression matching libraries and Intel SIMD commands. The method for rule matching is as follows: match according to a keyword set, where the keywords include ethnic information, nationality information, marital history information, educational background information, and degree information; In step S4, the document attributes are detected and recorded as data features. The document attributes include type, size, and name; Chinese entity naming includes name, address, and naming of employment units. The same type of feature files include construction plans, medical record books, test reports, and operation logs; In step S3, when there are doubts about the recognition results of the content-based feature extraction method, step S4 is automatically called to perform auxiliary verification through the word segmentation recognition method based on natural language processing.

4. The method for multi-modal extraction and recognition of unstructured data features according to claim 1, characterized in that: Step S4 includes the following steps: S41. The Chinese word segmentation model of the multi-modal feature extraction module performs word segmentation on the continuous text in Chinese entity naming and the same type of feature files, converts it into a word sequence, marks the key information including person names and place names, and then removes the stop words to obtain the preprocessed words; S42. Use TF-IDF for feature item selection and weight calculation: For each preprocessed word, calculate the word frequency in the current text and the inverse document frequency in the entire corpus, and take the product of the word frequency and the inverse document frequency as the weight of the preprocessed word; Sort the words according to the weight size, select the top N words with higher weights as feature items, filter out the remaining words, and obtain the preprocessed text; S43. Use the Doc2Vec model to perform vector representation on the preprocessed text and output it to the data processing and analysis module.

5. The method for multi-modal extraction and recognition of unstructured data features according to claim 4, characterized in that: In step S41, the method for removing stop words is as follows: establish a stop word list, traverse the word sequence after word segmentation, delete the words in the stop word list, and retain the words with actual meanings; The stop words are words without actual semantics, including "de", "le", "zai".

6. The method for multi-modal extraction and recognition of unstructured data features according to claim 2, characterized in that: Step S5 includes the following steps: S51. BERT model training: Collect text data including personal health physiological information, property information, and documents in the telecommunications field, clean and annotate the data, annotate the key information including disease names, amounts, and phone numbers, and divide the annotated data into a training set and a test set according to a ratio of 8:2; Use the pre-trained BERT model as the basic model. In the training stage, input the training set data into the BERT model, add a fully connected layer as the output layer, set the output categories, use the cross-entropy loss function and the Adam optimizer to train the model, and adjust the model parameters to gradually reduce the loss value of the model on the training set; S52. Input the test set data into the pre-trained BERT model, and fine-tune and optimize the model parameters in combination with the low-rank adaptation model of Lora; S53. Use the BERT model to perform semantic analysis and recognition on the plain text files including personal health physiological information, property information, and telecommunications field file information, judge the information category to which they belong, and output it to the data processing and analysis module, and enter step S7.

7. The method for multi-modal extraction and recognition of unstructured data features according to claim 6, characterized in that: In step S51, an additional output layer is added to the pre-trained BERT model to adapt to different downstream NLP tasks.

8. The method for multi-modal extraction and recognition of unstructured data features according to claim 1, characterized in that: Step S6 includes the following steps: S61. Use the trained deep learning architecture to classify the pictures; S62. When sensitive information is included, use a deep learning-based text detection algorithm to detect the text region in the image and output the coordinate information of the text region. S63. Cropping the text region according to its coordinates, and then performing noise reduction, binarization, segmentation and normalization to convert the text image into a computer-processable format. S64. Use the OCR engine to recognize the processed text image and convert the text in the image into a computer-readable text format. S65. Perform proofreading and correction, remove incorrectly identified characters, obtain the extracted features, and proceed to step S7.

9. The method for multi-modal extraction and recognition of unstructured data features according to claim 8, characterized in that: In step S61, the deep learning architecture includes MobileNet and MobileVit; In step S62, the deep learning-based text detection algorithms include EAST and CTPN; In step S63, the noise reduction method is median filtering, and the binarization method is Otsu's algorithm. In step S64, the OCR engine is either the Tesseract OCR engine or a CRNN model.

10. The method for multi-modal extraction and recognition of unstructured data features according to claim 1, characterized in that: The data access module, the multi-modal feature extraction module, the data processing and analysis module, and the result output module are connected in sequence to obtain an unstructured data feature multi-modal extraction system.

Images