Method and system for managing unstructured data in data channel

Abstract

The invention provides a method and a system for managing unstructured data in a data channel. The method comprises the following steps: accessing multi-source heterogeneous data, configuring various data source information, and capturing the unstructured data; identifying unstructured data, and performing deep processing and knowledge extraction; storing the unstructured data into a unified data model, intelligently classifying and indexing, and safely storing the unstructured data; updating and maintaining the metadata in real time to form a unified structured data view; entity, relation and attribute information is extracted, and a knowledge graph is constructed; the query intention of the user is understood, relevant information is retrieved from the knowledge graph and the data view, and context information is provided; an API is ecologically integrated and opened, and barrier-free circulation and sharing of data of a service system, a data platform and an analysis tool are achieved. According to the invention, a stable and extensible data platform architecture is constructed, the data sharing and cooperative work efficiency is improved, the data processing speed and accuracy are improved, and the security and compliance in transmission, storage and processing are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to a method and system for managing unstructured data in a data center. Background Art

[0002] In the current big data era, the amount of data faced by enterprises is growing explosively, especially the proportion of unstructured data (such as text, images, audio, video, etc.) has increased sharply. Unstructured data has become an important part of enterprise data assets with its rich content, diverse forms and huge value potential.

[0003] However, how to effectively manage, analyze and utilize these unstructured data has become a major challenge facing enterprises. At present, the management of unstructured data mainly has the following shortcomings:

[0004] 1. Since unstructured data has complex structures and diverse forms, including text, images, videos, audio and other types, and is stored in different systems and devices, it is difficult to integrate data and achieve unified management and efficient use.

[0005] 2. Traditional data processing methods are unable to cope with the processing needs of massive unstructured data. They are slow, inefficient, and prone to errors.

[0006] 3. Data between different systems and departments are often independent of each other, forming data islands, which leads to the inability to effectively share data and work collaboratively, affecting overall operational efficiency.

[0007] 4. The management of unstructured data involves many issues such as data security, privacy protection and compliance. Existing data security technologies are insufficient in processing unstructured data and cannot meet the increasingly stringent data security requirements of enterprises.

[0008] 5. The processing and analysis of unstructured data requires the comprehensive use of multiple technologies, such as NLP, image recognition, speech recognition, etc. The technical threshold is high, and it requires professional technical personnel and equipment support, and the human resource cost is high.

[0009] 6. Traditional data processing methods make it difficult to extract valuable knowledge and information from unstructured data, and are unable to perform intelligent search and reuse, limiting the full realization of data value. Summary of the invention

[0010] In view of this, the purpose of the present invention is to propose a method and system for managing unstructured data in a data middle platform, to build an efficient, intelligent, secure and reliable data middle platform management architecture, to achieve comprehensive integration, efficient processing, secure storage and intelligent analysis applications of unstructured data from different systems and devices, to integrate multiple intelligent processing technologies, to achieve deep mining and analysis of unstructured data, to ensure the security and compliance of unstructured data transmission, storage and processing, to form a structured data view, to lower the technical threshold for unstructured data processing and analysis, to build an enterprise knowledge graph, to achieve the extraction, reuse and intelligent search functions of knowledge and information in unstructured data, to promote the upgrading and development of unstructured data management, to provide strong support for the digital transformation and intelligent upgrading of enterprises, and to create greater data value for enterprises.

[0011] The present invention provides a method for managing unstructured data in a data center, comprising the following steps:

[0012] S1. Access multi-source heterogeneous data, configure various data source information, connect to the target data source corresponding to the multi-source heterogeneous data according to the configured data source information, and capture the unstructured data of the target data source;

[0013] Specifically, the various data source information includes: social media information, enterprise internal system information, Internet of Things device information, etc.; the unstructured data includes: text, images, audio, video, etc.

[0014] S2. Identify the unstructured data through an intelligent recognition engine, and perform deep processing and knowledge extraction on the unstructured data;

[0015] Specifically, the intelligent recognition engine integrates OCR (optical character recognition), NLP (natural language processing) and image recognition technologies to automatically extract text, tables, images and audio content from a variety of unstructured data sources such as pictures, scans, PDF documents, audio files, video files, etc.;

[0016] The present invention is based on an intelligent recognition engine that automatically recognizes and classifies key information in unstructured data, such as entities, relationships, emotions, etc. in text, objects, scenes, etc. in images. The intelligent recognition engine has a high degree of accuracy and generalization ability, and can process unstructured data of various types and formats.

[0017] S3. Define a unified data model, save the unstructured data after deep processing and knowledge extraction into the data model, intelligently classify and index the data in the data model, build a data security and privacy protection system, and securely store unstructured data;

[0018] Preferably, the defined unified data model includes the following basic fields:

[0019] ID: A unique identifier used to uniquely identify each data item;

[0020] Type: data type, such as text, image, audio, video;

[0021] Content: The actual content of the unstructured data, which can be binary data or a URL that references the data;

[0022] Metadata: Metadata, which describes additional information about the data, such as creation time, author, description, tags, etc.

[0023] Source: data source information, such as the uploader of the data, upload time, etc.;

[0024] S4. Design a dynamic metadata management mechanism to update and maintain the metadata of unstructured data in real time to form a unified structured data view;

[0025] Preferably, a multi-dimensional integration framework is designed to integrate unstructured data from different sources and formats to form a structured data view. The framework supports pre-processing operations such as data cleaning, deduplication, and standardization to ensure data accuracy and consistency.

[0026] The present invention uses a dynamic metadata management mechanism to update and maintain the metadata of unstructured data in real time. As unstructured data continues to change and increase, metadata information is updated in a timely manner, and metadata information is added, modified or deleted to ensure the accuracy and timeliness of metadata. Through dynamic metadata management, the entire life cycle of data can be tracked in real time to ensure data security and compliance.

[0027] Specifically, the method for forming a unified structured data view is to store unstructured data from different channels and of different types into the unified structured data model, and use the unified data model as the data view to perform unified management and analysis of unstructured data.

[0028] S5. Extract entities, relationships, and attribute information from unstructured data through an intelligent recognition engine, build a knowledge graph, and display the knowledge graph;

[0029] Specifically, the method for constructing a knowledge graph includes: storing information such as entities, relationships, attributes, etc. in the identified and extracted unstructured data in the graph database Neo4j, and using Neo4j to provide efficient graph query and analysis capabilities; the method for displaying a knowledge graph includes: displaying the knowledge graph information stored in Neo4j in the form of a knowledge graph, displaying the associations between entities, and providing graph query and analysis capabilities; entities can be entered or selected to query associations, and entities can be clicked to continue drilling down to analyze associations.

[0030] The present invention uses the key information extracted by the intelligent recognition engine to construct a knowledge graph. The knowledge graph represents the relationship between entities in a graph structure, which can intuitively display the internal connection and hierarchical structure of the data. Through the knowledge graph, users can query and understand data more conveniently and discover potential business value.

[0031] S6. Through the intelligent search engine, the user's query intention is understood, relevant information is retrieved from the knowledge graph and data view, and rich contextual information is provided;

[0032] Specifically, understanding the user's query intention includes: using natural language processing (NLP) in the intelligent search engine to parse the query sentence input by the user, perform word segmentation, part-of-speech tagging, and syntactic analysis, and analyze the keywords and phrases in the query and the relationship between them, laying the foundation for semantic analysis; by understanding these keywords, phrases, and sentences, semantic analysis is performed to further analyze the user's query intention; for example, for the query "What are the latest movies", semantic analysis will identify that the user wants to obtain information about the latest movies, not just web pages containing the two words "recent" and "movie";

[0033] Retrieving relevant information from the knowledge graph and data view and providing rich contextual information includes: retrieving relevant information from the knowledge graph and the unified structured data view based on the understood query intent of the user; the contextual information is the surrounding environment or background information related to the query conditions input by the user when performing information retrieval. Providing rich contextual information helps to more accurately understand and parse the query intent of the user, thereby improving the accuracy and relevance of the query results.

[0034] The intelligent search engine supports content-based unstructured data retrieval, uses natural language processing, semantic analysis and other technologies to understand the user's query intent, and retrieves relevant information from the knowledge graph and integrated data sets. Not only does it improve the accuracy of search results, but it also provides rich contextual information to help users better understand the data.

[0035] S7, ecological integration and open API, providing rich API interfaces to achieve barrier-free flow and sharing of data with business systems, data platforms and analytical tools.

[0036] Specifically, the method for providing a rich API interface includes: opening the knowledge graph and the unified structured data view to the outside in the form of an API interface, allowing other business systems to authorize calling the interface to obtain data, and the authorized call includes: authorized calling user name, calling password, calling validity period and other information; by calling the API interface, the knowledge graph data and data view data can be directly queried; at the same time, detailed call logs are recorded to facilitate security audits;

[0037] The present invention realizes barrier-free circulation and sharing of data by defining a unified data model and data interface.

[0038] Furthermore, the method of configuring various data source information in step S1 includes: configuring an acquisition method for acquiring unstructured data from various data sources, wherein the acquisition method includes: an API interface method, a data crawling method, and a file upload method;

[0039] The API interface method includes: configuring the API interface address, port, return data type, and authentication method information;

[0040] The data capture method includes: configuring the capture target address, whether there is authentication, and authentication method information;

[0041] The file upload method includes: configuring the target file location, login username, and password information.

[0042] Furthermore, the method for deep processing of unstructured data in step S2 includes:

[0043] For different types of unstructured data, corresponding processing technologies are used for deep processing. The deep processing includes: cleaning and converting unstructured data; the cleaning includes: removing duplicate data, format conversion, error correction, and ensuring the effectiveness and accuracy of subsequent processing; the conversion includes: converting unstructured data into structured or semi-structured forms to facilitate subsequent analysis and application;

[0044] The method for extracting knowledge from unstructured data includes: for text data, using NLP technology to extract key information of keywords, entities, and emotions; for image data, using image recognition technology to identify the features of objects, scenes, and colors in the image; for audio data, extracting audio content through speech recognition and emotion analysis technology; for audio and video data, extracting key frames and scene information through video analysis technology.

[0045] Furthermore, the method for intelligently classifying data in step S3 includes:

[0046] Intelligently classify the unstructured data after deep processing, and build an index based on the classification results; the intelligent classification includes: text classification, image classification, and audio classification;

[0047] The text classification includes: extracting and learning text features through machine learning algorithms: decision tree, support vector machine SVM, random forest, AdaBoost, LightGBM or XGBoost to achieve more accurate classification, understanding and processing natural language content in documents through natural language processing (NLP) technology, and extracting useful information from a large amount of text data for classification;

[0048] The image classification includes: converting the information in the image into a feature vector through a feature extraction algorithm such as SIFT, SURF or HOG, and performing classification prediction on the extracted feature vector through a classifier such as SVM, KNN, decision tree, random forest and neural network;

[0049] The audio classification includes: extracting the frequency spectrum features and Mel frequency cepstral coefficients (MFCC) of the audio as the basis for classification; and saving the classification results of the intelligent classification to the Metadata field of the corresponding information;

[0050] The method for indexing data in the S3 step includes: creating indexes for the ID, Type, and fields in metadata (such as tags) of the data model to facilitate subsequent efficient retrieval.

[0051] By building an efficient indexing mechanism, users can quickly locate the data they need, greatly improving the user experience.

[0052] Furthermore, the method of constructing a data security and privacy protection system in step S3 includes: data encryption, access control, audit tracking and data desensitization;

[0053] The data encryption includes: using advanced encryption algorithms such as AES, RSA, SM2, and SM4 to encrypt data transmission and encrypt sensitive information stored in the database to ensure that the data is not stolen or tampered with by unauthorized third parties;

[0054] The access control includes: identity authentication, ensuring that the identity is legitimate before allowing access to data, verifying the user name, password, and biometric identity information, confirming the user's true identity, and preventing illegal users from accessing data; setting different access levels and permissions for different users, limiting the user's access to data and the scope of operation, and further ensuring data security;

[0055] The audit trail includes: monitoring and recording the data access process, recording the user's access behavior, operation history and abnormal situations, so as to facilitate investigation and tracking when security issues occur, so as to take timely measures;

[0056] The data desensitization includes: using desensitization algorithm replacement, masking, data obfuscation, and data simulation to deform the sensitive data and then transmit and store it to achieve the protection of sensitive privacy data.

[0057] The present invention builds a comprehensive data security and privacy protection system and establishes a complete security and compliance governance system to ensure the security and compliance of unstructured data during storage, processing, and transmission.

[0058] Furthermore, the method for designing a dynamic metadata management mechanism in step S4 includes: dynamically generating metadata of the unstructured data in the data model, and updating and maintaining the metadata in real time;

[0059] The metadata includes: creation time, author, description, tag, data source, data format, processing status, and access rights;

[0060] The real-time updating and maintenance of metadata includes: updating and maintaining the processing status and access rights of the metadata.

[0061] Furthermore, the intelligent recognition engine in the step S5 of extracting key information through an intelligent recognition engine is an AI algorithm set consisting of a word segmentation algorithm, a part-of-speech tagging algorithm, a named entity recognition algorithm, a syntactic analysis algorithm, a semantic analysis algorithm, and a sentiment analysis algorithm, and these AI algorithms are used to perform entity extraction, relationship extraction, and attribute extraction, and identify and extract entity, relationship, and attribute information in unstructured data;

[0062] Wherein, the entity extraction is to identify entities that meet the definition from unstructured data;

[0063] The relationship extraction is to extract the relationship triples that conform to the facts from the unstructured data, that is, to determine whether there is a certain semantic directed relationship between two entities;

[0064] The attribute extraction is to extract attribute information used to describe the characteristics of the entity.

[0065] The present invention also provides a system for managing unstructured data in a data center, which executes the method for managing unstructured data in a data center as described above, including:

[0066] Capture unstructured data module: used to access multi-source heterogeneous data, configure various data source information, connect to the target data source corresponding to the multi-source heterogeneous data according to the configured data source information, and capture the unstructured data of the target data source;

[0067] Unstructured data recognition module: used to recognize the unstructured data through an intelligent recognition engine, and perform in-depth processing and knowledge extraction on the unstructured data;

[0068] Secure storage of unstructured data module: used to define a unified data model, save the unstructured data after deep processing and knowledge extraction into the data model, intelligently classify and index the data in the data model, build a data security and privacy protection system, and securely store unstructured data;

[0069] Forming a structured data view module: used to design a dynamic metadata management mechanism to update and maintain the metadata of unstructured data in real time and form a unified structured data view;

[0070] Knowledge graph construction module: used to extract entities, relationships, and attribute information from unstructured data through an intelligent recognition engine, construct a knowledge graph, and display the knowledge graph;

[0071] Retrieval information module: used to understand the user's query intent through the intelligent retrieval engine, retrieve relevant information from the knowledge graph and data view, and provide rich contextual information;

[0072] Data circulation and sharing module: used for ecological integration and open API, providing rich API interfaces to achieve barrier-free circulation and sharing of data with business systems, data platforms and analysis tools.

[0073] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the steps of the method for managing unstructured data by a data middle platform as described above are implemented.

[0074] The present invention also provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the program, the steps of the method for managing unstructured data by a data middle platform as described above are implemented.

[0075] Compared with the prior art, the present invention has the following beneficial effects:

[0076] The method and system for managing unstructured data by a data middle platform provided by the present invention realizes comprehensive integration, efficient processing, secure storage and intelligent analysis of unstructured data from different systems and devices by constructing a stable and scalable data middle platform architecture, breaking the data island phenomenon and improving the efficiency of data sharing and collaborative work; integrating multiple intelligent processing technologies, such as NLP, image recognition, speech recognition, etc., to realize deep mining and analysis of unstructured data, improve the speed and accuracy of data processing, and reduce the cost and error rate of manual intervention; establishing a complete data security management system, and ensuring the security and compliance of unstructured data during transmission, storage and processing by means of data encryption, access control and audit tracking; forming a structured data view, providing intuitive and easy-to-use tools, reducing the technical threshold for unstructured data processing and analysis, and enabling enterprises to use these data resources more conveniently; constructing an enterprise knowledge graph, realizing the extraction, reuse and intelligent search functions of knowledge and information in unstructured data, providing strong support for enterprise decision-making, and conducive to creating greater data value for enterprises. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the following detailed description of the preferred embodiment.The drawings are only for the purpose of illustrating the preferred embodiments and are not to be construed as limiting the invention.

[0078] In the attached picture:

[0079] Figure 1 This is a schematic diagram of the basic process of managing unstructured data by the data center in an embodiment of the present invention;

[0080] Figure 2 This is a flow chart of a method for managing unstructured data in a data center according to the present invention;

[0081] Figure 3 The figure is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0082] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and products consistent with some aspects of the present disclosure as detailed in the appended claims.

[0083] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0084] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0085] The embodiments of the present invention are described in further detail below.

[0086] The embodiment of the present invention provides a method for managing unstructured data in a data center. Figure 2 As shown, the following steps are included:

[0087] S1. Access multi-source heterogeneous data, configure various data source information, connect to the target data source corresponding to the multi-source heterogeneous data according to the configured data source information, and capture the unstructured data of the target data source;

[0088] The various data source information includes: social media information, enterprise internal system information, Internet of Things device information, etc.; the unstructured data includes: text, images, audio, video, etc.

[0089] The method for configuring various data source information includes: configuring an acquisition method for acquiring unstructured data from various data sources, wherein the acquisition method includes: an API interface method, a data crawling method, and a file upload method;

[0090] The API interface method includes: configuring the API interface address, port, return data type, and authentication method information;

[0091] The data capture method includes: configuring the capture target address, whether there is authentication, and authentication method information;

[0092] The file upload method includes: configuring the target file location, login username, and password information.

[0093] S2. Identify the unstructured data through an intelligent recognition engine, and perform deep processing and knowledge extraction on the unstructured data;

[0094] The method for deep processing of unstructured data includes:

[0095] For different types of unstructured data, corresponding processing technologies are used for deep processing. The deep processing includes: cleaning and converting unstructured data; the cleaning includes: removing duplicate data, format conversion, error correction, and ensuring the effectiveness and accuracy of subsequent processing; the conversion includes: converting unstructured data into structured or semi-structured forms to facilitate subsequent analysis and application;

[0096] The method for extracting knowledge from unstructured data includes: for text data, using NLP technology to extract key information of keywords, entities, and emotions; for image data, using image recognition technology to identify the features of objects, scenes, and colors in the image; for audio data, extracting audio content through speech recognition and emotion analysis technology; for audio and video data, extracting key frames and scene information through video analysis technology.

[0097] The intelligent recognition engine integrates OCR (optical character recognition), NLP (natural language processing) and image recognition technologies. Through OCR (optical character recognition), NLP (natural language processing) and image recognition technologies, it automatically extracts text, tables, images and audio content from various unstructured data sources such as pictures, scans, PDF documents, audio files, video files, etc.

[0098] S3. Define a unified data model, save the unstructured data after deep processing and knowledge extraction into the data model, intelligently classify and index the data in the data model, build a data security and privacy protection system, and securely store unstructured data;

[0099] Methods for intelligently classifying data include:

[0100] Intelligently classify the unstructured data after deep processing, and build an index based on the classification results; the intelligent classification includes: text classification, image classification, and audio classification;

[0101] The text classification includes: extracting and learning text features through machine learning algorithms: decision trees, achieving more accurate classification, understanding and processing natural language content in documents through natural language processing (NLP) technology, and extracting useful information from a large amount of text data for classification;

[0102] The image classification includes: converting the information in the image into a feature vector through a feature extraction algorithm SIFT, and performing classification prediction on the extracted feature vector through a classifier SVM, KNN, decision tree, random forest and neural network;

[0103] The audio classification includes: extracting the frequency spectrum features and Mel frequency cepstral coefficients (MFCC) of the audio as the basis for classification; and saving the classification results of the intelligent classification to the Metadata field of the corresponding information;

[0104] The method for indexing data includes: creating indexes for the fields in the ID, Type, and metadata of the data model for subsequent efficient retrieval. By building an efficient indexing mechanism, the data required by the user can be quickly located, greatly improving the user experience.

[0105] Methods for building a data security and privacy protection system include: data encryption, access control, audit tracking, and data desensitization;

[0106] The data encryption includes: using the advanced encryption algorithm AES to encrypt data transmission, encrypt sensitive information stored in the database, and ensure that the data is not stolen or tampered with by unauthorized third parties;

[0107] The access control includes: identity authentication, ensuring that the identity is legitimate before allowing access to data, verifying the user name, password, and biometric identity information, confirming the user's true identity, and preventing illegal users from accessing data; setting different access levels and permissions for different users, limiting the user's access to data and the scope of operation, and further ensuring data security;

[0108] The audit trail includes: monitoring and recording the data access process, recording the user's access behavior, operation history and abnormal situations, so as to facilitate investigation and tracking when security issues occur, so as to take timely measures;

[0109] The data desensitization includes: using desensitization algorithm replacement, masking, data obfuscation, and data simulation to deform the sensitive data and then transmit and store it to achieve the protection of sensitive privacy data.

[0110] By building a comprehensive data security and privacy protection system and establishing a sound security and compliance governance system, we can ensure the security and compliance of unstructured data during storage, processing, and transmission.

[0111] In this embodiment, the unified data model defined includes the following basic fields:

[0112] ID: A unique identifier used to uniquely identify each data item;

[0113] Type: data type, such as text, image, audio, video;

[0114] Content: The actual content of the unstructured data, which can be binary data or a URL that references the data;

[0115] Metadata: Metadata, which describes additional information about the data, such as creation time, author, description, tags, etc.

[0116] Source: data source information, such as the uploader of the data, upload time, etc.;

[0117] S4. Design a dynamic metadata management mechanism to update and maintain the metadata of unstructured data in real time to form a unified structured data view;

[0118] The method for designing a dynamic metadata management mechanism includes: dynamically generating metadata of unstructured data in the data model, and updating and maintaining the metadata in real time;

[0119] The metadata includes: creation time, author, description, tag, data source, data format, processing status, and access rights;

[0120] The real-time updating and maintenance of metadata includes: updating and maintaining the processing status and access rights of the metadata.

[0121] Preferably, a multi-dimensional integration framework is designed to integrate unstructured data from different sources and formats to form a structured data view. The framework supports pre-processing operations such as data cleaning, deduplication, and standardization to ensure data accuracy and consistency.

[0122] The metadata of unstructured data is updated and maintained in real time through the dynamic metadata management mechanism. As unstructured data continues to change and increase, metadata information is updated in a timely manner, and metadata information is added, modified or deleted to ensure the accuracy and timeliness of metadata. Through dynamic metadata management, the entire life cycle of data can be tracked in real time to ensure data security and compliance.

[0123] The method comprises storing unstructured data from different channels and of different types in the unified structured data model, adopting the unified data model as a data view, performing unified management and analysis of the unstructured data, and forming a unified structured data view.

[0124] S5. Extract entities, relationships, and attribute information from unstructured data through an intelligent recognition engine, build a knowledge graph, and display the knowledge graph;

[0125] Extracting key information through an intelligent recognition engine The intelligent recognition engine is an AI algorithm set consisting of a word segmentation algorithm, a part-of-speech tagging algorithm, a named entity recognition algorithm, a syntactic analysis algorithm, a semantic analysis algorithm, and a sentiment analysis algorithm. These AI algorithms are used to perform entity extraction, relationship extraction, and attribute extraction, and identify and extract entity, relationship, and attribute information in unstructured data;

[0126] Wherein, the entity extraction is to identify entities that meet the definition from unstructured data;

[0127] The relationship extraction is to extract the relationship triples that conform to the facts from the unstructured data, that is, to determine whether there is a certain semantic directed relationship between two entities;

[0128] The attribute extraction is to extract attribute information used to describe the characteristics of the entity.

[0129] The method of constructing a knowledge graph includes: storing the entities, relationships, attributes and other information in the identified and extracted unstructured data in the graph database Neo4j, and using Neo4j to provide efficient graph query and analysis capabilities; the method of displaying a knowledge graph includes: displaying the knowledge graph information stored in Neo4j in the form of a knowledge graph, displaying the associations between entities, and providing graph query and analysis capabilities; you can enter or select entities to query associations, and you can click on entities to continue drilling down to analyze the associations.

[0130] The key information extracted by the intelligent recognition engine is used to build a knowledge graph. The knowledge graph represents the relationship between entities in a graph structure, which can intuitively display the internal connection and hierarchy of the data. Through the knowledge graph, users can query and understand data more conveniently and discover potential business value.

[0131] S6. Through the intelligent search engine, the user's query intention is understood, relevant information is retrieved from the knowledge graph and data view, and rich contextual information is provided;

[0132] Understanding the user's query intention includes: using natural language processing (NLP) in the intelligent search engine to parse the query sentence input by the user, perform word segmentation, part-of-speech tagging, and syntactic analysis, analyze the keywords and phrases in the query and the relationship between them, and lay the foundation for semantic analysis; by understanding these keywords, phrases, and sentences, perform semantic analysis, and further analyze the user's query intention;

[0133] Retrieving relevant information from the knowledge graph and data view and providing rich contextual information includes: retrieving relevant information from the knowledge graph and the unified structured data view based on the understood query intent of the user; the contextual information is the surrounding environment or background information related to the query conditions input by the user when performing information retrieval. Providing rich contextual information helps to more accurately understand and parse the query intent of the user, thereby improving the accuracy and relevance of the query results.

[0134] The intelligent search engine supports content-based unstructured data retrieval, uses natural language processing, semantic analysis and other technologies to understand the user's query intent, and retrieves relevant information from the knowledge graph and integrated data sets. Not only does it improve the accuracy of search results, but it also provides rich contextual information to help users better understand the data.

[0135] S7, ecological integration and open API, providing rich API interfaces to achieve barrier-free flow and sharing of data with business systems, data platforms and analytical tools.

[0136] The method of providing a rich API interface includes: opening the knowledge graph and the unified structured data view to the outside world in the form of an API interface, allowing other business systems to authorize calling the interface to obtain data, and the authorized call includes: authorized calling user name, calling password, calling validity period and other information; by calling the API interface, the knowledge graph data and data view data can be directly queried; at the same time, detailed call logs are recorded to facilitate security audits. By defining a unified data model and data interface, barrier-free circulation and sharing of data is achieved.

[0137] Figure 1 The basic process of managing unstructured data by the data center in this embodiment is shown.

[0138] The embodiment of the present invention further provides a system for managing unstructured data by a data middle platform, and executes the method for managing unstructured data by a data middle platform as described above, including:

[0139] Capture unstructured data module: used to access multi-source heterogeneous data, configure various data source information, connect to the target data source corresponding to the multi-source heterogeneous data according to the configured data source information, and capture the unstructured data of the target data source;

[0140] Unstructured data recognition module: used to recognize the unstructured data through an intelligent recognition engine, and perform in-depth processing and knowledge extraction on the unstructured data;

[0141] Secure storage of unstructured data module: used to define a unified data model, save the unstructured data after deep processing and knowledge extraction into the data model, intelligently classify and index the data in the data model, build a data security and privacy protection system, and securely store unstructured data;

[0142] Forming a structured data view module: used to design a dynamic metadata management mechanism to update and maintain the metadata of unstructured data in real time and form a unified structured data view;

[0143] Knowledge graph construction module: used to extract entities, relationships, and attribute information from unstructured data through an intelligent recognition engine, construct a knowledge graph, and display the knowledge graph;

[0144] Retrieval information module: used to understand the user's query intent through the intelligent retrieval engine, retrieve relevant information from the knowledge graph and data view, and provide rich contextual information;

[0145] Data circulation and sharing module: used for ecological integration and open API, providing rich API interfaces to achieve barrier-free circulation and sharing of data with business systems, data platforms and analysis tools.

[0146] The method and system for managing unstructured data by the data middle platform of this embodiment realizes the comprehensive integration, efficient processing, secure storage and intelligent analysis of unstructured data from different systems and devices by building a stable and scalable data middle platform architecture, breaking the data island phenomenon and improving the efficiency of data sharing and collaborative work; integrating multiple intelligent processing technologies, including NLP, image recognition, speech recognition, etc., to realize deep mining and analysis of unstructured data, improve the speed and accuracy of data processing, and reduce the cost and error rate of manual intervention; establishing a complete data security management system, and ensuring the security and compliance of unstructured data during transmission, storage and processing through means such as data encryption, access control and audit tracking; forming a structured data view, providing intuitive and easy-to-use tools, reducing the technical threshold for unstructured data processing and analysis, and enabling enterprises to use these data resources more conveniently; building an enterprise knowledge graph, realizing the extraction, reuse and intelligent search functions of knowledge and information in unstructured data.

[0147] An embodiment of the present invention further provides a computer device, Figure 3 is a schematic diagram of the structure of a computer device provided by an embodiment of the present invention; see the accompanying drawings Figure 3 As shown, the computer device includes: an input system 23, an output system 24, a memory 22 and a processor 21; the memory 22 is used to store one or more programs; when the one or more programs are executed by the one or more processors 21, the one or more processors 21 implement the method for managing unstructured data in the data middle platform provided in the above embodiment; wherein the input system 23, the output system 24, the memory 22 and the processor 21 can be connected by a bus or other means, Figure 3 The example of connecting through bus is taken in the following.

[0148] The memory 22 is a readable and writable storage medium of a computing device, which can be used to store software programs, computer executable programs, such as the program instructions corresponding to the method for managing unstructured data in the data middle platform described in the embodiment of the present invention; the memory 22 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required for a function; the data storage area can store data created according to the use of the device, etc.; in addition, the memory 22 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device; in some instances, the memory 22 can further include a memory remotely arranged relative to the processor 21, and these remote memories can be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0149] The input system 23 may be used to receive input digital or character information, and to generate key signal input related to user settings and function control of the device; the output system 24 may include display devices such as display screens.

[0150] The processor 21 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory 22, that is, implements the above-mentioned method of managing unstructured data by the data middle platform.

[0151] The computer device provided above can be used to execute the method for managing unstructured data at the data center provided in the above embodiment, and has corresponding functions and beneficial effects.

[0152] The embodiment of the present invention also provides a storage medium containing computer executable instructions, which are used to execute the method of managing unstructured data in the data center provided by the above embodiment when executed by a computer processor. The storage medium is any of various types of memory devices or storage devices, and the storage medium includes: installation media, such as CD-ROM, floppy disk or tape system; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disk or optical storage); registers or other similar types of memory elements, etc.; the storage medium may also include other types of memory or combinations thereof; in addition, the storage medium may be located in a first computer system in which the program is executed, or may be located in a different second computer system, which is connected to the first computer system via a network (such as the Internet); the second computer system may provide program instructions to the first computer for execution. The storage medium includes two or more storage media that can reside in different locations (for example, in different computer systems connected via a network). The storage medium can store program instructions (for example, specifically implemented as a computer program) that can be executed by one or more processors.

[0153] Of course, the storage medium containing computer executable instructions provided in an embodiment of the present invention, whose computer executable instructions are not limited to the method for managing unstructured data by a data middle platform as described in the above embodiment, can also execute related operations in the method for managing unstructured data by a data middle platform provided in any embodiment of the present invention.

[0154] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments, but it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0155] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for managing unstructured data in a data center, characterized in that: The following steps are involved: S1. Access multi-source heterogeneous data, configure various data source information, connect to the target data source corresponding to the multi-source heterogeneous data according to the configured data source information, and capture the unstructured data of the target data source; S2. Identify the unstructured data through an intelligent recognition engine, and perform deep processing and knowledge extraction on the unstructured data; S3. Define a unified data model, save the unstructured data after deep processing and knowledge extraction into the data model, intelligently classify and index the data in the data model, build a data security and privacy protection system, and securely store unstructured data; S4. Design a dynamic metadata management mechanism to update and maintain the metadata of unstructured data in real time to form a unified structured data view; S5. Extract entities, relationships, and attribute information from unstructured data through an intelligent recognition engine, build a knowledge graph, and display the knowledge graph; S6. Through the intelligent search engine, the user's query intention is understood, relevant information is retrieved from the knowledge graph and data view, and rich contextual information is provided; S7, ecological integration and open API, providing rich API interfaces to achieve barrier-free flow and sharing of data with business systems, data platforms and analytical tools.

2. The method for managing unstructured data by a data center according to claim 1, characterized in that: The method for configuring various data source information in step S1 includes: configuring an acquisition method for acquiring unstructured data from various data sources, wherein the acquisition method includes: an API interface method, a data crawling method, and a file upload method; The API interface method includes: configuring the API interface address, port, return data type, and authentication method information; The data capture method includes: configuring the capture target address, whether there is authentication, and authentication method information; The file upload method includes: configuring the target file location, login username, and password information.

3. The method for managing unstructured data by a data center according to claim 1, characterized in that: The method for deep processing of unstructured data in step S2 includes: For different types of unstructured data, corresponding processing technologies are used for deep processing. The deep processing includes: cleaning and converting unstructured data; the cleaning includes: removing duplicate data, format conversion, error correction, and ensuring the effectiveness and accuracy of subsequent processing; the conversion includes: converting unstructured data into structured or semi-structured forms to facilitate subsequent analysis and application; The method for extracting knowledge from unstructured data includes: for text data, using NLP technology to extract key information of keywords, entities, and emotions; for image data, using image recognition technology to identify the features of objects, scenes, and colors in the image; for audio data, extracting audio content through speech recognition and emotion analysis technology; for audio and video data, extracting key frames and scene information through video analysis technology.

4. The method for managing unstructured data by a data center according to claim 1, characterized in that: The method for intelligently classifying data in step S3 includes: Intelligently classify the unstructured data after deep processing, and build an index based on the classification results; the intelligent classification includes: text classification, image classification, and audio classification; The text classification includes: extracting and learning text features through machine learning algorithms: decision tree, support vector machine SVM, random forest, AdaBoost, LightGBM or XGBoost to achieve more accurate classification, understanding and processing natural language content in documents through natural language processing (NLP) technology, and extracting useful information from a large amount of text data for classification; The image classification includes: converting the information in the image into a feature vector through a feature extraction algorithm such as SIFT, SURF or HOG, and performing classification prediction on the extracted feature vector through a classifier such as SVM, KNN, decision tree, random forest and neural network; The audio classification includes: extracting the frequency spectrum features and Mel frequency cepstral coefficients (MFCC) of the audio as the basis for classification; and saving the classification results of the intelligent classification to the Metadata field of the corresponding information; The method for indexing data in the S3 step includes: creating indexes for the fields in the ID, Type, and metadata of the data model to facilitate subsequent efficient retrieval.

5. The method for managing unstructured data by a data middle platform according to claim 1, characterized in that: The method of building a data security and privacy protection system in step S3 includes: data encryption, access control, audit tracking and data desensitization; The data encryption includes: using advanced encryption algorithms such as AES, RSA, SM2, and SM4 to encrypt data transmission and encrypt sensitive information stored in the database to ensure that the data is not stolen or tampered with by unauthorized third parties; The access control includes: identity authentication, ensuring that the identity is legitimate before allowing access to data, verifying the user name, password, and biometric identity information, confirming the user's true identity, and preventing illegal users from accessing data; setting different access levels and permissions for different users, limiting the user's access to data and the scope of operation, and further ensuring data security; The audit trail includes: monitoring and recording the data access process, recording the user's access behavior, operation history and abnormal situations, so as to facilitate investigation and tracking when security issues occur, so as to take timely measures; The data desensitization includes: using desensitization algorithm replacement, masking, data obfuscation, and data simulation to deform sensitive data and then transmit and store it to achieve the protection of sensitive privacy data.

6. The method for managing unstructured data by a data middle platform according to claim 1, characterized in that: The method for designing a dynamic metadata management mechanism in step S4 includes: dynamically generating metadata of the unstructured data in the data model, and updating and maintaining the metadata in real time; The metadata includes: creation time, author, description, tag, data source, data format, processing status, and access rights; The real-time updating and maintenance of metadata includes: updating and maintaining the processing status and access rights of the metadata.

7. The method for managing unstructured data by a data center according to claim 1, characterized in that: The intelligent recognition engine in step S5 of extracting key information through an intelligent recognition engine is an AI algorithm set consisting of a word segmentation algorithm, a part-of-speech tagging algorithm, a named entity recognition algorithm, a syntactic analysis algorithm, a semantic analysis algorithm, and a sentiment analysis algorithm. These AI algorithms are used to perform entity extraction, relationship extraction, and attribute extraction, and identify and extract entity, relationship, and attribute information in unstructured data; Wherein, the entity extraction is to identify entities that meet the definition from unstructured data; The relationship extraction is to extract the relationship triples that conform to the facts from the unstructured data, that is, to determine whether there is a certain semantic directed relationship between two entities; The attribute extraction is to extract attribute information used to describe the characteristics of the entity.

8. A system for managing unstructured data in a data center, characterized in that: The method for managing unstructured data by a data middle platform according to any one of claims 1 to 7 comprises: Capture unstructured data module: used to access multi-source heterogeneous data, configure various data source information, connect to the target data source corresponding to the multi-source heterogeneous data according to the configured data source information, and capture the unstructured data of the target data source; Unstructured data recognition module: used to recognize the unstructured data through an intelligent recognition engine, and perform in-depth processing and knowledge extraction on the unstructured data; Secure storage of unstructured data module: used to define a unified data model, save the unstructured data after deep processing and knowledge extraction into the data model, intelligently classify and index the data in the data model, build a data security and privacy protection system, and securely store unstructured data; Forming a structured data view module: used to design a dynamic metadata management mechanism to update and maintain the metadata of unstructured data in real time and form a unified structured data view; Knowledge graph construction module: used to extract entities, relationships, and attribute information from unstructured data through an intelligent recognition engine, construct a knowledge graph, and display the knowledge graph; Retrieval information module: used to understand the user's query intent through the intelligent retrieval engine, retrieve relevant information from the knowledge graph and data view, and provide rich contextual information; Data circulation and sharing module: used for ecological integration and open API, providing rich API interfaces to achieve barrier-free circulation and sharing of data with business systems, data platforms and analysis tools.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method for managing unstructured data by a data center as described in any one of claims 1 to 7 are implemented.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method for managing unstructured data by a data middle platform as described in any one of claims 1-7 are implemented.

Images