Multi-source heterogeneous data acquisition and storage method and system, terminal and medium

Abstract

The invention discloses a multi-source heterogeneous data collection and storage method and system, a terminal and a medium, and the method comprises the steps: carrying out the collection of data from various data sources through an adapter and task scheduling according to a collection strategy, and obtaining the collection data; preprocessing the collected data to obtain preprocessed data; and classifying the preprocessed data based on data types and access characteristics, and constructing a cache layer, a structured storage layer and an unstructured storage layer according to a classification result so as to store the preprocessed data. The method comprises the following steps of: acquiring data from a multi-source heterogeneous data source according to an acquisition strategy through an adapter and task scheduling; after the collected data is preprocessed, a cache layer, a structured storage layer and an unstructured storage layer are constructed according to the data type and access characteristics of the data, so that high efficiency, flexibility and expandability of data collection and storage are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data collection and storage, and particularly relates to a multi-source heterogeneous data collection and storage method, system, terminal and medium. BACKGROUND

[0002] In the era of big data, with the rapid development of information technology and the continuous expansion of various application scenarios, data presents the characteristics of wide sources, various formats and rapid growth. Multi-source heterogeneous data, such as structured database data, semi-structured log files, unstructured text and multimedia data, widely exists in finance, medical treatment, communication, transportation and other fields. These data are often generated by different collection devices, business systems or network protocols, and have differentiated format standards and storage methods, which brings great challenges to unified data collection and centralized management. The traditional data processing architecture exposes many problems when facing such complex data: on the one hand, due to the lack of flexible data interfaces and standardized collection mechanisms, it is difficult to efficiently converge data from different sources; on the other hand, the traditional storage system is usually optimized for a specific type of data, and it is difficult to adapt to the unified storage needs of diversified data structures, which easily causes problems such as data redundancy, poor consistency and low storage efficiency.

[0003] Therefore, the prior art still needs to be improved and improved. SUMMARY

[0004] The technical problem to be solved by the present application is that, in view of the above defects of the prior art, a multi-source heterogeneous data collection and storage method, system, terminal and medium are provided, aiming at solving the problems of poor interface compatibility, insufficient format standardization, low collection efficiency and serious storage redundancy in the process of collecting and storing multi-source heterogeneous data by the traditional data processing architecture.

[0005] In order to solve the above technical problems, the technical scheme adopted by the present application is as follows:

[0006] In a first aspect, the present application provides a multi-source heterogeneous data collection and storage method, wherein the method comprises:

[0007] Collecting data from various data sources according to a collection strategy through an adapter and task scheduling to obtain collected data;

[0008] Preprocessing the collected data to obtain preprocessed data;

[0009] Classifying the preprocessed data based on data types and access characteristics, and constructing a cache layer, a structured storage layer and an unstructured storage layer according to the classification results to store the preprocessed data.

[0010] In an implementation mode, the data collection from various data sources according to the collection strategy through the adapter and the task scheduling, to obtain the collection data, includes:

[0011] Developing the data source adapter based on actual needs;

[0012] According to the data source adapter, the configuration of the data source information is performed to obtain the configuration information;

[0013] Based on the configuration information, the data collection strategy is defined;

[0014] According to the collection strategy, the task scheduling of data collection is set.

[0015] In an implementation mode, the preprocessing of the collection data to obtain the preprocessing data includes:

[0016] The collection data is cleaned to remove duplicate, error or invalid data to obtain clean data after preliminary cleaning;

[0017] The clean data is converted to a standard format to obtain standardized data with unified structure;

[0018] Based on the key field, the standardized data is associated and fused to obtain the preprocessing data.

[0019] In an implementation mode, the preprocessing data is classified based on data type and access characteristics, including:

[0020] The preprocessing data is information extracted to obtain the data type and access characteristics of the preprocessing data;

[0021] Based on the data type and the access characteristics, the preprocessing data is divided into high-frequency access data, low-frequency access data and unstructured data.

[0022] In an implementation mode, the cache layer, the structured storage layer and the unstructured storage layer are constructed according to the classification results, including:

[0023] The storage requirement analysis of the classification results is performed to determine the storage characteristics of the high-frequency access data, the low-frequency access data and the unstructured data;

[0024] Based on the storage characteristics of the high-frequency access data, the cache layer for fast read-write operation is constructed;

[0025] Based on the storage characteristics of the low-frequency access data, the structured storage layer for persistent storage is constructed;

[0026] Based on the storage characteristics of the unstructured data, the unstructured storage layer for large file storage is constructed.

[0027] In an implementation manner, the storing of the preprocessed data comprises:

[0028] configuring a unified data routing strategy;

[0029] establishing a cross-level data synchronization mechanism and a unified access interface according to the data routing strategy;

[0030] based on the synchronization mechanism and the access interface, guiding different types of preprocessed data to corresponding storage layers for storage and management.

[0031] In an implementation manner, the storing of the preprocessed data comprises:

[0032] establishing a database index for high-frequency access data and low-frequency access data;

[0033] constructing an inverted index for text type unstructured data;

[0034] extracting a feature vector for multimedia type unstructured data, and establishing an index structure based on the feature vector and using a similarity matching algorithm.

[0035] In a second aspect, the embodiments of the present application further provide a multi-source heterogeneous data collection and storage system, wherein the system comprises:

[0036] a data acquisition module, configured to acquire data from various data sources according to a collection strategy through an adapter and task scheduling, to obtain collected data;

[0037] a preprocessed data acquisition module, configured to preprocess the collected data to obtain preprocessed data;

[0038] a storage module, configured to classify the preprocessed data based on data types and access characteristics, and to construct a cache layer, a structured storage layer and an unstructured storage layer according to the classification results, to store the preprocessed data.

[0039] In a third aspect, the embodiments of the present application further provide a terminal, wherein the terminal comprises a memory, a processor, and a multi-source heterogeneous data collection and storage program stored in the memory and executable on the processor, and when the processor executes the multi-source heterogeneous data collection and storage program, the steps of the multi-source heterogeneous data collection and storage method in any of the above solutions are implemented.

[0040] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, wherein the computer readable storage medium stores a multi-source heterogeneous data collection and storage program, and the multi-source heterogeneous data collection and storage program is executed by a processor to implement the steps of the multi-source heterogeneous data collection and storage method in any of the above solutions.

[0041] Beneficial effects: The present application provides a multi-source heterogeneous data collection and storage method. Compared with the prior art, the present application first collects data from various data sources according to a collection strategy through an adapter and task scheduling to obtain collected data; then, the collected data is preprocessed to obtain preprocessed data. This step effectively removes noise and invalid information through cleaning, format conversion, deduplication, standardization and other operations on the collected raw data, improves data quality, lays a good foundation for subsequent data processing and analysis, and enhances the system's ability to guarantee data consistency and usability; then, the preprocessed data is classified based on data types and access characteristics, and a cache layer, a structured storage layer and an unstructured storage layer are constructed according to the classification results for storing the preprocessed data. This step realizes fine management and efficient storage of data, and selects the most suitable storage mode according to different data usage scenarios: the cache layer is used to accelerate the response speed of high-frequency access data, the structured storage layer is suitable for efficient query and management of relational data, and the unstructured storage layer is used for flexible storage of complex data such as text and images, thereby improving the overall storage efficiency, access performance and scalability of the system. Overall, the present application constructs a complete data processing flow from data collection, preprocessing to intelligent layered storage through the above three steps, fully utilizes the advantages of the adapter and task scheduling mechanism in multi-source data access, and combines data preprocessing and layered storage strategies to comprehensively improve the data processing efficiency, storage flexibility and access performance of the system, and has good engineering practicability and technical popularization value. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 The flowchart of the specific implementation of the multi-source heterogeneous data collection and storage method provided by the embodiments of the present application.

[0043] Figure 2 The flowchart of obtaining preprocessed data in the multi-source heterogeneous data collection and storage method provided by the embodiments of the present application.

[0044] Figure 3 The flowchart of storing and managing preprocessed data in the multi-source heterogeneous data collection and storage method provided by the embodiments of the present application.

[0045] Figure 4is a principle block diagram of a multi-source heterogeneous data collection and storage system provided by an embodiment of the present application.

[0046] Figure 5 is a principle block diagram of an internal structure of a terminal provided by an embodiment of the present application. DETAILED DESCRIPTION

[0047] To make the objectives, technical solutions and effects of the present application clearer and more explicit, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0048] In the era of big data, with the rapid development of information technology and the continuous expansion of various application scenarios, data presents the characteristics of wide sources, diverse formats and rapid growth. Multi-source heterogeneous data, such as structured database data, semi-structured log files, unstructured text and multimedia data, etc., widely exist in finance, medical treatment, communication, transportation and other fields. These data are often generated by different collection devices, business systems or network protocols, have differentiated format standards and storage methods, and bring great challenges to unified data collection and centralized management. Traditional data processing architecture exposes many problems when facing such complex data: on the one hand, due to the lack of flexible data interface and standardized collection mechanism, different source data is difficult to efficiently converge; on the other hand, traditional storage systems are usually optimized for specific types of data, and are difficult to adapt to the unified storage needs of diversified data structures, which easily causes problems such as data redundancy, poor consistency and low storage efficiency.

[0049] To solve the above problems, the embodiment provides a multi-source heterogeneous data collection and storage method. In specific implementation, the embodiment first collects data from various data sources according to a collection strategy through an adapter and task scheduling to obtain collected data; then, the collected data is preprocessed to obtain preprocessed data. This step effectively removes noise and invalid information through cleaning, format conversion, deduplication, standardization and other operations on the collected raw data, improves data quality, lays a good foundation for subsequent data processing and analysis, and enhances the system's guarantee ability for data consistency and availability; then, the preprocessed data is classified based on data types and access characteristics, and a cache layer, a structured storage layer and an unstructured storage layer are constructed according to the classification results to store the preprocessed data. This step realizes fine management and efficient storage of data, and selects the most suitable storage mode according to different data usage scenarios: the cache layer is used to accelerate the response speed of high-frequency access data, the structured storage layer is suitable for efficient query and management of relational data, and the unstructured storage layer is used for flexible storage of complex data such as text and image. Therefore, the overall storage efficiency, access performance and scalability of the system are improved. Overall, the present application constructs a complete data processing flow from data collection, preprocessing to intelligent layered storage through the above three steps, fully utilizes the advantages of the adapter and task scheduling mechanism in multi-source data access, and combines data preprocessing and layered storage strategy to realize comprehensive improvement of the system in data processing efficiency, storage flexibility and access performance, and has good engineering practicability and technical popularization value.

[0050] The multi-source heterogeneous data collection and storage method provided by the embodiment can be applied to a smart terminal, such as a smart phone. Figure 1 The method comprises the following steps:

[0051] In step S100, data is collected from various data sources according to a collection strategy through an adapter and task scheduling to obtain collected data.

[0052] In the embodiment, first, based on actual demand, a data source adapter is developed, the data source adapter includes a database adapter, a file system adapter, a web crawler adapter and a sensor adapter and the like, each adapter is designed to optimize a specific type of data source, can automatically identify and adapt the format, structure and communication protocol of the data source, thereby improving the compatibility and access efficiency of the system to heterogeneous data sources. Next, according to the data source adapter, the data source information is configured to obtain the configuration information, the data source information includes specific data source connection parameters and access paths, such as the host address, port number, username, password of the database, the file path and format of the file system, and the target website and crawling rules of the web crawler, etc., to ensure that the collection task has the basic conditions for accurate execution, and enhances the flexibility and maintainability of the system configuration. Then, based on the configuration information, the data collection strategy is defined, this step is based on the configuration information of the data source, and the key strategy parameters such as collection frequency, data filtering conditions and collection range are determined, so that the collection process can be adjusted as needed, improving the pertinence and timeliness of data acquisition, meeting the needs of diversified business scenarios. Subsequently, according to the collection strategy, the task scheduling of data collection is set, this step sets the triggering mode and execution time of the collection task according to the collection strategy, supports timed tasks (such as execution at 2 o'clock every morning), periodic tasks (such as execution once an hour) or real-time tasks (triggered by a specific event), thereby realizing the automation and fine control of task execution, improving resource utilization and the orderliness of task execution. Finally, through the adapter and task scheduling, the data is collected from various data sources according to the collection strategy to obtain the collected data. In this stage, each adapter starts the collection process according to the configured parameters and strategies and the task scheduling mechanism, for example, the database adapter extracts data from the relational database through SQL query, and the web crawler adapter extracts content from the web page according to the rules, thereby realizing stable, reliable and automated large-scale data acquisition, and guaranteeing the quality and integrity of the collection results. In summary, the entire process realizes the unified access and intelligent collection of multi-source heterogeneous data through phased design. The logic between each step is clear and closely connected, from adapter development to configuration management, strategy formulation, task scheduling to the final data collection execution, which effectively improves the adaptability, running efficiency and management convenience of the data collection system, and provides high-quality and sustainable data support for subsequent data processing, analysis and application.

[0053] Step S200, preprocessing the collected data to obtain preprocessed data.

[0054] In the embodiment, firstly, the collected data is preprocessed to obtain preprocessed data; the step effectively removes noise interference, improves data quality, and ensures that the data relied on in the subsequent analysis and processing process has higher accuracy, consistency and integrity. The cleaning process can eliminate invalid or redundant data, the standardization operation unifies the representation form of data from different sources, and the missing value and abnormal value processing enhances the integrity and reliability of the data set, laying a solid foundation for constructing a high-quality data analysis model.

[0055] Specifically, the step S200 includes the following steps:

[0056] Step S201, cleaning the collected data to remove repeated, erroneous or invalid data to obtain clean data after preliminary cleaning;

[0057] Step S202, performing format standardization conversion on the clean data to obtain standardized data with unified structure;

[0058] Step S203, associating and fusing the standardized data based on key fields to obtain preprocessed data.

[0059] In one implementation manner, as Figure 2As shown, first, the collected data is cleaned to remove duplicate, incorrect or invalid data, obtaining clean data after preliminary cleaning. This step is the basic link of the whole data preprocessing process, and the core purpose is to improve the accuracy and integrity of the data. In actual operation, data cleaning systematically identifies and processes problems such as missing values, outliers, inconsistent formats and duplicate records in the original data set, such as deleting records with missing key fields (such as user identification, timestamp, etc.), eliminating redundant duplicate entries, correcting illegal characters, logical errors or data content beyond the reasonable range, thereby effectively reducing the interference that noise data may cause to the subsequent analysis process, and ensuring that the data entering the next stage has high reliability and consistency. Then, the clean data is standardized and converted in format, obtaining standardized data with uniform structure. This step aims to solve the differences in format, unit, coding, etc. of multi-source heterogeneous data, and by unifying various data to a consistent representation, the standardization and operability of the data are enhanced. Specifically, date and time fields are unified to a standard format (such as "YYYY-MM-DD HH:MM:SS"), text character sets are unified to UTF-8 or other universal encoding, and numerical fields are unit-converted, precision-controlled and format-standardized, thereby improving the compatibility of data between different systems and providing good foundation support for subsequent storage, query and analysis. Then, the standardized data is associated and fused based on key fields, obtaining preprocessed data, which is a key step to realize multi-source data integration and improve data semantic integrity. By identifying key fields (such as user ID, product ID, order number, etc.) with associated significance in each data set, data records from different business systems or data sources are effectively connected to form a more comprehensive and more business valuable integrated view. For example, user behavior data on multiple platforms can be aggregated to build a complete user portrait, or product information can be associated with inventory, sales, evaluation and other dimension data to generate an integrated product data model, thereby significantly improving the usability and analysis depth of the data. In summary, through the three-stage processing of data cleaning, format standardization and association fusion, not only the data quality, structural consistency and semantic expression ability are effectively improved, but also the interoperability and integration capability of the data in cross-system and cross-platform scenarios are enhanced, providing high-quality, structured and semantically clear data support for subsequent data processing, which has important theoretical research significance and wide application practical value.

[0060] Step S300, classifying the preprocessed data based on data types and access characteristics, and constructing a cache layer, a structured storage layer and an unstructured storage layer according to the classification results for storing the preprocessed data.

[0061] In this embodiment, first, the pre-processed data is classified based on data types and access characteristics. Then, a cache layer, a structured storage layer and an unstructured storage layer are constructed according to the classification results for storage of the pre-processed data. This embodiment reasonably divides data into different types of storage levels by analyzing the format characteristics (such as text, image, structured table, etc.) and access frequency (such as high-frequency real-time access or low-frequency batch processing) of the data. Among them, the cache layer is used to store data with high access frequency and sensitive response time, which improves the system response speed and concurrent processing capability; the structured storage layer is used for data with fixed format and relationship characteristics, and a database mechanism is used to realize efficient query and transaction processing; the unstructured storage layer is suitable for complex format data such as pictures, videos and logs, and object storage or distributed file system is used to guarantee the integrity and scalability. This hierarchical storage architecture not only optimizes the data access efficiency, but also effectively reduces the system resource consumption, improves the flexibility and scalability of the storage system. In summary, by classifying the pre-processed data according to its type and access characteristics and constructing a corresponding multi-level storage system, the fine management of the data storage strategy is realized, and the overall performance and resource utilization of the system are improved.

[0062] Specifically, the step S300 comprises the following steps:

[0063] Step S301, information extraction is performed on the pre-processed data to obtain data types and access characteristics of the pre-processed data;

[0064] Step S302, based on the data types and the access characteristics, the pre-processed data is divided into high-frequency access data, low-frequency access data and unstructured data;

[0065] Step S303, storage requirement analysis is performed on the classification results to determine the storage characteristics of the high-frequency access data, the low-frequency access data and the unstructured data;

[0066] Step S304, based on the storage characteristics of the high-frequency access data, a cache layer for fast read-write operation is constructed;

[0067] Step S305, based on the storage characteristics of the low-frequency access data, a structured storage layer for persistent storage is constructed;

[0068] Step S306, based on the storage characteristics of the unstructured data, an unstructured storage layer for large file storage is constructed;

[0069] Step S307, a unified data routing strategy is configured;

[0070] Step S308, according to the data routing strategy, a cross-level data synchronization mechanism and a unified access interface are established;

[0071] Step S309, based on the synchronization mechanism and the access interface, direct different types of pre-processed data to corresponding storage layers for storage and management.

[0072] In one implementation, as Figure 3As shown, first, information extraction is performed on the preprocessed data to obtain the data type and access characteristics of the preprocessed data. This step provides accurate basis for subsequent classification and storage strategy through analysis of the content, format, structure and access frequency of the preprocessed data, ensuring the scientificity and pertinence of the entire system design. Then, based on the data type and access characteristics, the preprocessed data is divided into high-frequency access data, low-frequency access data and unstructured data. This classification process helps to develop differentiated storage strategies according to different data usage scenarios and access patterns, thereby improving the overall performance and resource utilization of the system. Subsequently, storage requirement analysis is performed on the classified results to determine the storage characteristics of high-frequency access data, low-frequency access data and unstructured data. Through in-depth analysis of the differences in read-write frequency, real-time requirements, data size and scalability requirements of various data, the different demands of each type of data on storage performance, reliability and cost are clarified, providing technical support for building a reasonable storage hierarchy. Next, based on the storage characteristics of high-frequency access data, a cache layer for fast read-write operations is constructed, such as using a memory database like Redis. The real-time sales data of popular products, user session information and other frequently accessed content are placed in the cache, which can significantly improve data read speed and system response efficiency, while effectively reducing the pressure on the backend database, suitable for high-concurrency access scenarios. Then, based on the storage characteristics of low-frequency access data, a structured storage layer for persistent storage is constructed, such as using MySQL, Oracle and other relational databases. It can efficiently store and manage data with fixed format and structure, such as user information table, order table and product table. Through reasonable table structure design and index optimization, query efficiency and transaction processing capacity are improved, ensuring data integrity and consistency. Next, based on the storage characteristics of unstructured data, an unstructured storage layer for large file storage is constructed, such as using HDFS and other distributed file systems. It can realize reliable storage and efficient access of unstructured data such as images, audio, video and documents, with good horizontal scalability, suitable for long-term storage and processing of massive unstructured data. Subsequently, a unified data routing strategy is configured. This strategy dynamically determines which layer the data should be directed to based on the business attributes, access path and storage hierarchy of the data, thereby realizing intelligent scheduling and load balancing of data and improving the flexibility and adaptability of the overall architecture.Next, according to the data routing strategy, a cross-level data synchronization mechanism and a unified access interface are established, by constructing a unified API interface and a data synchronization channel, transparent access and automatic flow of data between storage layers can be realized, consistency and availability of data between different levels are guaranteed, system complexity and development and maintenance cost are reduced; then, based on the synchronization mechanism and the access interface, different types of preprocessed data are guided to the corresponding storage layer for storage and management, this step realizes a complete closed loop from data recognition to storage landing, the entire system has the ability of on-demand allocation and automatic scheduling, the overall performance and stability of the system are improved, and the needs of future business growth and data scale expansion are supported. In summary, the embodiment builds a multi-level collaborative storage architecture including a cache layer, a structured storage layer and an unstructured storage layer, and combines a unified data routing strategy and a cross-level access interface, realizing the classified management and efficient storage of high-frequency access data, low-frequency access data and unstructured data, effectively improving the response speed, expansion capability and data management flexibility of the system, while reducing storage costs.

[0073] In an implementation manner, after the storage of the pre-processed data, in order to further improve the retrieval efficiency and query ability of the system, corresponding indexes and retrieval mechanisms need to be established for different types of data. Specifically, for structured data such as high-frequency access data and low-frequency access data, by creating traditional database indexes (such as B-tree indexes), the keyword-based query speed can be significantly improved, the amount of data scanned by the database can be reduced, and thus the system response time can be optimized; for example, establishing a primary key index for user ID or order number in the user information table or order table can realize millisecond-level data positioning and enhance the database query performance. At the same time, for text unstructured data (such as log files, article content, comment information, etc.), an inverted index technology is used for index construction, which can realize fast keyword retrieval of massive text content and greatly improve the search efficiency and usability of unstructured text; for example, using Elasticsearch to build a full-text search engine to provide real-time keyword search function for news websites, so that users can quickly find related articles. For multimedia unstructured data such as images, audio, and video, an efficient index structure is established through feature vector extraction and similarity matching algorithm (such as KNN, Faiss, etc.) to realize content-based retrieval function; for example, using convolutional neural network (CNN) to extract color, texture or semantic features of images, and storing these high-dimensional feature vectors in a vector database, when a user uploads a picture for search, the system can quickly match the visually similar image results by calculating the similarity between feature vectors, meeting the needs of image recognition, copyright detection, and image search. Overall, the advantages of this index and retrieval mechanism are: first, it improves the access efficiency of various types of data, especially when facing large-scale data sets, it can still maintain good response performance; second, it enhances the intelligent service ability of the system, supports diversified retrieval needs from structured to unstructured data; third, it realizes efficient management and collaborative query of different data types on a unified platform, improves the discoverability and operability of data. Through the implementation of hierarchical index strategy, not only the utilization efficiency of storage and computing resources is optimized, but also a solid data support is provided for subsequent intelligent analysis and business application.

[0074] In summary, the embodiment first collects data from various data sources according to a collection strategy through adapter and task scheduling to obtain collected data; then, the collected data is preprocessed to obtain preprocessed data, which effectively removes noise and invalid information through cleaning, format conversion, deduplication, standardization and other operations on the collected raw data, improves data quality, lays a good foundation for subsequent data processing and analysis, and enhances the system's ability to ensure data consistency and availability; then, the preprocessed data is classified based on data types and access characteristics, and a cache layer, a structured storage layer and an unstructured storage layer are constructed according to the classification results for storing the preprocessed data. This step realizes fine management and efficient storage of data, and selects the most suitable storage mode according to different data usage scenarios: the cache layer is used to accelerate the response speed of high-frequency access data, the structured storage layer is suitable for efficient query and management of relational data, and the unstructured storage layer is used for flexible storage of complex data such as text and images, thereby improving the overall storage efficiency, access performance and scalability of the system. Overall, the application constructs a complete data processing flow from data collection, preprocessing to intelligent layered storage through the above three steps, fully utilizes the advantages of adapter and task scheduling mechanism in multi-source data access, combines data preprocessing and layered storage strategy, realizes the overall improvement of the system in data processing efficiency, storage flexibility and access performance, and has good engineering practicability and technical popularization value.

[0075] As shown in Figure 4 The embodiment also provides a multi-source heterogeneous data collection and storage system, which comprises a collected data acquisition module 10, a preprocessed data acquisition module 20 and a preprocessed data acquisition module 30. Specifically, the collected data acquisition module 10 is used to collect data from various data sources according to a collection strategy through adapter and task scheduling to obtain collected data. The preprocessed data acquisition module 20 is used to preprocess the collected data to obtain preprocessed data. The storage module 30 is used to classify the preprocessed data based on data types and access characteristics, and construct a cache layer, a structured storage layer and an unstructured storage layer according to the classification results for storing the preprocessed data.

[0076] In an implementation manner, the preprocessed data acquisition module 20 comprises:

[0077] A clean data acquisition unit is configured to clean the collected data to remove duplicate, erroneous or invalid data, and obtain clean data after preliminary cleaning;

[0078] A standardized data acquisition unit is configured to perform format standardization conversion on the clean data to obtain standardized data with unified structure.

[0079] A preprocessing data acquisition unit is configured to perform associated fusion on the standardized data based on the key field to obtain preprocessing data.

[0080] In an implementation manner, the storage module 30 comprises:

[0081] A data type and access characteristic acquisition unit is configured to perform information extraction on the preprocessing data to obtain data types and access characteristics of the preprocessing data.

[0082] A division unit is configured to divide the preprocessing data into high-frequency access data, low-frequency access data and unstructured data based on the data types and the access characteristics.

[0083] A storage characteristic determination unit is configured to perform storage demand analysis on the classified results to determine storage characteristics of the high-frequency access data, the low-frequency access data and the unstructured data.

[0084] A cache layer construction unit is configured to construct a cache layer for fast read-write operation based on the storage characteristics of the high-frequency access data.

[0085] A structured storage layer construction unit is configured to construct a structured storage layer for persistent storage based on the storage characteristics of the low-frequency access data.

[0086] An unstructured storage layer construction unit is configured to construct an unstructured storage layer for large file storage based on the storage characteristics of the unstructured data.

[0087] A routing strategy configuration unit is configured to configure a unified data routing strategy.

[0088] A data synchronization mechanism and unified access interface establishment unit is configured to establish a cross-level data synchronization mechanism and unified access interface according to the data routing strategy.

[0089] A storage and management unit is configured to guide different types of preprocessing data to corresponding storage layers for storage and management based on the synchronization mechanism and the access interface.

[0090] The working principles of the various modules in the multi-source heterogeneous data acquisition and storage system of the embodiment are the same as the principles of the various steps in the method embodiment, which will not be repeated here.

[0091] Based on the above embodiment, the application further provides a terminal, and a principle block diagram of the terminal can be as shown in Figure 5 The terminal can include one or more processors 100 Figure 5The memory 101 and the computer program 102 stored in the memory 101 and executable on the one or more processors 100, for example, a program for collecting and storing multi-source heterogeneous data, are shown in the figure. The one or more processors 100 can implement each step in the method embodiment for collecting and storing multi-source heterogeneous data when executing the computer program 102. Alternatively, the one or more processors 100 can implement the functions of each module / unit in the method embodiment for collecting and storing multi-source heterogeneous data when executing the computer program 102, which is not limited here.

[0092] In an embodiment, the processor 100 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0093] In an embodiment, the memory 101 can be an internal storage unit of the electronic device, for example, a hard disk or a memory of the electronic device. The memory 101 can also be an external storage device of the electronic device, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. provided on the electronic device. Further, the memory 101 can include both the internal storage unit and the external storage device of the electronic device. The memory 101 is used to store computer programs and other programs and data required by the terminal. The memory 101 can also be used to temporarily store data that has been output or will be output.

[0094] Those skilled in the art can understand that, Figure 5 The block diagram shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the terminal to which the scheme of the present application is applied. The specific terminal can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0095] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing relevant hardware. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, operating database or other medium used in the embodiments of the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0096] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for collecting and storing multi-source heterogeneous data, characterized in that, The method comprises: collecting data from various data sources according to a collection strategy through an adapter and task scheduling to obtain collected data; preprocessing the collected data to obtain preprocessed data; classifying the preprocessed data based on data types and access characteristics, and constructing a cache layer, a structured storage layer and an unstructured storage layer according to the classification results for storing the preprocessed data.

2. The method for collecting and storing multi-source heterogeneous data according to claim 1, characterized in that, Before the collecting data from various data sources according to a collection strategy through an adapter and task scheduling to obtain collected data, the method comprises: developing a data source adapter based on actual requirements; configuring data source information according to the data source adapter to obtain configuration information; defining a data collection strategy based on the configuration information; setting a task schedule for data collection according to the collection strategy.

3. The method for collecting and storing multi-source heterogeneous data according to claim 2, characterized in that, The preprocessing the collected data to obtain preprocessed data comprises: cleaning the collected data to remove duplicate, incorrect or invalid data to obtain clean data after preliminary cleaning; performing format standardization conversion on the clean data to obtain standardized data with unified structure; associating and fusing the standardized data based on key fields to obtain preprocessed data.

4. The method for collecting and storing multi-source heterogeneous data according to claim 3, characterized in that: The classifying the preprocessed data based on data types and access characteristics comprises: extracting information from the preprocessed data to obtain data types and access characteristics of the preprocessed data; dividing the preprocessed data into high-frequency access data, low-frequency access data and unstructured data based on the data types and the access characteristics.

5. The method for collecting and storing multi-source heterogeneous data according to claim 4, characterized in that, The constructing a cache layer, a structured storage layer and an unstructured storage layer according to the classification results comprises: analyzing storage requirements of the classification results to determine storage characteristics of the high-frequency access data, the low-frequency access data and the unstructured data; constructing a cache layer for fast read-write operation based on the storage characteristics of the high-frequency access data; constructing a structured storage layer for persistent storage based on the storage characteristics of the low-frequency access data; constructing an unstructured storage layer for large file storage based on the storage characteristics of the unstructured data.

6. The method for collecting and storing multi-source heterogeneous data according to claim 5, characterized in that, The storing the preprocessed data comprises: configuring a unified data routing strategy; establishing a cross-level data synchronization mechanism and a unified access interface according to the data routing strategy; directing different types of preprocessed data to corresponding storage layers for storage and management based on the synchronization mechanism and the access interface.

7. The method for collecting and storing multi-source heterogeneous data according to claim 6, characterized in that, After the storing the preprocessed data, the method comprises: establishing a database index for the high-frequency access data and the low-frequency access data; constructing an inverted index for text-based unstructured data; extracting feature vectors from multimedia-based unstructured data, and establishing an index structure using a similarity matching algorithm based on the feature vectors.

8. A multi-source heterogeneous data acquisition and storage system, characterized in that, The system comprises: a collected data acquisition module configured to collect data from various data sources according to a collection strategy through an adapter and task scheduling to obtain collected data; a preprocessed data acquisition module configured to preprocess the collected data to obtain preprocessed data; The storage module is configured to classify the preprocessed data based on data types and access characteristics, and construct a cache layer, a structured storage layer and an unstructured storage layer according to the classification results, so as to store the preprocessed data.

9. A terminal, characterized by comprising: The terminal comprises a memory, a processor, and a multi-source heterogeneous data acquisition and storage program stored in the memory and executable on the processor.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores the multi-source heterogeneous data acquisition and storage program, and the multi-source heterogeneous data acquisition and storage program is executable on the processor to implement the steps of the multi-source heterogeneous data acquisition and storage method according to any one of claims 1-7.

Images