Intensive data storage methods and systems

Abstract

This application relates to an intensive data storage method and system. The method includes: acquiring first source data in real time, where the first source data is massive big data spanning multiple disciplines; setting format feature values, entry tags, and target entry feature values ​​for the first source data based on its format type, where the entry tags include entry success tags and entry failure tags; acquiring second source data and determining whether its content conforms to the format requirements based on the format feature values; if it does, performing format conversion on the second source data; performing format conversion on the second source data based on the format feature values ​​to obtain a non-relational database message body that conforms to the target entry feature values; decomposing the non-relational database message body according to the target entry feature values ​​and encapsulating it into an optimal entry granularity; and storing the non-relational database message body in a non-relational database using multi-threading technology based on the target entry feature values ​​and the optimal entry granularity.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and in particular to an intensive data storage method and system. Background Technology

[0002] With the rapid development of the internet, the types and scale of data on the internet are growing at an astonishing rate, and the era of big data has become a reality. Currently, data storage includes unstructured data storage (for unstructured data) and structured data storage (for structured data). Unstructured data, also known as non-relational data, can be logically expressed using a two-dimensional table structure and resides in non-relational databases, such as distributed search servers (Elasticsearch, ES) that use JavaScript Object Notation (JSON) to index data via Hypertext Transfer Protocol. In contrast, unstructured data refers to data that is inconvenient to represent using the two-dimensional logic of a database. This includes all formats of office documents, text, images, Extensible Markup Language (XML), HyperText Markup Language (HTML), various reports, images, and audio / video information, etc., and is stored in relational databases. As big data is increasingly applied across various industries, querying massive amounts of data faces unprecedented challenges.

[0003] In the field of big data, ensuring high performance, high storage capacity, and other requirements, especially in business scenarios involving time-series queries of personnel information, often necessitates the use of relational databases for storing and processing time-series big data. However, the inherent limitations of relational databases prevent them from efficiently storing and querying time-series big data, thus impacting its business applications. While relational databases are used for storing and processing big data, their inherent weaknesses prevent efficient storage and querying. Furthermore, different file formats and message notification types require customized programs for data storage, resulting in significant waste of human and physical resources, excessive storage space consumption, and reduced query efficiency. Currently, technical personnel are using Druid, a specialized massive data time-series library, to manage time-series big data, enabling efficient storage and rapid processing of massive amounts of time-series data. However, in the current scenario, if the time-series data is highly discrete and has significant latency, it can cause serious data ingestion scheduling problems for Druid during data consumption. For example, segmented data may fail to load from the distributed system infrastructure Hadoop, and the source data dataset MySQL may be under heavy load, making the data ingestion task highly likely to fail. Furthermore, in the data query phase, the current use of single-label storage and the large data volume result in poor query performance.

[0004] In summary, conventional methods for storing and processing big data suffer from low storage and query efficiency due to the limitations of relational databases. Summary of the Invention

[0005] Therefore, it is necessary to provide an intensive data storage method and system that can uniformly process file content and message types of different dimensions into the data format of the non-relational database to be stored, thereby improving the efficiency of big data storage ingestion and retrieval.

[0006] In a first aspect, this application provides an intensive data storage method, the method comprising:

[0007] Real-time acquisition of primary source data, which is massive big data spanning multiple disciplines;

[0008] Based on the format type of the first source data, set format feature values, entry tags and target entry feature values ​​for the first source data, wherein the entry tags include entry success tags and entry failure tags;

[0009] Acquire the second source data and determine whether the content of the second source data meets the format requirements based on the format feature value of the first source data. If it does, perform format conversion on the second source data.

[0010] The second source data is format-converted according to the format feature value, so that the second source data obtains a non-relational database message body that conforms to the target inbound feature value;

[0011] The message body of the non-relational database is decomposed and encapsulated into the optimal granularity for inbound storage based on the target inbound feature value.

[0012] Based on the target inbound feature value and the optimal inbound granularity, the non-relational database message body is stored in a non-relational database using multi-threading technology.

[0013] In one embodiment, the step of acquiring the second source data and determining whether the content of the second source data conforms to the format requirements based on the format feature value of the first source data includes:

[0014] If the content of the second source data does not meet the format requirements of the first source data, then discard the second source data and mark it as an ingestion failure tag;

[0015] Mark the second source data that meets the format requirements of the first source data with the "successful entry" label.

[0016] In one embodiment, the method further includes:

[0017] The second source data with the aforementioned failed entry label is re-entered into the database to ensure the integrity of the second source data.

[0018] In one embodiment, the step of storing the non-relational database message body in a non-relational database using multi-threading technology then includes:

[0019] The data stored in the non-relational database is sent to the upper-layer application, which reads the data stored in the non-relational database and analyzes and judges it.

[0020] In one embodiment, the method further includes:

[0021] Record a data processing log, which is used to access the second source data with the "successful entry" label and the second source data with the "failed entry" label.

[0022] In one embodiment, the optimal granularity of data entry is equal to the ratio of the product of the message body data volume, error rate, and target database coefficient to the product of the bandwidth coefficient, idle / busy time coefficient, and concurrent thread coefficient.

[0023] Secondly, this application provides an intensive data storage system, the system comprising:

[0024] The data acquisition module is used to acquire the first source data in real time, which is a massive amount of data from multiple disciplines.

[0025] The data configuration module is used to set format feature values, entry tags and target entry feature values ​​for the first source data according to the format type of the first source data. The entry tags include entry success tags and entry failure tags.

[0026] The extraction and filtering module is used to obtain the second source data and determine whether the content of the second source data meets the format requirements based on the format feature value of the first source data. If it does, the second source data is converted into a format.

[0027] The data processing module is used to perform format conversion on the second source data according to the format feature value, so that the second source data obtains a non-relational database message body that conforms to the target database entry feature value;

[0028] The granularity configuration module is used to decompose the non-relational database message body according to the target inbound feature value and encapsulate it into the optimal inbound granularity.

[0029] A multi-threaded storage module is used to store the non-relational database message body in a non-relational database using multi-threading technology, based on the target ingestion feature value and the optimal ingestion granularity.

[0030] In one embodiment, the system further includes:

[0031] The non-relational database module is used to send the data stored in the non-relational database to the upper-layer application, and the upper-layer application is used to read the data stored in the non-relational database and analyze and judge it.

[0032] The log retrieval module is used to record data processing logs, which are used to retrieve second source data with the "successful entry" label and second source data with the "failed entry" label.

[0033] The re-entry module is used to re-enter the second source data with the entry failure tag to ensure the integrity of the second source data.

[0034] Thirdly, this application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0035] Real-time acquisition of primary source data, which is massive big data spanning multiple disciplines;

[0036] Based on the format type of the first source data, set format feature values, entry tags and target entry feature values ​​for the first source data, wherein the entry tags include entry success tags and entry failure tags;

[0037] Acquire the second source data and determine whether the content of the second source data meets the format requirements based on the format feature value of the first source data. If it does, perform format conversion on the second source data.

[0038] The second source data is format-converted according to the format feature value, so that the second source data obtains a non-relational database message body that conforms to the target inbound feature value;

[0039] The message body of the non-relational database is decomposed and encapsulated into the optimal granularity for inbound storage based on the target inbound feature value.

[0040] Based on the target inbound feature value and the optimal inbound granularity, the non-relational database message body is stored in a non-relational database using multi-threading technology.

[0041] Fourthly, this application provides a computer storage medium storing a computer program, which, when executed by a processor, performs the following steps:

[0042] Real-time acquisition of primary source data, which is massive big data spanning multiple disciplines;

[0043] Based on the format type of the first source data, set format feature values, entry tags and target entry feature values ​​for the first source data, wherein the entry tags include entry success tags and entry failure tags;

[0044] Acquire the second source data and determine whether the content of the second source data meets the format requirements based on the format feature value of the first source data. If it does, perform format conversion on the second source data.

[0045] The second source data is format-converted according to the format feature value, so that the second source data obtains a non-relational database message body that conforms to the target inbound feature value;

[0046] The message body of the non-relational database is decomposed and encapsulated into the optimal granularity for inbound storage based on the target inbound feature value.

[0047] Based on the target inbound feature value and the optimal inbound granularity, the non-relational database message body is stored in a non-relational database using multi-threading technology.

[0048] The advantages of this invention are as follows: The aforementioned intensive data storage method and system acquire massive amounts of cross-disciplinary big data in real time, and set format feature values, storage labels, and target storage feature values ​​according to the format type of the acquired big data. During the acquisition of source data, the system determines whether the acquired source data meets the format requirements based on the pre-set format feature values, and performs format conversion when the source data meets the requirements, so that the source data obtains a non-relational database message body that conforms to the preset target storage feature values. Subsequently, the source data is decomposed and encapsulated into the optimal storage granularity according to the target storage feature values ​​and the optimal storage granularity. Finally, the corresponding non-relational database message body is stored in a non-relational database using multi-threading technology based on the target storage feature values ​​and the optimal storage granularity. This method effectively improves the storage and query efficiency of big data by uniformly processing file content or message types of different dimensions into the data format of the non-relational database to be stored, and encapsulates it into the database for relevant programs to call, while reducing the storage space of big data to a certain extent. Attached Figure Description

[0049] Figure 1 This is a flowchart of a compact data storage method according to an embodiment of this application;

[0050] Figure 2 This is a flowchart of a compact data storage method according to an embodiment of this application;

[0051] Figure 3 This is a schematic diagram of the system architecture of an intensive data storage scheme according to an embodiment of this application;

[0052] Figure 4 This is a schematic diagram of the structure of an intensive data storage system according to an embodiment of this application;

[0053] Figure 5 This is a schematic diagram of the structure of an intensive data storage system according to an embodiment of this application;

[0054] Figure 6 This is an internal structural diagram of a computer device according to an embodiment of this application. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] like Figure 1 As shown, in one embodiment, a compact data storage method includes the following steps:

[0057] Step S110: Obtain the first source data in real time. The first source data is a massive amount of data from multiple disciplines.

[0058] Specifically, the server should try to acquire massive amounts of data from across different disciplines.

[0059] Step S120: Based on the format type of the first source data, set the format feature value, entry label and target entry feature value for the first source data. The entry label includes an entry success label and an entry failure label.

[0060] Specifically, the server sets format feature values, entry tags, and target entry feature values ​​for the source data based on the format type of the source data obtained in step S110. The entry tags include entry success tags and entry failure tags.

[0061] Step S130: Obtain the second source data and determine whether the content of the second source data meets the format requirements based on the format feature value of the first source data. If it does, perform format conversion on the second source data.

[0062] Specifically, the server obtains new source data and determines whether the content of the new source data meets the format requirements based on the pre-set format feature values ​​of the source data. If it does, the server performs format conversion on the newly obtained source data.

[0063] Step S140: The second source data is format-converted according to the format feature value, so that the second source data obtains a non-relational database message body that conforms to the target database entry feature value.

[0064] Specifically, the server performs format conversion on the newly acquired source data according to the pre-set format feature values, so that the source data obtains a non-relational data message body that conforms to the target database entry feature values.

[0065] Step S150: Decompose the message body of the non-relational database according to the target inbound feature value and encapsulate it into the optimal inbound granularity.

[0066] Specifically, the server decomposes the message body of the non-relational database according to the target inbound feature value and encapsulates it into the optimal inbound granularity to avoid wasting storage resources.

[0067] Step S160: Based on the target inbound feature value and the optimal inbound granularity, the non-relational database message body is stored in the non-relational database using multi-threading technology.

[0068] Specifically, the client stores the non-relational database message body in the non-relational database based on the input characteristics of the source data and the optimal input granularity, using multi-threading technology.

[0069] The aforementioned intensive data storage method acquires massive amounts of cross-disciplinary big data in real time and sets format feature values, storage labels, and target storage feature values ​​for the acquired big data based on its format type. During the acquisition of source data, the method determines whether the acquired source data meets the format requirements based on the pre-set format feature values. If the source data meets the format requirements, it is converted to obtain a non-relational database message body that conforms to the preset target storage feature values. Subsequently, the source data is decomposed and encapsulated into the optimal storage granularity based on the target storage feature values ​​and the optimal storage granularity. Finally, based on the target storage feature values ​​and the optimal storage granularity, the corresponding non-relational database message body is stored in a non-relational database using multi-threading technology. This method effectively improves the efficiency of big data storage and querying by uniformly processing file content or message types of different dimensions into the data format to be stored in the non-relational database and encapsulating it for use by relevant programs. It also reduces the storage space of big data to a certain extent.

[0070] like Figure 2 As shown, in one embodiment, a compact data storage method includes the following steps:

[0071] Step S210: Obtain the first source data in real time. The first source data is a massive amount of data from multiple disciplines.

[0072] Specifically, the server should try to acquire massive amounts of data from across different disciplines.

[0073] Step S220: Based on the format type of the first source data, set the format feature value, entry label and target entry feature value for the first source data. The entry label includes an entry success label and an entry failure label.

[0074] Specifically, the server sets format feature values, entry tags, and target entry feature values ​​for the source data based on the format type of the source data obtained in step S210. The entry tags include entry success tags and entry failure tags.

[0075] Step S230: Obtain the second source data and determine whether the content of the second source data meets the format requirements based on the format feature value of the first source data. If it does, perform format conversion on the second source data.

[0076] Specifically, the server obtains new source data and determines whether the content of the new source data meets the format requirements based on the pre-set format feature values ​​of the source data. If it does, the server performs format conversion on the newly obtained source data and marks it as successfully inserted into the database. If the content of the newly obtained source data does not meet the preset format requirements, the server discards the source data and marks it as failed to be inserted into the database.

[0077] Step S240: Re-enter the second source data with the entry failure label to ensure the integrity of the second source data.

[0078] Specifically, to ensure the integrity of newly acquired source data, the server will re-enter source data with the entry failure tag.

[0079] Step S250: Convert the format of the second source data according to the format feature value so that the second source data obtains a non-relational database message body that conforms to the target database entry feature value.

[0080] Specifically, the server performs format conversion on the newly acquired source data according to the pre-set format feature values, so that the source data obtains a non-relational data message body that conforms to the target database entry feature values.

[0081] Step S260: Decompose the non-relational database message body according to the target inbound feature value and encapsulate it into the optimal inbound granularity.

[0082] Specifically, the server decomposes the message body of the non-relational database based on the target database input characteristics and encapsulates it into the optimal input granularity to avoid wasting storage resources. The optimal input granularity is equal to the ratio of the product of the message body data volume, error rate, and target database coefficient to the product of the bandwidth coefficient, idle / busy time coefficient, and concurrent thread coefficient.

[0083] Step S270: Based on the target inbound feature value and the optimal inbound granularity, the non-relational database message body is stored in the non-relational database using multi-threading technology.

[0084] Specifically, the client stores the non-relational database message body in the non-relational database based on the input characteristics of the source data and the optimal input granularity, using multi-threading technology.

[0085] Step S280: The data stored in the non-relational database is sent to the upper-layer application, which reads the data stored in the non-relational database and analyzes and judges it.

[0086] Specifically, the client sends the data stored in the non-relational database to the upper-layer application, which then reads and analyzes the data stored in the non-relational database.

[0087] Step S290: Record the data processing log. The data processing log is used to retrieve the second source data with the "successful entry" label and the second source data with the "failed entry" label.

[0088] Specifically, during the data storage process, the server records data processing logs, which allow users to easily access source data that was successfully imported and source data that failed to be imported.

[0089] The aforementioned intensive data storage method acquires massive amounts of cross-disciplinary big data in real time and sets format feature values, storage labels, and target storage feature values ​​for the acquired big data based on its format type. During the acquisition of source data, the method determines whether the acquired source data meets the format requirements based on the pre-set format feature values. If the source data meets the format requirements, it is converted to obtain a non-relational database message body that conforms to the preset target storage feature values. Subsequently, the source data is decomposed and encapsulated into the optimal storage granularity based on the target storage feature values ​​and the optimal storage granularity. Finally, based on the target storage feature values ​​and the optimal storage granularity, the corresponding non-relational database message body is stored in a non-relational database using multi-threading technology. This method effectively improves the efficiency of big data storage and querying by uniformly processing file content or message types of different dimensions into the data format to be stored in the non-relational database and encapsulating it for use by relevant programs. It also reduces the storage space of big data to a certain extent.

[0090] like Figure 3 As shown, in one embodiment, an intensive data storage scheme achieves high efficiency in data storage and retrieval by uniformly processing file content or message types of different dimensions into a non-relational database data format for storage and encapsulating it into the database for program calls. The system implementing this scheme includes a data receiving unit, a data configuration unit, an extraction and filtering unit, a data processing unit, a granular configuration unit, a multi-threaded storage unit, a non-relational database unit, a log retrieval unit, and a re-entry unit.

[0091] The data receiving unit receives file or message notifications in real time to obtain source data. The file or message notification is a massive amount of data across disciplines. The file type can be common CSV, XML, JSON, etc., and the message type can be Kafka, API message, etc.

[0092] The data configuration unit sets format feature values, inbound labels, and target inbound feature values ​​based on the format type of the source data. For example, when the source data format type is Kafka message data, the target inbound feature value is set to target time series database, the inbound label is set to pending inbound, and the target inbound feature value is set to the feature values ​​of non-relational databases such as target time series databases and graph databases.

[0093] The extraction and filtering unit is used to intercept illegal files or information. Specifically, it determines whether the original data content conforms to the format rules based on the source data format type. If it conforms to the format rules, the source data is sent to the data processing unit for format conversion. If it does not conform, the corresponding source data is discarded and the entry label is set to failure.

[0094] The data processing unit is used to convert source data into the inbound data format required by different heterogeneous databases. This includes unifying the source data based on format feature values ​​to obtain a non-relational database message body that conforms to the target inbound feature values. Unifying the source data includes processing the source data into a JSON format string and converting it into the inbound data format required by different heterogeneous databases, which is then used as the storage non-relational database message body.

[0095] The granularity configuration unit decomposes and encapsulates the non-relational database message body into optimal inbound granularity based on the target inbound feature values. Optimal inbound granularity = (Message body data volume × Error rate × Target database coefficient) / (Bandwidth coefficient × Idle / busy time coefficient × Concurrency thread coefficient). Wherein, a larger message body data volume, a lower error rate, wider bandwidth, a slower idle time, and fewer concurrent threads require a larger granularity. The target database coefficient varies depending on the target inbound feature values.

[0096] The multi-threaded storage unit, based on the target input characteristic values ​​and optimal input granularity, uses interface calls and multi-threading technology to store the non-relational database message body into a non-relational database unit, and sets the input tag to "completed". The multi-threaded storage processing unit encapsulates the native input API for program calls, which can greatly improve the efficiency and success rate of input while reducing storage space.

[0097] Non-relational databases (NRDs) provide data query capabilities, offering the data stored within them to upper-layer applications for analysis and judgment. NRDs include key-value stores, columnar stores, document stores, graph databases, and time-series databases. By using specialized storage methods, they reduce storage space by half compared to relational databases, significantly improving query speed and making them a crucial technology for handling massive amounts of data. NRDs typically store data as objects, with relationships between objects determined by their individual attributes. For example, time-series databases are primarily used to process time-stamped data. Due to their unique storage method and ability to rapidly process massive amounts of time-series data, they are widely used in the Internet of Things (IoT), connected vehicles, industrial internet, and monitoring and analysis, enabling the control and analysis of overall performance indicators.

[0098] The log call unit is used to record data processing logs and to retrieve and display completed and failed tasks and key nodes.

[0099] For content that fails to be imported into the database for various reasons, the re-import unit logs the information and then schedules the messages to be re-imported into the database to ensure the integrity of the source data.

[0100] The aforementioned intensive data storage solution handles file content or message types from different professional perspectives, unifying them into the data format required for storage in a non-relational database. It then encapsulates this data into a native import API for program calls, significantly improving import efficiency and success rate while reducing storage space. Compared to existing methods using relational databases for storing and processing big data, this solution halves database storage space, greatly improves query speed, and significantly reduces resource consumption. Furthermore, unlike existing methods that require developing specific programs to handle different non-relational database import formats, resulting in wasted human and material resources, this solution significantly reduces labor, resource, and time costs.

[0101] like Figure 4 As shown, in one embodiment, an intensive data storage system includes a data acquisition module 410, a data configuration module 420, an extraction and filtering module 430, a data processing module 440, a granular configuration module 450, and a multi-threaded storage module 460.

[0102] The data acquisition module 410 is used to acquire the first source data in real time. The first source data is a massive amount of data from multiple disciplines.

[0103] The data configuration module 420 is used to set format feature values, entry tags and target entry feature values ​​for the first source data according to the format type of the first source data. The entry tags include entry success tags and entry failure tags.

[0104] The extraction and filtering module 430 is used to obtain the second source data and determine whether the content of the second source data meets the format requirements based on the format feature value of the first source data. If it does, the second source data is converted into a format.

[0105] The data processing module 440 is used to perform format conversion on the second source data according to the format feature value, so that the second source data obtains a non-relational database message body that conforms to the target database entry feature value.

[0106] The granularity configuration module 450 is used to decompose the non-relational database message body according to the target inbound feature value and encapsulate it into the optimal inbound granularity.

[0107] The multi-threaded storage module 460 is used to store the non-relational database message body in the non-relational database based on the target ingestion feature value and the optimal ingestion granularity, using multi-threading technology.

[0108] The aforementioned intensive data storage system acquires massive amounts of cross-disciplinary big data in real time and sets format feature values, storage labels, and target storage feature values ​​for the acquired big data based on its format type. During the acquisition of source data, the system determines whether the acquired source data meets the format requirements based on the pre-set format feature values. If the source data meets the format requirements, it is converted to obtain a non-relational database message body that conforms to the preset target storage feature values. Subsequently, the source data is decomposed and encapsulated into the optimal storage granularity based on the target storage feature values ​​and the optimal storage granularity. Finally, based on the target storage feature values ​​and the optimal storage granularity, the corresponding non-relational database message body is stored in a non-relational database using multi-threading technology. This system effectively improves the efficiency of big data storage and querying by uniformly processing file content or message types of different dimensions into the data format to be stored in the non-relational database and encapsulating it for use by relevant programs. It also reduces the storage space required for big data to some extent.

[0109] like Figure 5 As shown, in one embodiment, an intensive data storage system includes a data acquisition module 510, a data configuration module 520, an extraction and filtering module 530, a re-entry module 540, a data processing module 550, a granular configuration module 560, a multi-threaded storage module 570, a non-relational database module 580, and a log retrieval module 590.

[0110] The data acquisition module 510 is used to acquire the first source data in real time, which is a massive amount of data from multiple disciplines.

[0111] The data configuration module 520 is used to set format feature values, entry tags and target entry feature values ​​for the first source data according to the format type of the first source data. The entry tags include entry success tags and entry failure tags.

[0112] The extraction and filtering module 530 is used to obtain the second source data and determine whether the content of the second source data meets the format requirements based on the format feature value of the first source data. If it does, the second source data is converted into a format.

[0113] The re-entry module 540 is used to re-enter the second source data with the entry failure tag to ensure the integrity of the second source data.

[0114] The data processing module 550 is used to perform format conversion on the second source data according to the format feature value, so that the second source data obtains a non-relational database message body that conforms to the target database entry feature value.

[0115] The granularity configuration module 560 is used to decompose the message body of the non-relational database according to the target inbound feature value and encapsulate it into the optimal inbound granularity.

[0116] The multi-threaded storage module 570 is used to store the message body of a non-relational database in a non-relational database using multi-threading technology, based on the target ingestion feature value and the optimal ingestion granularity.

[0117] The NoSQL database module 580 is used to send data stored in the NoSQL database to the upper-layer application, which then reads and analyzes the data stored in the NoSQL database.

[0118] The log retrieval module 590 is used to record data processing logs. The data processing logs are used to retrieve the first source data with the "successful entry" tag and the second source data with the "failed entry" tag.

[0119] The aforementioned intensive data storage system acquires massive amounts of cross-disciplinary big data in real time and sets format feature values, storage labels, and target storage feature values ​​for the acquired big data based on its format type. During the acquisition of source data, the system determines whether the acquired source data meets the format requirements based on the pre-set format feature values. If the source data meets the format requirements, it is converted to obtain a non-relational database message body that conforms to the preset target storage feature values. Subsequently, the source data is decomposed and encapsulated into the optimal storage granularity based on the target storage feature values ​​and the optimal storage granularity. Finally, based on the target storage feature values ​​and the optimal storage granularity, the corresponding non-relational database message body is stored in a non-relational database using multi-threading technology. This system effectively improves the efficiency of big data storage and querying by uniformly processing file content or message types of different dimensions into the data format to be stored in the non-relational database and encapsulating it for use by relevant programs. It also reduces the storage space required for big data to some extent.

[0120] In one embodiment, a computer device is provided, which may be a smart terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a compact data storage method.

[0121] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0122] In one embodiment, a computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above-described method embodiments. In another embodiment, a computer storage medium stores a computer program, the computer program being executed by a processor to implement the steps in the above-described method embodiments.

[0123] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the steps in the above method embodiments.

[0124] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0125] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0126] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A method for intensive data storage, characterized in that, The method includes: Real-time acquisition of primary source data, which is massive big data spanning multiple disciplines; Based on the format type of the first source data, set format feature values, entry tags and target entry feature values ​​for the first source data, wherein the entry tags include entry success tags and entry failure tags; Acquire the second source data and determine whether the content of the second source data meets the format requirements based on the format feature value of the first source data. If it does, perform format conversion on the second source data. The second source data is format-converted according to the format feature value, so that the second source data obtains a non-relational database message body that conforms to the target inbound feature value; The message body of the non-relational database is decomposed and encapsulated into the optimal granularity for inbound storage based on the target inbound feature value. Based on the target inbound feature value and the optimal inbound granularity, the non-relational database message body is stored in a non-relational database using multi-threading technology; The step of acquiring the second source data and determining whether the content of the second source data conforms to the format requirements based on the format feature value of the first source data includes: If the content of the second source data does not meet the format requirements of the first source data, then discard the second source data and mark it as an ingestion failure tag; Mark the second source data that meets the format requirements of the first source data with the "successful entry" label. The method further includes: The second source data with the aforementioned failed entry label is re-entered into the database to ensure the integrity of the second source data; The optimal granularity of data entry is equal to the ratio of the product of the message body data volume, error rate, and target database coefficient to the product of the bandwidth coefficient, idle / busy time coefficient, and concurrent thread coefficient.

2. The intensive data storage method according to claim 1, characterized in that, The step of storing the non-relational database message body in a non-relational database using multi-threading technology, followed by: The data stored in the non-relational database is sent to the upper-layer application, which reads the data stored in the non-relational database and analyzes and judges it.

3. The intensive data storage method according to claim 1, characterized in that, The method further includes: Record a data processing log, which is used to access the second source data with the "successful entry" label and the second source data with the "failed entry" label.

4. An intensive data storage system, characterized in that, The system includes: The data acquisition module is used to acquire the first source data in real time, which is a massive amount of data from multiple disciplines. The data configuration module is used to set format feature values, entry tags and target entry feature values ​​for the first source data according to the format type of the first source data. The entry tags include entry success tags and entry failure tags. The extraction and filtering module is used to obtain the second source data and determine whether the content of the second source data meets the format requirements based on the format feature value of the first source data. If it does, the second source data is converted into a format. The data processing module is used to perform format conversion on the second source data according to the format feature value, so that the second source data obtains a non-relational database message body that conforms to the target database entry feature value; The granularity configuration module is used to decompose the non-relational database message body according to the target inbound feature value and encapsulate it into the optimal inbound granularity. A multi-threaded storage module is used to store the non-relational database message body in a non-relational database using multi-threading technology, based on the target inbound feature value and the optimal inbound granularity. Specifically, the extraction and filtering module is used to discard the second source data and mark it with the entry failure tag if the content of the second source data does not conform to the format requirements of the first source data; Mark the second source data that meets the format requirements of the first source data with the "successful entry" label. The system further includes a re-entry module, used to re-enter the second source data with the entry failure tag to ensure the integrity of the second source data; The optimal granularity of data entry is equal to the ratio of the product of the message body data volume, error rate, and target database coefficient to the product of the bandwidth coefficient, idle / busy time coefficient, and concurrent thread coefficient.

5. The intensive data storage system according to claim 4, characterized in that, The system also includes: The non-relational database module is used to send the data stored in the non-relational database to the upper-layer application, and the upper-layer application is used to read the data stored in the non-relational database and analyze and judge it. The log retrieval module is used to record data processing logs, which are used to retrieve second source data with the "successful entry" tag and second source data with the "failed entry" tag.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 3.

7. A computer storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.

Images