Format file-based data storage method and device, equipment and medium

Abstract

The invention relates to the field of big data, can be applied to the field of financial science and technology, and discloses a formatted file-based data storage method, device and equipment and a medium, the method comprises the following steps: accessing a target platform, and obtaining original data from the target platform; obtaining common information and additional information from the original data; obtaining a field value of the additional information, and obtaining a type prefix corresponding to the field value; combining the type prefix and the field value in a preset mode to obtain a character string; performing escape processing on the character string to obtain a processed character string; and adding a common field and an additional field in the CSV format file, storing common information through the common field, and storing the processed character string through the additional field to obtain a storage file of the original data. The storage amount of original data can be effectively reduced, and the utilization efficiency of storage resources is improved.

Description

Technical Field

[0001] The present invention relates to the field of big data and can be applied to the field of financial technology. In particular, the present invention relates to a data storage method, device, equipment and medium based on formatted files. Background Art

[0002] Driven by the current wave of digitalization, digital transformation in various industries has accelerated, and the sources of data generation are extensive and diverse. From real-time interactive data from smart terminals to massive business records generated by corporate operations, the scale of raw data is getting larger and larger, which puts higher demands on data storage technology.

[0003] While traditional CSV files are simple and easy to use, they exhibit significant shortcomings when processing large amounts of raw data. This is because traditional CSV files record raw data line by line in simple text format, repeatedly storing placeholders during the recording process. These repeated placeholders increase the storage space occupied by the raw data. Therefore, how to store raw data is a technical problem that needs to be solved urgently. Summary of the Invention

[0004] The present invention provides a data storage method, device, computer equipment and storage medium based on format files to solve the technical problem of how to store original data.

[0005] In a first aspect, a data storage method based on a format file is provided, comprising:

[0006] Access the target platform and obtain raw data from the target platform;

[0007] Obtain common and additional information from original data;

[0008] Get the field value of the additional information and the type prefix corresponding to the field value;

[0009] Use the preset method to combine the type prefix and field value to obtain a string;

[0010] Escape the string to obtain the processed string;

[0011] Add common fields and additional fields to the CSV format file, use the common fields to store common information, and use the additional fields to store processed character strings to obtain a storage file for the original data.

[0012] Furthermore, the obtaining of common information and additional information from the original data includes:

[0013] Obtaining a preset format of the original data, parsing the original data using a parsing method corresponding to the preset format, and obtaining the parsed original data;

[0014] Obtain data records from the parsed raw data, and obtain common information and additional information in the data records.

[0015] Furthermore, the obtaining of the field value of the additional information and the obtaining of the type prefix corresponding to the field value include:

[0016] Get the field value of the additional information, query the field value from the query table, and get the query result;

[0017] Get the data type of the field value from the query result, and determine the type prefix corresponding to the field value based on the data type of the field value.

[0018] Furthermore, the type prefix and the field value are combined in a preset manner to obtain a string, including:

[0019] Concatenate the type prefix and the field value to obtain the concatenated value;

[0020] Access the preset file, load the programming tool from the preset file, and convert the concatenated value into a string using the built-in conversion function of the programming tool.

[0021] Furthermore, the character string is escaped to obtain a processed character string, including:

[0022] Determine whether the string contains special characters. Special characters include comma, quotation mark, newline, carriage return, tab, backslash, or any combination thereof.

[0023] When a character string contains special characters, the character string is input into the escape module, and the escape module is used to escape the character string to obtain a processed character string.

[0024] Furthermore, the common fields and the extra fields are added to the CSV format file, the common information is stored in the common fields, and the processed character strings are stored in the extra fields to obtain a storage file of the original data, including:

[0025] Access the data processing tool, export a CSV format file from the data processing tool, and add common fields and additional fields to the first row of the CSV format file;

[0026] The field values ​​of the common data are stored in the cells of the column where the common fields are located, and the processed character strings are stored in the cells of the column where the additional fields are located, thereby obtaining a storage file of the original data.

[0027] Furthermore, after adding common fields and additional fields to the CSV format file, storing common information through the common fields, and storing processed character strings through the additional fields to obtain a storage file of the original data, the data storage method includes:

[0028] Read the preset upload time and determine whether the current time is the upload time;

[0029] If the current time is the upload time, the push instruction is obtained, the push instruction is executed, and the storage file is pushed to the data warehouse corresponding to the target platform.

[0030] In a second aspect, a data storage device based on a format file is provided, comprising:

[0031] The first acquisition module is used to access the target platform and obtain original data from the target platform;

[0032] A second acquisition module is used to obtain common information and additional information from the original data;

[0033] The third acquisition module is used to obtain the field value of the additional information and the type prefix corresponding to the field value;

[0034] The combination module is used to combine the type prefix and field value in a preset way to obtain a string;

[0035] The processing module is used to perform escape processing on the character string to obtain the processed character string;

[0036] The storage module is used to add common fields and additional fields to the CSV format file, store common information through the common fields, and store processed character strings through the additional fields to obtain a storage file of the original data.

[0037] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned data storage method when executing the computer program.

[0038] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned data storage method are implemented.

[0039] The present application provides a data storage method, apparatus, computer equipment, and storage medium based on a formatted file, which access a target platform and obtain original data from the target platform; obtain common information and additional information from the original data; obtain the field value of the additional information and the type prefix corresponding to the field value; combine the type prefix and the field value in a preset manner to obtain a string; perform escape processing on the string to obtain a processed string; add common fields and additional fields to a CSV formatted file, store the common information through the common fields, and store the processed string through the additional fields to obtain a storage file of the original data. The beneficial effects are in two aspects. On the one hand, common fields and additional fields are added to a CSV formatted file, store the common information through the common fields, and store the escaped string through the additional fields to obtain a storage file of the original data. Since the escaped string is stored on demand using the additional fields, the situation of repeated storage of placeholders is avoided, and thus the storage space occupied by the original data is not increased, thereby solving the technical problem of how to store the original data, and can effectively reduce the storage amount of the original data and improve the utilization efficiency of storage resources. On the other hand, it is not affected by human intervention, which is conducive to improving the reliability of the storage file of the original data. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0041] Figure 1 This is a schematic diagram of an application environment of a data storage method according to an embodiment of the present invention;

[0042] Figure 2 A schematic flow chart of a data storage method provided by an embodiment of the present invention;

[0043] Figure 3 yes Figure 2 A schematic flow chart of a specific implementation of step S23;

[0044] Figure 4 yes Figure 2 A schematic flow chart of a specific implementation of step S25;

[0045] Figure 5 yes Figure 2 A schematic flow chart of a specific implementation of step S26;

[0046] Figure 6 is a structural diagram of a data storage device according to an embodiment of the present invention;

[0047] Figure 7 is a structural diagram of a computer device in one embodiment of the present invention;

[0048] Figure 8 FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0050] See also Figure 1 , Figure 1 This is a schematic diagram of an application environment of a data storage method according to an embodiment of the present invention. The data storage method provided by the embodiment of the present invention can be applied in the following situations: Figure 1 In an application environment, a client communicates with a server through a network.

[0051] The server accesses the target platform through the client and obtains the original data from the target platform;

[0052] Obtain common and additional information from original data;

[0053] Get the field value of the additional information and the type prefix corresponding to the field value;

[0054] Use the preset method to combine the type prefix and field value to obtain a string;

[0055] Escape the string to obtain the processed string;

[0056] Add common fields and additional fields to the CSV format file, use the common fields to store common information, and use the additional fields to store processed character strings to obtain a storage file for the original data.

[0057] In the solution implemented by the above-mentioned data storage method, device, equipment and medium, the beneficial effects are in two aspects. On the one hand, common fields and additional fields are added to the CSV format file, common information is stored through the common fields, and escaped strings are stored through the additional fields to obtain a storage file of the original data. Since the escaped strings are stored on demand using additional fields, the situation of repeated storage of placeholders is avoided, and the storage space occupied by the original data will not be increased. Therefore, the technical problem of how to store the original data is solved, and the storage amount of the original data can be effectively reduced, and the utilization efficiency of storage resources is improved. On the other hand, it will not be affected by human intervention, so it is conducive to improving the reliability of the storage file of the original data.

[0058] The device running the client is referred to as a client device.

[0059] The device running the server is referred to as the server device.

[0060] The client devices may include but are not limited to various personal computers, laptops, smartphones, tablet computers, and portable wearable devices.

[0061] The server device can be implemented as an independent server or a server cluster composed of multiple servers. The present invention will be described in detail below through specific embodiments.

[0062] See also Figure 2 , Figure 2 A flowchart of a data storage method provided in one embodiment of the present invention includes the following steps:

[0063] S21, access the target platform and obtain original data from the target platform;

[0064] Among them, accessing the target platform and obtaining raw data from the target platform include:

[0065] Access the target platform, send a data acquisition request to the target platform, receive a response message returned by the target platform based on the data acquisition request, and when the response message is a success message, access the business subsystem of the target platform and obtain the original data from the business subsystem of the target platform.

[0066] Among them, the target platforms include but are not limited to financial platforms, technology platforms, insurance platforms, and medical platforms.

[0067] Financial platforms are online platforms that leverage the internet, big data, and blockchain technologies to provide users with diversified financial services. They transcend the time and space constraints of traditional financial services and, by integrating financial resources, efficiently match supply and demand for funds. These platforms offer a wide range of financial products and services, including loans, investments, insurance, wealth management, and payments.

[0068] Technology platforms are online platforms built on cutting-edge information technologies such as artificial intelligence, cloud computing, the Internet of Things, and blockchain. Leveraging technology as their foundation, they integrate diverse resources and services, providing an ecological space for efficient collaboration and value creation among diverse entities. Functionally, these platforms can break down information silos and enable data sharing and circulation.

[0069] Insurance platforms are online platforms that leverage modern information technologies such as the internet, big data, and artificial intelligence to connect insurance product providers, insurance intermediaries, and policyholders. These platforms transcend the time and space constraints of traditional insurance sales and services, offering policyholders a rich variety of insurance product displays and comparisons, enabling them to precisely select the appropriate coverage based on their needs. Intelligent underwriting systems enable rapid assessments of policyholders' health and risk factors, improving underwriting efficiency. In the claims process, online submission of information and intelligent review processes significantly shorten the claims settlement cycle.

[0070] The medical platform is an online platform built on cutting-edge technologies such as the internet, cloud computing, big data, and artificial intelligence. Focused on patient needs, it organically integrates various stakeholders, including medical institutions, medical professionals, pharmaceutical companies, and patients, breaking down the time and space barriers of traditional medical services. Through the platform, patients can conveniently make online appointments, receive remote consultations, access electronic medical records, and query medical reports, reducing wait times and enhancing their medical experience.

[0071] For ease of explanation, the following examples are given:

[0072] For example, when the target platform is a financial platform, access the financial platform, send a data acquisition request to the financial platform, receive a response message returned by the financial platform based on the data acquisition request, and when the response message is a success message, access the business subsystem of the financial platform and obtain the original data from the business subsystem of the financial platform.

[0073] For example, when the target platform is a technology platform, access the technology platform, send a data acquisition request to the technology platform, receive a response message returned by the technology platform based on the data acquisition request, and when the response message is a success message, access the business subsystem of the technology platform to obtain the original data from the business subsystem of the technology platform.

[0074] For example, when the target platform is an insurance platform, access the insurance platform, send a data acquisition request to the insurance platform, receive a response message returned by the insurance platform based on the data acquisition request, and when the response message is a success message, access the business subsystem of the insurance platform and obtain the original data from the business subsystem of the insurance platform.

[0075] For example, when the target platform is a medical platform, access the medical platform, send a data acquisition request to the medical platform, receive a response message returned by the medical platform based on the data acquisition request, and when the response message is a success message, access the business subsystem of the medical platform to obtain the original data from the business subsystem of the medical platform.

[0076] S22, obtain common information and additional information from the original data;

[0077] The obtaining of common information and additional information from the original data includes:

[0078] Obtaining a preset format of the original data, parsing the original data using a parsing method corresponding to the preset format, and obtaining the parsed original data;

[0079] Obtain data records from the parsed raw data, and obtain common information and additional information in the data records.

[0080] Among them, shared information reflects the common characteristics or patterns in the original data. By analyzing shared information, we can extract universal conclusions and patterns, provide macro-level guidance for the overall business, help users grasp the general direction, formulate unified strategies, and reduce the blindness of decision-making.

[0081] Among them, additional information reflects the differences and individuality between different data records. Mining additional information is conducive to discovering special situations, potential needs or new business growth points, thereby providing a basis for personalized services, precision marketing, etc., and improving customer satisfaction and market competitiveness.

[0082] Among them, obtaining common information and additional information in data records and saving common information and additional information can avoid analytical deviations caused by data loss. When facing data tracing, auditing, etc., complete data can be checked, ensuring business compliance and decision-making reliability, and safeguarding the stable development of the enterprise.

[0083] S23, obtaining the field value of the additional information and obtaining the type prefix corresponding to the field value;

[0084] Among them, the type prefix is ​​a prefix character or character combination used in programming naming specifications to identify the data type or specific attribute of identifiers such as variables and constants.

[0085] Among them, the type prefix provides a clear identification for the field value. When the program reads and processes these field values, it can quickly and accurately determine their data type based on the prefix, and then parse them according to the preset correct logic and rules, avoiding parsing errors caused by type ambiguity or confusion, and greatly improving the accuracy and reliability of data processing.

[0086] S24, combining the type prefix and the field value in a preset manner to obtain a string;

[0087] The method combines the type prefix and the field value in a preset manner to obtain a string, including:

[0088] Concatenate the type prefix and the field value to obtain the concatenated value;

[0089] Access the preset file, load the programming tool from the preset file, and convert the concatenated value into a string using the built-in conversion function of the programming tool.

[0090] Among them, the type prefix and field value are spliced ​​together to obtain the spliced ​​value. Since the spliced ​​value itself carries the type prefix, the program can quickly and accurately identify the type of the field value based on the spliced ​​value, and then parse and operate according to the correct logic, greatly improving processing efficiency and accuracy.

[0091] Strings are a universal and highly compatible format. Using built-in conversion functions in programming tools to convert spliced ​​values ​​into strings ensures smooth transmission and stable storage of spliced ​​values ​​across different platforms and systems, avoiding loss or errors in spliced ​​values ​​due to format differences.

[0092] S25, performing escape processing on the character string to obtain a processed character string;

[0093] Strings often contain special characters, such as quotes and newlines. Left unhandled, these characters can disrupt code syntax or corrupt data structures. Escape processing converts special characters into ordinary character representations using specific symbols. This avoids syntax errors caused by these characters and ensures correct string parsing and execution.

[0094] For ease of explanation, the following examples are given:

[0095] For example, the string is: Nanshan District, Science and Technology Park Area.

[0096] This string contains a comma. If it is not escaped, it will cause a string write error, splitting the complete address into two invalid fields. The two invalid fields are: 1. Nanshan District; 2. Science and Technology Park Area. Therefore, the string needs to be escaped to obtain the processed string.

[0097] The processed string is: "Nanshan District, Science and Technology Park Area". The CSV format file writer can correctly recognize that this is a complete field value including commas and will not cause string writing errors.

[0098] S26, adding common fields and additional fields to the CSV format file, storing common information through the common fields, and storing processed character strings through the additional fields, to obtain a storage file of the original data.

[0099] The full Chinese name of the CSV format file is: Comma Separated Values ​​file, and the full English name of the CSV format file is: Comma-Separated Values.

[0100] Among them, since the original data has a lot of characteristics of similar structures and only slight content differences, if the complete information of the original data is stored directly, the repeated placeholder part will take up a lot of storage space, causing unnecessary redundancy. The processed character string is the difference content, and the additional field is used to record the difference content to avoid the repeated storage of placeholders. Therefore, the storage space occupied by the original data will not be increased, thus solving the technical problem of how to store the original data.

[0101] Wherein, after adding common fields and additional fields to the CSV format file, storing common information through the common fields, storing processed character strings through the additional fields, and obtaining a storage file of the original data, the data storage method includes:

[0102] Read the preset upload time and determine whether the current time is the upload time; if the current time is the upload time, obtain the push instruction, execute the push instruction, and push the stored file to the data warehouse corresponding to the target platform.

[0103] In the embodiment of the present invention, the beneficial effects are in two aspects. On the one hand, common fields and additional fields are added to the CSV format file, common information is stored in the common fields, and escaped strings are stored in the additional fields to obtain a storage file of the original data. Since the escaped strings are stored on demand using the additional fields, the situation of repeated storage of placeholders is avoided, and the storage space occupied by the original data is not increased. Therefore, the technical problem of how to store the original data is solved, the storage amount of the original data can be effectively reduced, and the utilization efficiency of storage resources is improved. On the other hand, it will not be affected by human intervention, which is conducive to improving the reliability of the storage file of the original data. Please refer to Figure 3 , Figure 3 yes Figure 2 A specific implementation flow diagram of step S23 is described in detail as follows:

[0104] S31, obtaining the field value of the additional information, querying the field value from the query table, and obtaining the query result;

[0105] S32, obtaining the data type of the field value from the query result, and determining the type prefix corresponding to the field value according to the data type of the field value.

[0106] In an embodiment of the present invention, the type prefix provides a clear identification for the field value. The type prefix can clearly inform the reading program of the type of each field value, so that the program can parse the data in a correct manner, avoid parsing errors caused by ambiguous data types, and ensure that the field values ​​are accurately imported into the CSV format file.

[0107] See also Figure 4 , Figure 4 yes Figure 2 A specific implementation flow diagram of step S25 is described in detail as follows:

[0108] S41, determining whether the character string contains special characters, where special characters include one or a combination of comma, quotation mark, line feed, carriage return, tab, and backslash;

[0109] S42, when the character string contains special characters, input the character string into the escape module, and perform escape processing on the character string through the escape module to obtain a processed character string.

[0110] In an embodiment of the present invention, when a string contains special characters, the string is input into an escape module, and the string is escaped by the escape module to obtain a processed string, which can eliminate the risk of syntax conflicts in cross-system interactions, thereby ensuring the integrity and compatibility of data during transmission or storage.

[0111] See also Figure 5 , Figure 5 yes Figure 2 A specific implementation flow diagram of step S26 is described in detail as follows:

[0112] S51, accessing a data processing tool, exporting a CSV format file from the data processing tool, and adding common fields and additional fields to the first row of the CSV format file;

[0113] Among them, data processing tools are professional software or systems used to perform a series of operations such as collecting, cleaning, converting, analyzing, and visualizing various types of data.

[0114] Access the data processing tool, export a CSV format file from the data processing tool, and add common fields and additional fields to the first row of the CSV format file, including:

[0115] Access the data processing tool through a preset network, export a CSV format file from the data processing tool, and add common fields and additional fields to the first row of the CSV format file. The preset network includes one of a 3G network, a 4G network, a 5G network, and a WIFI network, or a combination thereof.

[0116] S52 , storing the field values ​​of the common data in the cells of the column where the common fields are located, and storing the processed character strings in the cells of the column where the additional fields are located, to obtain a storage file of the original data.

[0117] In an embodiment of the present invention, the field values ​​of the common data are stored in the cells of the column where the common fields are located, and the processed character strings are stored in the cells of the column where the additional fields are located, so as to obtain a storage file of the original data. Since the escaped character strings are stored on demand using the additional fields, the situation of repeated storage of placeholders is avoided, and the storage space occupied by the original data is not increased. Therefore, the technical problem of how to store the original data is solved, the storage amount of the original data can be effectively reduced, and the utilization efficiency of storage resources is improved.

[0118] See also Figure 6 , Figure 6 FIG. 1 is a schematic diagram of a structure of a data storage device according to an embodiment of the present invention. Figure 6 As shown, the data storage device includes a first acquisition module 101, a second acquisition module 102, a third acquisition module 103, a combination module 104, a processing module 105, and a storage module 106. The functional modules are described in detail as follows:

[0119] The first acquisition module 101 is used to access the target platform and obtain original data from the target platform;

[0120] A second acquisition module 102 is used to acquire common information and additional information from the original data;

[0121] The third acquisition module 103 is used to obtain the field value of the additional information and obtain the type prefix corresponding to the field value;

[0122] A combining module 104 is configured to combine the type prefix and the field value in a preset manner to obtain a character string;

[0123] The processing module 105 is used to perform escape processing on the character string to obtain a processed character string;

[0124] The storage module 106 is used to add common fields and additional fields to the CSV format file, store common information through the common fields, and store processed character strings through the additional fields to obtain a storage file of the original data.

[0125] In one embodiment, the second acquisition module 102 includes:

[0126] The first acquisition subunit is used to acquire a preset format of the original data, and parse the original data using a parsing method corresponding to the preset format to obtain the parsed original data;

[0127] The second acquisition subunit is used to acquire data records from the parsed original data, and acquire common information and additional information in the data records.

[0128] In one embodiment, the third acquisition module 103 includes:

[0129] The third acquisition subunit is used to obtain the field value of the additional information, query the field value from the query table, and obtain the query result;

[0130] The determination subunit is used to obtain the data type of the field value from the query result, and determine the type prefix corresponding to the field value based on the data type of the field value.

[0131] In one embodiment, the combining module 104 includes:

[0132] The splicing subunit is used to splice the type prefix and the field value to obtain the spliced ​​value;

[0133] The conversion subunit is used to access the preset file, load the programming tool from the preset file, and convert the spliced ​​value into a string through the built-in conversion function of the programming tool.

[0134] In one embodiment, the processing module 105 includes:

[0135] The judgment subunit is used to judge whether the string contains special characters, and special characters include one or a combination of commas, quotation marks, newline characters, carriage returns, tabs, and backslashes;

[0136] The processing subunit is used to input the character string into the escape module when the character string contains special characters, and perform escape processing on the character string through the escape module to obtain a processed character string.

[0137] In one embodiment, the storage module 106 includes:

[0138] Add a subunit for accessing the data processing tool, exporting a CSV format file from the data processing tool, and adding common fields and additional fields to the first row of the CSV format file;

[0139] The storage subunit is used to store the field values ​​of the common data through the cells in the column where the common fields are located, and to store the processed character strings through the cells in the column where the additional fields are located, so as to obtain a storage file of the original data.

[0140] In one embodiment, the data storage device further comprises:

[0141] The reading module is used to read the preset upload time and determine whether the current time is the upload time;

[0142] The push module is used to obtain the push instruction if the current time is the upload time, execute the push instruction, and push the storage file to the data warehouse corresponding to the target platform.

[0143] In an embodiment of the present invention, the beneficial effects lie in two aspects. On the one hand, common fields and additional fields are added to a CSV format file, common information is stored through the common fields, and escaped strings are stored through the additional fields to obtain a storage file of the original data. Since the escaped strings are stored on demand using additional fields, repeated storage of placeholders is avoided, and the storage space occupied by the original data is not increased, thereby solving the technical problem of how to store the original data, effectively reducing the storage amount of the original data, and improving the utilization efficiency of storage resources. On the other hand, it will not be affected by human intervention, and is therefore conducive to improving the reliability of the storage file of the original data.

[0144] For the specific definition of the data storage device, please refer to the definition of the data storage method above, which will not be repeated here.

[0145] Each module in the aforementioned data storage device may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0146] See also Figure 7 , Figure 7 This is a schematic diagram of a computer device in one embodiment of the present invention. In one embodiment, a computer device is provided. The computer device may be a server device, and its internal structure diagram may be as shown in FIG. Figure 7 The computer device includes a processor, a memory, a network interface and a database connected via a system bus.

[0147] The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. When executed by the processor, the computer program can implement the functions or steps of a format file-based data storage method on the server device.

[0148] See also Figure 8 , Figure 8 This is another structural diagram of a computer device in one embodiment of the present invention. In one embodiment, a computer device is provided. The computer device may be a client device, and its internal structure diagram may be as shown in FIG. Figure 8As shown. The computer device includes a processor, memory, a network interface, a display screen, and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. When the computer program is executed by the processor, the functions or steps of a format file-based data storage method can be implemented on the client device.

[0149] In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor.

[0150] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant description in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0151] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0152] The above description and accompanying drawings sufficiently illustrate the embodiments of the present disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent only possible variations. Unless explicitly required, individual components and functions are optional, and the order of operations may vary. Portions and subsamples of some embodiments may be included in or replaced with portions and subsamples of other embodiments. Moreover, the terms used in this application are only used to describe the embodiments and are not used to limit the claims.

[0153] In this document, each embodiment may focus on the differences from other embodiments, and similar parts between the embodiments can be referenced. For methods, products, etc. disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, then the relevant parts can be referenced to the description of the method section.

[0154] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. Technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present disclosure. Technicians can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0155] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units can be merely a logical functional division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some sub-samples can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to implement this embodiment. In addition, the functional units in the embodiments of the present disclosure can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0156] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architectures, functions and operations of the systems, methods and computer program products according to the embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of the code, and the module, program segment or part of the code contains one or more executable instructions for implementing the specified logical functions. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, which can depend on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in an order different from that disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flow charts, and combinations of blocks in the block diagrams and / or flow charts, may be implemented using a dedicated hardware-based system that performs the specified functions or actions, or may be implemented using a combination of dedicated hardware and computer instructions. In addition, any software tools or components not developed by our company that appear in the embodiments of this application are for illustrative purposes only and do not represent actual use.

Claims

1. A data storage method based on format files, characterized in that: include: Access the target platform and obtain raw data from the target platform; Obtain common and additional information from original data; Get the field value of the additional information and the type prefix corresponding to the field value; Use the preset method to combine the type prefix and field value to obtain a string; Escape the string to obtain the processed string; Add common fields and additional fields to the CSV format file, use the common fields to store common information, and use the additional fields to store processed character strings to obtain a storage file for the original data.

2. The data storage method according to claim 1, wherein: The acquisition of common information and additional information from the original data includes: Obtaining a preset format of the original data, parsing the original data using a parsing method corresponding to the preset format, and obtaining the parsed original data; Obtain data records from the parsed raw data, and obtain common information and additional information in the data records.

3. The data storage method according to claim 1, wherein: The method of obtaining the field value of the additional information and obtaining the type prefix corresponding to the field value includes: Get the field value of the additional information, query the field value from the query table, and get the query result; Get the data type of the field value from the query result, and determine the type prefix corresponding to the field value based on the data type of the field value.

4. The data storage method according to claim 1, wherein: The preset method is used to combine the type prefix and the field value to obtain a string, including: Concatenate the type prefix and the field value to obtain the concatenated value; Access the preset file, load the programming tool from the preset file, and convert the concatenated value into a string using the built-in conversion function of the programming tool.

5. The data storage method according to claim 1, wherein: The character string is escaped to obtain a processed character string, including: Determine whether the string contains special characters. Special characters include comma, quotation mark, newline, carriage return, tab, backslash, or any combination thereof. When a character string contains special characters, the character string is input into the escape module, and the escape module is used to escape the character string to obtain a processed character string.

6. The data storage method according to claim 1, wherein: The method of adding common fields and extra fields to the CSV format file, storing common information through the common fields and storing processed character strings through the extra fields, and obtaining a storage file of the original data includes: Access the data processing tool, export a CSV format file from the data processing tool, and add common fields and additional fields to the first row of the CSV format file; The field values ​​of the common data are stored in the cells of the column where the common fields are located, and the processed character strings are stored in the cells of the column where the additional fields are located, thereby obtaining a storage file of the original data.

7. The data storage method according to any one of claims 1 to 6, characterized in that: After adding common fields and additional fields to the CSV format file, storing common information through the common fields, and storing processed character strings through the additional fields to obtain a storage file of the original data, the data storage method includes: Read the preset upload time and determine whether the current time is the upload time; If the current time is the upload time, the push instruction is obtained, the push instruction is executed, and the storage file is pushed to the data warehouse corresponding to the target platform.

8. A data storage device based on a format file, characterized in that: include: The first acquisition module is used to access the target platform and obtain original data from the target platform; A second acquisition module is used to obtain common information and additional information from the original data; The third acquisition module is used to obtain the field value of the additional information and the type prefix corresponding to the field value; The combination module is used to combine the type prefix and field value in a preset way to obtain a string; The processing module is used to perform escape processing on the character string to obtain the processed character string; The storage module is used to add common fields and additional fields to the CSV format file, store common information through the common fields, and store processed character strings through the additional fields to obtain a storage file of the original data.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the data storage method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the data storage method according to any one of claims 1 to 7 are implemented.

Images