Mass data form storage method and device, electronic equipment and storage medium

Abstract

The invention relates to a mass data form storage method and device, electronic equipment and a storage medium, and belongs to the technical field of data storage.The mass data form storage method comprises the steps that a preset number of pieces of data with the update time closest to the current time in a data form is determined as hot storage data, determining other data except the hot storage data in the data form as cold storage data; and automatically fragmenting and storing the hot storage data based on a non-relational database, and storing the cold storage data based on a cloud storage service. According to the invention, while the massive data form storage is realized, the complexity of the storage scheme is reduced, and the maintenance cost is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a method, device, electronic device and storage medium for storing a mass data form. Background Art

[0002] Existing relational databases, such as MySQL databases, split and store data into separate libraries and tables when storing data. However, when faced with large amounts of basic data and fast-growing data, existing relational databases still have many limitations.

[0003] Relational databases are not suitable for scenarios where the structure of massive data tables changes frequently. Physical sub-libraries and sub-tables cannot dynamically support long-term massive growth of data, are not conducive to node-side data access, require manual intervention and adjustment, and are difficult to maintain.

[0004] Therefore, how to reduce the complexity of storing massive data forms and reduce maintenance costs has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of this, it is necessary to provide a method, device, electronic device and storage medium for storing massive data forms, so as to solve the problems of complexity and high maintenance cost of current massive data form storage solutions.

[0006] In order to solve the above problems, the present invention provides a method for storing a mass data form, comprising: Determine a preset number of data in the data form whose update time is closest to the current time as hot storage data, and determine other data in the data form except the hot storage data as cold storage data; The hot storage data is automatically sharded and stored based on a non-relational database, and the cold storage data is stored based on a cloud storage service.

[0007] In a possible implementation, automatically sharding and storing the hot storage data based on a non-relational database includes: Automatically sharding the hot storage data based on a non-relational database; Routes read and write operations to the corresponding shards and stores metadata for multiple shards based on the configuration service; The hot storage data is stored in a plurality of shards.

[0008] In a possible implementation, the non-relational database includes a MongoDB database.

[0009] In a possible implementation, storing the cold storage data based on a cloud storage service includes: The cold storage data is stored based on the log service in the cloud storage service.

[0010] In a possible implementation, the method further includes: Determining the data type requested by the data form query request information based on the received data form query request information; In the case where the data type of the data form query request information request is hot storage data, providing a real-time query service; In the case where the data type of the data form query request information request is cold storage data, a non-real-time query service is provided.

[0011] In a possible implementation, when the data type of the data form query request information request is hot storage data, providing a real-time query service includes: Determining a target shard in metadata of a plurality of shards based on the data form query request information; The data requested by the data form query request information is obtained from the target shard.

[0012] In a possible implementation, when the data type requested by the data form query request information is cold storage data, providing a non-real-time query service includes: Sending the data form query request information to a cloud storage service; Receive target data sent by the cloud storage service, where the target data includes data requested by the data form query request information.

[0013] The present invention also provides a mass data form storage device, comprising: A determination module, used to determine a preset number of data in the data form whose update time is closest to the current time as hot storage data, and determine other data in the data form except the hot storage data as cold storage data; The storage module is used to automatically slice and store the hot storage data based on a non-relational database, and to store the cold storage data based on a cloud storage service.

[0014] The present invention also provides an electronic device, including a memory and a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for storing a mass data form as described above is implemented.

[0015] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for storing a mass data form as described above is implemented.

[0016] The beneficial effects of the present invention are as follows: the method, device, electronic device and storage medium for storing a massive data form provided by the present invention reduce the pressure of real-time data storage by separating the data in the data form into hot and cold data, thereby reducing the complexity of data storage, and automatically sharding and storing the hot storage data through a non-relational database, and storing the cold storage data through a cloud storage service, thereby further reducing the complexity of data storage, and reducing the difficulty of data maintenance through automatic sharding and cloud storage services. While realizing the storage of massive data forms, the present invention reduces the complexity of storage solutions and reduces maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of a flow chart of an embodiment of a method for storing a mass data form provided by the present invention; Figure 2 A schematic diagram of data flow in an embodiment of a cold and hot storage separation solution provided by the present invention; Figure 3 A schematic diagram of the structure of an embodiment of a mass data form storage device provided by the present invention; Figure 4 A schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0019] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0020] In the description of the present invention, reference to an "embodiment" means that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive of other embodiments. It is explicitly and implicitly understood by those skilled in the art that the described embodiments may be combined with other embodiments.

[0021] Existing relational databases, such as MySQL databases, split and store data into separate libraries and tables when storing data. However, when faced with large amounts of basic data and fast-growing data, existing relational databases still have many limitations.

[0022] Relational databases are not suitable for scenarios where the structure of massive data tables changes frequently. Physical sub-libraries and sub-tables cannot dynamically support long-term massive growth of data, are not conducive to node-side data access, require manual intervention and adjustment, and are difficult to maintain.

[0023] Moreover, from the perspective of service stability, the existing massive data form storage solutions cannot well meet the future business development.

[0024] In order to find a solution that can well support the continuous expansion of data and does not require manual intervention to realize the storage of massive data forms, the present invention proposes a massive data form storage method.

[0025] The specific embodiments are described in detail below: A specific embodiment of the present invention discloses a method for storing a large amount of data in a form. Figure 1 Come and see, Figure 1 The flowchart of an embodiment of a method for storing a mass data form provided by the present invention includes step S101 and step S102, wherein: In step S101, a preset number of data pieces in a data form whose update time is closest to the current time are determined as hot storage data, and other data in the data form except the hot storage data are determined as cold storage data; In step S102, the hot storage data is automatically sharded and stored based on a non-relational database, and the cold storage data is stored based on a cloud storage service.

[0026] During implementation, the data in the data form can first be divided into hot storage data and cold storage data to reduce the pressure of real-time storage.

[0027] When classifying data types, a preset number of data in the data form whose update time is closest to the current time can be determined as hot storage data. The preset number can be dynamically adjusted according to the amount of data in the data form, and the present invention does not make specific limitations on this.

[0028] After the hot storage data is determined, other data in the data form except the hot storage data may be determined as cold storage data.

[0029] After completing the data type classification, hot storage data can be automatically sharded and stored based on the non-relational database, and cold storage data can be stored based on the cloud storage service to achieve the storage of massive data forms.

[0030] The method for storing a massive data form provided by the present invention can be applied to the storage scenario of a massive print data form, and can also be applied to the storage scenario of a massive map data form, and the present invention does not make any specific limitation on this.

[0031] Compared with the prior art, the method for storing massive data forms provided in this embodiment reduces the pressure of real-time data storage by separating the data in the data forms into hot and cold data, thereby reducing the complexity of data storage, and automatically shards and stores the hot storage data through a non-relational database, and stores the cold storage data through a cloud storage service, thereby further reducing the complexity of data storage, and reducing the difficulty of data maintenance through automatic sharding and cloud storage services. While realizing the storage of massive data forms, the present invention reduces the complexity of storage solutions and reduces maintenance costs.

[0032] Exemplarily, the automatically sharding and storing the hot storage data based on the non-relational database includes: Automatically sharding the hot storage data based on a non-relational database; Routes read and write operations to the corresponding shards and stores metadata for multiple shards based on the configuration service; The hot storage data is stored in a plurality of shards.

[0033] Specifically, when hot storage data is automatically sharded and stored based on a non-relational database, the hot storage data can be automatically sharded through the non-relational database first, and then the read and write operations are routed to the corresponding shards, and then the metadata of multiple shards are stored according to the configuration service, and finally the hot storage data is stored in multiple shards.

[0034] Exemplarily, the non-relational database includes a MongoDB database.

[0035] Specifically, when hot storage data is automatically sharded and stored, the non-relational database used may be a MongoDB database. In addition, other non-relational databases capable of automatic sharding may also be used, and the present invention does not specifically limit this.

[0036] For example, when using the MongoDB database to automatically shard and store hot storage data, Mongos can be responsible for routing read and write operations to the corresponding Shard, storing Shard metadata through ConfigServer, and storing database data through Shard.

[0037] Exemplarily, storing the cold storage data based on the cloud storage service includes: The cold storage data is stored based on the log service in the cloud storage service.

[0038] Specifically, when the cold storage data is stored according to the cloud storage service, the cold storage data may be stored through the Simple Log Service (SLS) in the cloud storage service.

[0039] For example, you can use the SLS log service of Alibaba Cloud or other cloud storage services to store cold storage data.

[0040] Exemplarily, the method further includes: Determining the data type requested by the data form query request information based on the received data form query request information; In the case where the data type of the data form query request information request is hot storage data, providing a real-time query service; In the case where the data type of the data form query request information request is cold storage data, a non-real-time query service is provided.

[0041] Specifically, after receiving the data form query request information sent by the user, firstly, the data type requested by the data form query request information can be determined according to the received data form query request information.

[0042] When the data type requested by the data form query request information is hot storage data, real-time query service can be provided to the user; when the data type requested by the data form query request information is cold storage data, non-real-time query service can be provided to the user.

[0043] Exemplarily, when the data type of the data form query request information request is hot storage data, providing a real-time query service includes: Determining a target shard in metadata of a plurality of shards based on the data form query request information; The data requested by the data form query request information is obtained from the target shard.

[0044] Specifically, when providing real-time query services to users, the target shard can first be determined in the metadata of multiple shards according to the data form query request information, and then the data requested by the data form query request information is obtained from the target shard, and then the data is sent to the user end.

[0045] Exemplarily, when the data type of the data form query request information request is cold storage data, providing a non-real-time query service includes: Sending the data form query request information to a cloud storage service; Receive target data sent by the cloud storage service, where the target data includes data requested by the data form query request information.

[0046] Specifically, when providing non-real-time query services to users, the data form query request information can be forwarded to the cloud storage service, and the cloud storage service can be waited for to return the target data, which may include the data requested by the data form query request information, and then the data can be sent to the user end.

[0047] The technical solution of the present invention is better described below with reference to a specific embodiment: The following takes the imprint data generated by user printing as an example to introduce the mass data form storage method provided by the present invention. Figure 2 Come and see, Figure 2 This is a data flow diagram of an embodiment of a cold and hot storage separation solution provided by the present invention. The specific steps of the solution are as follows: 1. Define the user's latest first m trace data as hot data for direct query by the user; data other than hot data is cold data and is not directly provided to the user. Provide background query function to meet the user's non-real-time query needs.

[0048] 2. Hot data storage uses MongoDB non-relational database and shard cluster technology. MongoDB shard cluster supports automatic sharding, which can get rid of manual shard management. The cluster automatically splits data and achieves load balancing. The cluster specifically includes three components: Mongos, ConfigServer, and Shard. Mongos is responsible for routing read and write operations to the corresponding Shard, ConfigServer is responsible for storing Shard metadata, and Shard is responsible for storing database data.

[0049] Cold data storage uses Alibaba Cloud's SLS log service, which not only provides one-stop data collection, processing, query and analysis functions, but also directly provides hot and cold data storage services. It has a powerful and expressive query language, supports real-time query and analysis of PB-level data, and provides PB-level data elastic scaling capabilities.

[0050] 3. The process of application server requesting hot data is as follows: (1) The upper-layer business end accesses the data read and write interface.

[0051] (2) The application server forwards the received read and write requests to the Mongos component.

[0052] (3) The Mongos component obtains the target Shard from the locally cached Shard cluster metadata and sends the read and write requests to the target Shard.

[0053] (4) The target Shard feeds the processed results back to Mongos, which aggregates the processed results and returns them to the application server.

[0054] Finally, the upper-level business end obtains the required data normally, and does not need to care about the sharding structure of the underlying data when making requests.

[0055] The present invention first splits the data into hot and cold data for separate storage, saving the data resource storage cost, and then deploys the hot data through the MongoDB non-relational database, using a sharding cluster to store it, effectively avoiding the drawbacks of manual physical sharding. The upper-level business end no longer needs to care about the specific sharding structure, as MongoDB automatically handles sharding routing and storage. In addition, as a non-relational database, MongoDB is also very friendly to data structure changes caused by business changes, and does not require any additional maintenance work.

[0056] The embodiment of the present invention also provides a mass data form storage device, combined with Figure 3 Come and see, Figure 3 This is a schematic diagram of the structure of an embodiment of a mass data form storage device provided by the present invention. The mass data form storage device 300 includes: A determination module 301 is used to determine a preset number of data in a data form whose update time is closest to the current time as hot storage data, and determine other data in the data form except the hot storage data as cold storage data; The storage module 302 is used to automatically slice and store the hot storage data based on a non-relational database, and to store the cold storage data based on a cloud storage service.

[0057] The specific implementation of each module of the massive data form storage device can refer to the description of the massive data form storage method mentioned above, and has similar beneficial effects, which will not be repeated here.

[0058] It should be noted that the mass data form storage device can be set up at the user backend where users gather, or can be set up at the server end that provides storage services, and the present invention does not make specific limitations on this.

[0059] The embodiment of the present invention further provides an electronic device, Figure 4 Come and see, Figure 4This is a structural diagram of an embodiment of an electronic device provided by the present invention. The electronic device 400 includes a processor 401, a memory 402, and a computer program stored in the memory 402 and executable on the processor 401. When the processor 401 executes the program, the mass data form storage method described above is implemented.

[0060] As a preferred embodiment, the electronic device 400 further includes a display 403 for displaying that the processor 401 executes the method for storing a massive amount of data form as described above.

[0061] Exemplarily, the computer program may be divided into one or more modules / units, one or more modules / units are stored in the memory 402, and executed by the processor 401 to complete the present invention. One or more modules / units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device 400. For example, the computer program may be divided into the determination module 301 and the storage module 302 in the above embodiment, and the specific functions of each module are as described above, and are not described one by one here.

[0062] The electronic device 400 may be a desktop computer, a notebook, a PDA or a smart phone with an adjustable camera module.

[0063] Among them, the processor 401 may be an integrated circuit chip with signal processing capabilities. The above-mentioned processor 401 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0064] The memory 402 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electric erasable programmable read-only memory (EEPROM), etc. The memory 402 is used to store programs, and the processor 401 executes the program after receiving the execution instruction. The process definition method disclosed in any of the embodiments of the present invention may be applied to the processor 401 or implemented by the processor 401.

[0065] The display 403 may be an LCD display screen or an LED display screen, for example, a display screen on a mobile phone.

[0066] Understandably, Figure 4 The structure shown is only a schematic diagram of the structure of the electronic device 400. The electronic device 400 may also include Figure 4 More or fewer components as shown. Figure 4 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0067] The electronic device provided according to the above embodiment of the present invention can be implemented with reference to the content specifically described in the method for storing massive data forms according to the present invention, and has similar beneficial effects as the method for storing massive data forms, which will not be described in detail here.

[0068] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method for storing a mass data form as described above is implemented.

[0069] Generally speaking, the computer instructions for implementing the method of the present invention can be carried by any combination of one or more computer-readable storage media. Non-transitory computer-readable storage media can include any computer-readable media except for the signal itself that is temporarily propagating.

[0070] Computer-readable storage media may be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0071] Computer program codes for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages, in particular, Python suitable for neural network computing and platform frameworks based on TensorFlow, PyTorch, etc. may be used. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0072] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0073] The present invention discloses a method, device, electronic device and storage medium for storing massive data forms. The method separates hot and cold data in a data form to reduce the pressure of real-time data storage and the complexity of data storage. The hot storage data is automatically sliced ​​and stored through a non-relational database, and the cold storage data is stored through a cloud storage service, which further reduces the complexity of data storage. The automatic slicing and cloud storage service reduce the difficulty of data maintenance. The present invention reduces the complexity of storage solutions and reduces maintenance costs while realizing the storage of massive data forms.

[0074] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for storing a large amount of data in a form, characterized in that: include: Determine a preset number of data in the data form whose update time is closest to the current time as hot storage data, and determine other data in the data form except the hot storage data as cold storage data; The hot storage data is automatically sharded and stored based on a non-relational database, and the cold storage data is stored based on a cloud storage service.

2. The method for storing a large amount of data in a table according to claim 1, characterized in that: The automatically sharding and storing the hot storage data based on the non-relational database includes: Automatically sharding the hot storage data based on a non-relational database; Routes read and write operations to the corresponding shards and stores metadata for multiple shards based on the configuration service; The hot storage data is stored in a plurality of shards.

3. The method for storing a large amount of data in a table according to claim 2, characterized in that: The non-relational database includes a MongoDB database.

4. The method for storing a large amount of data in a form according to claim 1, characterized in that: The storing of the cold storage data based on the cloud storage service includes: The cold storage data is stored based on the log service in the cloud storage service.

5. The method for storing a mass data form according to claim 2, characterized in that: The method further comprises: Determining the data type requested by the data form query request information based on the received data form query request information; In the case where the data type of the data form query request information request is hot storage data, providing a real-time query service; In the case where the data type of the data form query request information request is cold storage data, a non-real-time query service is provided.

6. The method for storing a mass data form according to claim 5, characterized in that: When the data type of the data form query request information request is hot storage data, providing a real-time query service includes: Determining a target shard in metadata of a plurality of shards based on the data form query request information; The data requested by the data form query request information is obtained from the target shard.

7. The method for storing a mass data form according to claim 5, characterized in that: When the data type requested by the data form query request information is cold storage data, providing a non-real-time query service includes: Sending the data form query request information to a cloud storage service; Receive target data sent by the cloud storage service, where the target data includes data requested by the data form query request information.

8. A mass data form storage device, characterized in that: include: A determination module, used to determine a preset number of data in the data form whose update time is closest to the current time as hot storage data, and determine other data in the data form except the hot storage data as cold storage data; The storage module is used to automatically slice and store the hot storage data based on a non-relational database, and to store the cold storage data based on a cloud storage service.

9. An electronic device, characterized in that: The invention comprises a memory and a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for storing a mass data form according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method for storing a mass data form as claimed in any one of claims 1 to 7 is implemented.

Images