A data storage method, device and electronic device based on multiple databases

Abstract

The present invention discloses a data storage method based on multiple databases, the method comprising: obtaining new data of a user, obtaining company information corresponding to the user; converting the new data and storing it in a MySQL storage system; asynchronously forwarding the new data to an automatic switching scheduling system, and judging whether the company data corresponding to the user has been migrated to an Elasticsearch storage system; if yes, storing the new data in the Elasticsearch storage system; if not, judging whether the data volume of the company exceeds a preset threshold; if yes, inserting the new data into a MongoDB storage system, and then copying the new data from the MongoDB storage system into the Elasticsearch storage system; if no, directly inserting the new data into the MongoDB storage system.

Description

Technical Field

[0001] The present application relates to the field of big data storage, and in particular to a data storage method, device and electronic device based on multiple databases. Background Art

[0002] Currently, more and more companies are using CRM (Customer Relationship Management) systems. As one of the important modules of the CRM system, the customer management module usually uses different fields in different companies. As a general SaaS service, the customer management module requires high scalability and high-performance search features.

[0003] In the prior art, some companies use the MySQL storage system as the main storage for user data, which has a low storage cost. However, due to the limitation of the MySQL B+ index, the query performance of arbitrary fields is poor when the data volume is large. In addition, some companies use the MongoDB storage system as the main storage for user data, which has a low storage cost and can make full use of the MongoDB BSON arbitrary custom field feature. However, due to the limitation of the index structure, the query efficiency is not good when the number of companies is large. Some companies also use the Elasticsearch storage system as the main storage for user data, which has high query performance, supports high scalability, and supports large data volumes, but the storage cost is high, resulting in waste of resources and poor transaction support. Summary of the invention

[0004] The technical problem to be solved by the embodiments of the present application is to provide a data storage method, device and electronic device based on multiple databases to solve the technical problem that the existing user data storage system cannot simultaneously support high performance of large data volume query and save costs as much as possible.

[0005] To achieve the above objectives, the present application embodiment adopts the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a data storage method based on multiple databases, the method comprising:

[0007] Obtain the user's newly added data and obtain the company information corresponding to the user;

[0008] The newly added data is converted and stored in the MySQL storage system;

[0009] Asynchronously forwarding the newly added data to the automatic switching scheduling system to determine whether the company data corresponding to the user has been migrated to the Elasticsearch storage system;

[0010] If yes, store the newly added data in the Elasticsearch storage system; if no, determine whether the data volume of the company exceeds a preset threshold;

[0011] If it exceeds, after inserting the newly added data into the MongoDB storage system, the newly added data is copied from the MongoDB storage system into the Elasticsearch storage system; if it does not exceed, the newly added data is directly inserted into the MongoDB storage system.

[0012] In a second aspect, an embodiment of the present application provides a data storage device based on multiple databases, the device comprising:

[0013] A first acquisition unit, used to acquire new data of a user and company information corresponding to the user;

[0014] The first storage unit is used to convert the newly added data and store it in the MySQL storage system;

[0015] A first forwarding unit, configured to asynchronously forward the newly added data to an automatic switching scheduling system;

[0016] A first judgment unit is used to judge whether the company data corresponding to the user has been migrated to the Elasticsearch storage system, and if so, store the newly added data in the Elasticsearch storage system;

[0017] The second judgment unit is used to judge whether the data volume of the company exceeds a preset threshold. If so, after inserting the new data into the MongoDB storage system, the new data is copied from the MongoDB storage system into the Elasticsearch storage system. If not, the new data is directly inserted into the MongoDB storage system.

[0018] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a set to be coded, or an instruction set, and the at least one instruction, the at least one program, the set to be coded, or the instruction set is executed by the processor to implement the data storage method based on multiple databases as described in the first aspect above.

[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores at least one instruction, at least one program, a set to be coded, or an instruction set, and the at least one instruction, the at least one program, the set to be coded, or the instruction set is executed by a processor to implement the data storage method based on multiple databases as described in the first aspect above.

[0020] The beneficial effects of the embodiments of the present application are as follows: the embodiments of the present application provide a data storage method, device and electronic device based on multiple databases, which dynamically implements data storage switching based on the company's data volume and the respective advantages of the three databases: MySQL transaction characteristics, MongoDB low-cost storage, and Elasticsearch high-performance query, without the need for human intervention, thereby achieving the technical effects of high-performance query and low-cost storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of a flow chart of a data storage method based on multiple databases provided in one embodiment of the present application;

[0022] Figure 2 A schematic diagram of the structure of a data storage device based on multiple databases provided in one embodiment of the present application;

[0023] Figure 3 A schematic diagram of the structure of an electronic device provided for one embodiment of the present application. DETAILED DESCRIPTION

[0024] The technical solution of the present application is further described in detail below through specific embodiments and in combination with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0025] The present application provides a data storage method, device and electronic device based on multiple databases to solve the technical problem that the existing user data storage system cannot simultaneously support high performance of large data volume query and save costs as much as possible.

[0026] The technical solutions provided by each embodiment of the present application are described in detail below in conjunction with the accompanying drawings.

[0027] Please refer to the attached Figure 1 , which shows a data storage method based on multiple databases provided by an embodiment of the present application, the method comprising:

[0028] S101, obtaining new data of a user and obtaining company information corresponding to the user;

[0029] S102, converting the newly added data and storing it in the MySQL storage system;

[0030] S103, asynchronously forwarding the newly added data to the automatic switching scheduling system to determine whether the company data corresponding to the user has been migrated to the Elasticsearch storage system;

[0031] S104, if yes, storing the newly added data in the Elasticsearch storage system; if no, determining whether the data volume of the company exceeds a preset threshold;

[0032] S105, if it exceeds, after inserting the newly added data into the MongoDB storage system, copying the newly added data from the MongoDB storage system into the Elasticsearch storage system; if it does not exceed, directly inserting the newly added data into the MongoDB storage system.

[0033] With respect to step S105, after the newly added data is copied from the MongoDB storage system into the Elasticsearch storage system, an asynchronous task is started to migrate all the company's data in the MongoDB storage system to the Elasticsearch storage system.

[0034] In one embodiment, after all the data of the company has been migrated, all the data of the company in the MongoDB storage system is deleted.

[0035] In one embodiment, before the company's data is migrated to the Elasticsearch storage system, a data query is performed on the MongoDB storage system, and after the company's data is migrated to the Elasticsearch storage system, a data query is performed on the Elasticsearch storage system.

[0036] Please refer to the attached Figure 2 , which shows a schematic diagram of the structure of a data storage device based on multiple databases provided by an embodiment of the present application, the device comprising:

[0037] The first acquisition unit 201 is used to acquire the new data of the user and the company information corresponding to the user;

[0038] The first storage unit 202 is used to convert the newly added data and store it in the MySQL storage system;

[0039] A first forwarding unit 203, configured to asynchronously forward the newly added data to the automatic switching scheduling system;

[0040] A first determination unit 204 is used to determine whether the company data corresponding to the user has been migrated to the Elasticsearch storage system, and if so, store the newly added data in the Elasticsearch storage system;

[0041] The second judgment unit 205 is used to judge whether the data volume of the company exceeds a preset threshold. If so, after inserting the new data into the MongoDB storage system, the new data is copied from the MongoDB storage system to the Elasticsearch storage system. If not, the new data is directly inserted into the MongoDB storage system.

[0042] Please refer to the attached Figure 3 , which shows a schematic diagram of the structure of an electronic device provided by an embodiment of the present application, which may include: at least one network interface 302, a memory 303 and at least one processor 301. The various components in the electronic device are coupled together through a bus system 304. It can be understood that the bus system 304 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 304 also includes a power bus, a control bus and a status signal bus, but for the sake of clarity, they are not shown in the attached figure. Figure 3 Various buses are labeled as bus system 304 .

[0043] In some implementations, the memory 303 stores the following elements, executable modules or data structures, or a subset thereof, or an extended set thereof: an operating system 3031 and an application program 3032 .

[0044] The operating system 3031 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various mailing services and process hardware-based tasks. The application 3032 includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present application can be included in the application.

[0045] In the above embodiment, the electronic device also includes: at least one instruction, at least one program, code set or instruction set stored in the memory 303, and the at least one instruction, the at least one program, the code set or instruction set can be executed by the processor 301 to implement the steps of any multi-database based data storage method described in the embodiments of the present application.

[0046] In one embodiment, the present application also provides a computer-readable storage medium, which stores at least one instruction, at least one program, a code set or an instruction set. When the at least one instruction, the at least one program, the code set or the instruction set is executed by a processor, the steps of any multi-database based data storage method described in the embodiments of the present application are implemented.

[0047] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be implemented by instructing the relevant hardware through a computer program, and the at least one instruction, at least one program, code set or instruction set can be stored in a non-volatile computer-readable storage medium. When the at least one instruction, at least one program, code set or instruction set is executed, it can implement the steps of any map drawing method described in the embodiment of the present application. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0048] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods, which are merely illustrative and not restrictive. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scopes, which all fall within the scope of protection of the present application.

Claims

1. A data storage method based on multiple databases, characterized in that: The method comprises: Obtain the user's newly added data and obtain the company information corresponding to the user; The newly added data is converted and stored in the MySQL storage system; Asynchronously forwarding the newly added data to the automatic switching scheduling system to determine whether the company data corresponding to the user has been migrated to the Elasticsearch storage system; If yes, store the newly added data in the Elasticsearch storage system; if no, determine whether the data volume of the company exceeds a preset threshold; If it exceeds, after inserting the newly added data into the MongoDB storage system, copy the newly added data from the MongoDB storage system into the Elasticsearch storage system; if it does not exceed, directly insert the newly added data into the MongoDB storage system; After copying the newly added data from the MongoDB storage system into the Elasticsearch storage system, start an asynchronous task to migrate all the company's data in the MongoDB storage system to the Elasticsearch storage system; Before the company's data is migrated to the Elasticsearch storage system, a data query is performed on the MongoDB storage system. After the company's data is migrated to the Elasticsearch storage system, a data query is performed on the Elasticsearch storage system.

2. The data storage method based on multiple databases according to claim 1 is characterized in that: When all the data of the company is migrated, all the data of the company in the MongoDB storage system is deleted.

3. A data storage device based on multiple databases, characterized in that: The device comprises: A first acquisition unit, used to acquire new data of a user and company information corresponding to the user; The first storage unit is used to convert the newly added data and store it in the MySQL storage system; A first forwarding unit, configured to asynchronously forward the newly added data to an automatic switching scheduling system; A first judgment unit is used to judge whether the company data corresponding to the user has been migrated to the Elasticsearch storage system, and if so, store the newly added data in the Elasticsearch storage system; The second judgment unit is used to judge whether the data volume of the company exceeds a preset threshold. If so, after inserting the newly added data into the MongoDB storage system, the newly added data is copied from the MongoDB storage system into the Elasticsearch storage system. If not, the newly added data is directly inserted into the MongoDB storage system. The data storage device is also used to: after copying the newly added data from the MongoDB storage system into the Elasticsearch storage system, start an asynchronous task to migrate all the company's data in the MongoDB storage system to the Elasticsearch storage system; Before the company's data is migrated to the Elasticsearch storage system, a data query is performed on the MongoDB storage system. After the company's data is migrated to the Elasticsearch storage system, a data query is performed on the Elasticsearch storage system.

4. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are executed by the processor to implement the multi-database based data storage method as described in any one of claims 1-2.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, at least one program, a set to be coded or an instruction set, and the at least one instruction, the at least one program, the set to be coded or the instruction set are executed by a processor to implement the multi-database based data storage method as described in any one of claims 1-2.

Images