A data storage method and device, electronic equipment and storage medium

Abstract

The application discloses a data storage method and device, electronic equipment and a storage medium, wherein the method comprises: storing the collected business data to a first memory queue in a business module, and storing the business data in the first memory queue to a target database according to the corresponding database type. The embodiment of the application stores the business data collected by the business module to the first memory queue first, and then stores the business data to the corresponding target database according to the database type corresponding to the business data in the first memory queue, so that the business data can be stored without a special audit service interface, the data storage can be applicable to the data storage requirements of different database environments, and the data storage process is more simple.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer technology, and in particular to a data storage method and device, electronic equipment and storage medium. BACKGROUND

[0002] Identity and Access Management (IAM) is a user security comprehensive guarantee platform integrating user management, identity authentication, authorization management and audit. In the prior art, for the business point data of the IAM system, a special audit service interface is very dependent on the implementation of the business data storage, resulting in a relatively complicated data storage process. SUMMARY

[0003] The present application provides a data storage method and device, electronic equipment and storage medium, so as to store the business data collected by the business module to the first memory queue first, and then store the business data in the first memory queue to the corresponding target database according to the database type corresponding to the business data, without the need for a special audit service interface to realize the storage of business data, which can be applied to different database environment data storage requirements, and the data storage process is more simple.

[0004] According to an aspect of the present application, a data storage method is provided, which comprises:

[0005] storing the business data collected by the business module to the first memory queue;

[0006] storing the business data in the first memory queue to the target database according to the database type corresponding to the business data.

[0007] According to another aspect of the present application, a data storage device is provided, which comprises:

[0008] a queue storage module for storing the business data collected by the business module to the first memory queue;

[0009] a data storage module for storing the business data in the first memory queue to the target database according to the database type corresponding to the business data.

[0010] According to another aspect of the present application, an electronic device is provided, which comprises:

[0011] at least one processor; and

[0012] a memory in communication connection with the at least one processor; wherein

[0013] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the data warehouse falling method according to any one of the embodiments of the application.

[0014] According to another aspect of the application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to implement the data warehouse falling method according to any one of the embodiments of the application when executed by the processor.

[0015] The technical solution of the embodiment of the application stores the collected business data to the first memory queue in the business module, and falls the business data in the first memory queue to the target database according to the corresponding database type. The embodiment of the application stores the business data collected by the business module to the first memory queue first, and then falls the business data to the corresponding target database according to the database type corresponding to the business data in the first memory queue. The falling of the business data to the database can be implemented without a special audit service interface, and the data falling process is more convenient.

[0016] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the application, nor is it used to limit the scope of the application. Other features of the application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0018] Figure 1 is a flowchart of a data warehouse falling method according to the first embodiment of the application;

[0019] Figure 2 is a flowchart of a data warehouse falling method according to the second embodiment of the application;

[0020] Figure 3 is a flowchart of a data warehouse falling failure compensation process according to the second embodiment of the application;

[0021] Figure 4 is a flowchart of a data warehouse falling method according to the third embodiment of the application;

[0022] Figure 5is a flow chart of a service data queue storage process according to the third embodiment of the present application;

[0023] Figure 6 is a flow chart of a service data storage strategy execution process according to the third embodiment of the present application;

[0024] Figure 7 is a flow chart of a failure data local persistence process according to the third embodiment of the present application;

[0025] Figure 8 is a flow chart of a failure file judgment and analysis process according to the third embodiment of the present application;

[0026] Figure 9 is a structural schematic diagram of a data storage device according to the fourth embodiment of the present application;

[0027] Figure 10 is a structural schematic diagram of an electronic device implementing the data storage method according to the embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the personnel in the art better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by the personnel in the art without creative labor should belong to the scope of protection of the present application.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily mean a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0030] Embodiment One

[0031] Figure 1A flowchart of a data storage method is provided for the first embodiment of the present application. The first embodiment can be applied to the case of business data storage. The method can be executed by a data storage device, which can be implemented in the form of hardware and / or software. The data storage device can be configured in an electronic device, such as a computer device or a server. Figure 1 As shown in FIG. 1, the data storage method provided by the first embodiment specifically includes the following steps:

[0032] S110, storing the collected business data to a first memory queue in a business module.

[0033] The business module can be a functional module for executing business logic. The number of business modules can be one or more. The business module can be located in a business system and generate business data for storage. The first memory queue can be a memory queue for temporarily storing business data generated by the business module.

[0034] In the embodiment of the present application, after one or more business modules of the business system generate business data for storage, the business data can be sequentially cached into the first memory queue, so as to realize the storage of the business data according to different storage strategies. In a specific embodiment, when the business module calls a pre-configured unified software development kit (Software Development Kit, SDK), the business data generated by the business module can be sequentially written into the first memory queue by calling the created thread.

[0035] S120, storing the business data in the first memory queue to a target database according to the corresponding database type.

[0036] The database type can be the database type corresponding to the database to which the business data is to be stored. The database type can include a relational database, a search engine type database, and a publish / subscribe type database, etc. The relational database can include a MySQL database, an Oracle database, a DB2 database, a SQL Server database, etc. The search engine type database can include an ElasticSearch database and a Solr database, etc. The publish / subscribe type database can include a Kafka database and a Redis database, etc. The target database can be the database to which the business data is to be stored.

[0037] In the embodiment of the application, before performing data landing, the created thread can be called to detect whether there is service data in the first memory queue, and if there is, the service data in the first memory queue is landed to the corresponding target database according to the corresponding database type. In a specific embodiment, a corresponding database can be configured for each service module in advance, and the database addresses configured by each service module are stored in a preset configuration file, so that when service data is detected in the first memory queue, the service data generated by each service module can be landed to the corresponding target database according to the database address associated with the service module saved in the preset configuration file, wherein the database type corresponding to the service data can include a relational database, a search engine type database and a publish / subscribe type database.

[0038] Further, on the basis of the above-mentioned embodiment of the application, after the execution of the service data landing fails, the failed service data can be compensated to ensure the consistency of the data in the landing process. Specifically, after the service data landing fails, the current failed service data is persisted to a file in a local directory, the persisted failed file is parsed through a thread reading mode, and the parsed data is written into the first memory queue again to execute the data landing process again.

[0039] The technical scheme of the embodiment of the application stores the collected service data of the service module to the first memory queue, and lands the service data in the first memory queue to the target database according to the corresponding database type. The embodiment of the application stores the service data collected by the service module to the first memory queue first, and then lands the service data to the corresponding target database according to the database type corresponding to the service data in the first memory queue. Without a special audit service interface, the landing of the service data can be realized, which can be applied to the data landing demand of different database environments, and the data landing process is more simple.

[0040] Embodiment two

[0041] Figure 2 A flowchart of a data landing method provided by the second embodiment of the application is based on the further optimization and extension of the above-mentioned embodiments, and can be combined with each optional technical scheme in the above-mentioned embodiments. As shown in Figure 2 The data landing method provided by the second embodiment specifically includes the following steps:

[0042] S210, when the service module calls the preset software development kit, the service data generated by the service module is sequentially cached to the first memory queue.

[0043] The preset software development kit (SDK) can be understood as a collection of development tools for establishing application software by a user for a specific software package, software framework, hardware platform, operating system, etc., and is used for assisting in developing related documents, examples and tools of a certain type of software product. The preset SDK can include one or more application programming interfaces (APIs), and the business module can implement the execution logic of the business data falling into the database by calling the preset SDK.

[0044] In the embodiment of the application, when one or more business modules of the business system call the preset SDK, the data line of the business module can be added to the first memory queue, so that the business data generated by the business module can be sequentially cached into the first memory queue. In a specific embodiment, when the business module calls the preset SDK, the thread created by calling can be used to sequentially write the business data generated by the business module into the first memory queue.

[0045] Further, on the basis of the above-mentioned embodiment of the application, before S220 to S240 are executed, the data falling into the database method provided in the second embodiment of the application can further include:

[0046] The thread created by calling is used to detect whether the business data exists in the first memory queue, and if the business data exists, the database type of the database to be fallen into of the business data is searched in the pre-configured configuration file.

[0047] S220, when the database type is a relational database, the Hibernate framework is called to perform data saving and transaction submission on the business data, so as to fall the business data into the target database corresponding to the database type.

[0048] The Hibernate framework can be an object-relational mapping framework, which can well shield the differences between different databases (such as MySQL, Oracle, etc.), automatically generate SQL statements, automatically execute, so that Java programmers can use object programming thinking to operate the database at will, and do not need to write SQL statements. When using Hibernate, only the entity class needs to be created, and Hibernate will automatically create or update the corresponding database table according to the created entity class. The relational database can be a database that uses a relational model to organize data, which stores data in the form of rows and columns. The relational database can include a MySQL database, an Oracle database, a DB2 database, a SQL Server database, etc.

[0049] In the embodiment of the present application, when the service data is detected in the first memory queue, the service data can be stored in the corresponding database according to the database type configured for the service data in advance. Specifically, if the database type corresponding to the service data is a relational database, the Hibernate framework can be directly called to save and commit the transaction of the service data, so as to store the service data in the target database corresponding to the database type. The relational database can include, but is not limited to, MySQL database, Oracle database, DB2 database, SQL Server database, etc.

[0050] In S230, when the database type is an ElasticSearch database, the Hibernate annotation of the entity class is reflected, the table name and field name corresponding to the database table obtained by reflection are parsed, and the parsed data is stored in the ElasticSearch database by using the Spring-Data framework.

[0051] The annotation can be a code-level description at the same level as a class, interface, and enumeration. The annotation can be declared in front of a package, class, field, method, local variable, method parameter, etc. to describe and annotate these elements. The annotation exists in the form of '@annotation name' in the code. The annotation does not directly affect the semantics of the program, but can be used to create documents, track dependencies in the code, and even perform basic compilation checks. The Spring-Data framework can be an open source framework for simplifying database, non-relational database, and index database access and supporting cloud services. The main goal of the Spring-Data framework is to make data access convenient and fast.

[0052] In the embodiment of the present application, if the database type corresponding to the service data is an ElasticSearch database, the Hibernate annotation of the entity class (i.e., the service data to be stored) can be reflected, and the table name and field name corresponding to the database table obtained by reflection can be parsed. Finally, the Spring-Data framework is called to save the parsed data to store the service data in the ElasticSearch database, so as to realize the storage of the service data. It should be understood that the ElasticSearch database is only an example, and other search engine type databases such as Solr database can also be used in actual application, and the embodiment of the present application does not limit this.

[0053] S240, when the database type is the Kafka database, reflecting the Hibernate annotation of the entity class, parsing the table name corresponding to the database table obtained by the reflection, and calling the application programming interface of the Kafka database to push the parsed data into the message queue with the table name as the name.

[0054] In the embodiment of the application, if the database type corresponding to the business data is a Kafka database, the Hibernate annotation of the entity class (i.e., the business data to be stored in the database) can be reflected, the table name corresponding to the database table obtained by the reflection can be parsed, and finally the API of the Kafka database can be called to push the parsed data into the message queue with the table name as the name, so as to realize the storage of the business data in the database. It should be understood that the above-mentioned Kafka database is only an example, and other publish / subscribe databases such as Redis database can also be used in actual application, and the embodiment of the application does not limit this.

[0055] Further, on the basis of the above-mentioned embodiment of the application, the data storage method provided in the second embodiment of the application can further include: a compensation processing process after the business data storage fails, such as Figure 3 as shown, specifically including the following steps:

[0056] S310, after the business data storage fails, saving the current failed data to a second memory queue.

[0057] In the embodiment of the application, the second memory queue can be a memory queue for temporarily storing the business data whose storage is failed. Specifically, if the business data throws an exception information when saving to the target database during the execution of the storage, the thread created can be called to write the business data whose storage is failed (i.e., the failed data) into the second memory queue in sequence.

[0058] S320, when the failed data exists in the second memory queue, saving the failed data as a temporary file in tmp format.

[0059] In the embodiment of the application, the thread created can be called to detect whether the failed data exists in the second memory queue every interval of a preset period, and if the failed data exists, the failed data in the second memory queue can be saved into a temporary file with tmp as the suffix.

[0060] S330, when the temporary file meets a preset file conversion condition, converting the temporary file into an flog file, wherein the preset file conversion condition at least includes: the file size of the temporary file is greater than a preset threshold, and the difference between the creation time of the temporary file and the current time is greater than a preset time threshold.

[0061] In the embodiment of the present application, the preset file transcription condition can refer to a condition configured in advance for determining whether the temporary file needs to be transcribed into a formal file. The preset file transcription condition can at least include: the file size of the temporary file is greater than a preset threshold, the difference between the creation time of the temporary file and the current time is greater than a preset time threshold, etc., wherein the current time can refer to the execution time of the thread currently processing the failed data. Specifically, when the temporary file meets the preset file transcription condition, the temporary file can be transcribed into a failed formal file, i.e., an flog file.

[0062] S340, when it is determined that the flog file exists under the error directory, the flog file is parsed according to a preset parsing mode, and the parsed data is stored in the first memory queue.

[0063] In the embodiment of the present application, the file under the failure directory can be event-listened by the configured timing task-single thread. When it is detected that the flog file is newly added under the failure directory, the preset parsing mode configured in advance is called to parse the flog file, and the parsed data is stored in the first memory queue. Then, the data landing process of S220 to S240 is executed again. Further, the parsing process of the flog file can specifically include the following steps: the data in the flog file is read as a json string; the json string is deserialized into an entity, the entity includes the full path of the class and the corresponding data; the object of the class is generated through reflection according to the full path of the class, and then the data is assigned to the object through reflection.

[0064] The technical scheme of the embodiment of the application is characterized in that when a preset software development kit is called by a business module, the business data generated by the business module is sequentially cached into a first memory queue; when the database type is a relational database, the Hibernate framework is called to perform data saving and transaction submission on the business data, so as to save the business data into a target database corresponding to the database type; when the database type is an ElasticSearch database, the Hibernate annotation of an entity class is reflected, the table name and the field name corresponding to the database table obtained through the reflection are parsed, and the parsed data is saved into the ElasticSearch database by using the Spring-Data framework; when the database type is a Kafka database, the Hibernate annotation of an entity class is reflected, the table name corresponding to the database table obtained through the reflection is parsed, and the parsed data is pushed into a message queue with the table name as the name by calling the application programming interface of the Kafka database. According to the embodiment of the application, the business data collected by the business module is first stored into the first memory queue, then the data saving strategy corresponding to different database types, i.e., the relational database, the ElasticSearch database and the Kafka database, is executed according to the database type corresponding to the business data in the first memory queue, and then the business data is saved into the corresponding target database, so that the saving of the business data can be realized without a special audit service interface, the data saving process is more convenient, and the data saving requirement of different database environments can be met.

[0065] Embodiment three

[0066] Figure 4 A flowchart of a data saving method provided by the third embodiment of the application is provided, and the embodiment provides an implementation manner of the data saving method, which can realize the saving of the business point data of different types of databases and the failure compensation after the saving execution fails. As shown in Figure 4 The data saving method provided by the third embodiment of the application specifically includes the following steps:

[0067] S410、When a preset software development kit is called by a business module, the business data generated by the business module is sequentially cached into a first memory queue.

[0068] In the embodiment of the application, as shown in Figure 5 When the saving method of the preset SDK is called by the business module data, the data line of the business module can be added to the first memory queue, so that the business data generated by the business module can be sequentially cached into the first memory queue.

[0069] It should be understood that, Figure 5The flow shown is an asynchronous step, that is, after the business system writes the business data generated by the business module into the first memory queue, the flow ends, and then the consumption and saving of the business data, that is, the data is written into the database, is performed.

[0070] S420, the thread consumes the business data in the first memory queue and calls the corresponding strategy class to write the business data into the database.

[0071] In the embodiment of the application, as shown in Figure 6 The created thread can detect whether the business data exists in the first memory queue, and if the business data exists, the database type of the to-be-written database corresponding to the business data is found in the pre-configured configuration file. The persistence strategy type configured in the configuration file is mapped to the corresponding strategy class, and the corresponding strategy class is executed on the business data to write the business data into the corresponding database. Specifically, if the database type is a relational database, the Hibernate framework is called to save the data and submit the transaction of the business data to write the business data into the target database corresponding to the database type; if the database type is an ElasticSearch database, the Hibernate annotation of the entity class is reflected, the table name and field name corresponding to the database table obtained by reflection are parsed, and the parsed data is written into the ElasticSearch database by using the Spring-Data framework; if the database type is a Kafka database, the Hibernate annotation of the entity class is reflected, the table name corresponding to the database table obtained by reflection is parsed, and the parsed data is pushed into the message queue with the table name as the name by calling the application programming interface of the Kafka database.

[0072] S430, after the business data fails to be written into the database, the current failed data is saved into the second memory queue; when the failed data exists in the second memory queue, the failed data is saved as a temporary file in tmp format; and when the temporary file meets the preset file conversion condition, the temporary file is converted into a flog file.

[0073] In the embodiment of the application, as shown in Figure 7As shown, when an exception occurs in saving the business data, i.e., the data fails to be stored in the database, the failed data can be written into the second memory queue in turn; then, another thread dedicated to processing the failed data scans the second memory queue, and if there is failed data, the batch of failed data is saved into a temporary file with tmp as the suffix; further, it is judged whether the temporary file meets preset file transcription conditions, which can at least include that the file size of the temporary file is greater than a preset threshold, the difference between the creation time of the temporary file and the execution time of the current thread is greater than a preset time threshold, etc., and if yes, the temporary file is transcribed into a failed formal file, i.e., an flog file.

[0074] S440, the thread calls event listening on the files under the failure directory, parses the flog file when a new flog file is added under the error directory, and stores the parsed data into the first memory queue.

[0075] In the embodiment of the application, as shown, Figure 8 As shown, a timing task-single thread can be configured to perform event listening on the files under the failure directory, and when a new flog file is added under the failure directory, a preset parsing mode configured in advance is called to parse the flog file, and the parsed data is stored into the first memory queue, and then the data storage process of S420 is performed again. The parsing process of the flog file can specifically include the following steps: the data in the flog file is read as a json string; the json string is deserialized into an entity, the entity includes the full path of a class and corresponding data; the object of the class is generated through reflection according to the full path of the class, and then the data is assigned to the object through reflection.

[0076] The technical scheme of the embodiment of the application is characterized in that when the service module calls the preset software development kit, the service data generated by the service module is sequentially cached into the first memory queue, the calling thread consumes the service data in the first memory queue, the corresponding strategy class is called to perform database storage processing on the service data, and after the service data fails to be stored in the database, the current failed data is saved to the second memory queue; when the failed data exists in the second memory queue, the failed data is saved as a temporary file in tmp format; when the temporary file meets preset file transcription conditions, the temporary file is transcribed into an flog file, the calling thread performs event listening on the files under the failed directory, when the flog file is added under the error directory, the flog file is parsed, and the parsed data is stored into the first memory queue. The embodiment of the application provides a simple and light-dependent data storage method, realizes that each service integration party can realize the data storage requirement for different database environments through a unified calling entrance of an SDK, and simultaneously guarantees the consistency of data and the compensation strategy of failure in the storage process, so that the data storage of the business data can be realized without a special audit service interface, the data storage method can be applicable to the data storage requirement of different database environments, and the data storage process is more simple and convenient.

[0077] Embodiment four

[0078] Figure 9 A structural schematic diagram of a data storage device provided by the fourth embodiment of the application is shown in FIG. 4. Figure 9 As shown in the figure, the device comprises:

[0079] The queue storage module 51 is configured to store the collected service data of the service module into the first memory queue.

[0080] The data storage module 52 is configured to store the service data in the first memory queue into the target database according to the corresponding database type.

[0081] The technical scheme of the embodiment of the application is characterized in that the queue storage module stores the collected service data of the service module into the first memory queue, and the data storage module stores the service data in the first memory queue into the target database according to the corresponding database type. The embodiment of the application stores the service data collected by the service module into the first memory queue first, and then stores the service data into the corresponding target database according to the database type corresponding to the service data in the first memory queue, so that the data storage of the business data can be realized without a special audit service interface, the data storage method can be applicable to the data storage requirement of different database environments, and the data storage process is more simple and convenient.

[0082] Further, on the basis of the above-mentioned application embodiment, the queue storage module 51 comprises:

[0083] The service data queue storage unit is configured to sequentially cache the service data generated by the service module into the first memory queue when the service module invokes the preset software development kit.

[0084] Further, on the basis of the above-mentioned embodiments, the data falling database module 52 comprises:

[0085] The first data falling database unit is configured to invoke the Hibernate framework to perform data saving and transaction submission on the service data when the database type is a relational database, so as to fall the service data into the target database corresponding to the database type.

[0086] The second data falling database unit is configured to reflect the Hibernate annotation of the entity class, analyze the table name and field name corresponding to the database table obtained by reflection, and use the Spring-Data framework to fall the analyzed data into the ElasticSearch database when the database type is the ElasticSearch database.

[0087] The third data falling database unit is configured to reflect the Hibernate annotation of the entity class, analyze the table name corresponding to the database table obtained by reflection, and use the application programming interface of the Kafka database to push the analyzed data into the message queue with the table name as the name when the database type is the Kafka database.

[0088] Further, on the basis of the above-mentioned embodiments, the data falling database device further comprises:

[0089] The failed data queue storage module is configured to save the current failed data into the second memory queue after the service data falling fails.

[0090] The temporary file creation module is configured to save the failed data as a temporary file in tmp format when the failed data exists in the second memory queue.

[0091] The flog file creation module is configured to convert the temporary file into an flog file when the temporary file meets preset file conversion conditions, wherein the preset file conversion conditions at least include that the file size of the temporary file is greater than a preset threshold, and the difference between the creation time of the temporary file and the current time is greater than a preset time threshold.

[0092] The file analysis module is configured to analyze the flog file according to a preset analysis method and store the analyzed data into the first memory queue when the flog file exists in the error directory.

[0093] Further, based on the above-mentioned embodiments, when the file analysis module performs the step of analyzing the flog file according to the preset analysis mode, the file analysis module is specifically configured to:

[0094] read the data in the flog file as a json string; deserialize the json string into an entity, the entity containing a full path of a class and corresponding data; generate an object of the class through the full path of the class, and then assign the data to the object through reflection.

[0095] The data falling into the database device provided by the embodiments has the functions and advantages of the data falling into the database method provided by any of the embodiments.

[0096] Embodiment five

[0097] Figure 10 A structural schematic diagram of an electronic device 60 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application described and / or claimed in this document.

[0098] As shown in Figure 10 The electronic device 60 includes at least one processor 61 and a memory, such as a read-only memory (ROM) 62, a random access memory (RAM) 63, etc., connected to the at least one processor 61 in communication, where the memory stores computer programs executable by the at least one processor. The processor 61 can perform various appropriate actions and processes according to the computer programs stored in the read-only memory (ROM) 62 or loaded from the storage unit 68 into the random access memory (RAM) 63. In the RAM 63, various programs and data required for the operation of the electronic device 60 can also be stored. The processor 61, the ROM 62, and the RAM 63 are connected to each other through a bus 64. An input / output (I / O) interface 65 is also connected to the bus 64.

[0099] A number of components in the electronic device 60 are connected to the I / O interface 65, including: an input unit 66, such as a keyboard, a mouse, etc.; an output unit 67, such as various types of displays, speakers, etc.; a storage unit 68, such as a magnetic disk, an optical disk, etc.; and a communication unit 69, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 69 allows the electronic device 60 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0100] The processor 61 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 61 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 61 performs various methods and processes described above, such as the data landing method.

[0101] In some embodiments, the data landing method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 68. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 60 via the ROM 62 and / or the communication unit 69. When the computer program is loaded onto the RAM 63 and executed by the processor 61, one or more steps of the data landing method described above can be performed. Alternatively, in other embodiments, the processor 61 can be configured to perform the data landing method by any other suitable means, such as by means of firmware.

[0102] The various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0103] Computer programs for implementing the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program, when executed, can cause instructions defined in the flow charts and / or block diagrams to be implemented. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package and partially on a remote machine or entirely on a remote machine or server.

[0104] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0105] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0106] The systems and techniques described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0107] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0108] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be executed in parallel, executed in series, or executed in different orders, as long as the desired results of the technical solutions of the present disclosure can be achieved, and the present disclosure is not limited herein.

[0109] The specific implementation described above does not constitute a limitation on the scope of protection of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for storing data in a database, characterized in that, The method includes: The collected business data is stored in the first memory queue in the business module. The business data in the first memory queue is written to the target database according to the corresponding database type; This also includes: After the business data fails to be written to the database, the current failed data is saved to the second memory queue; When the failed data exists in the second memory queue, the failed data is saved as a temporary file in tmp format; When the temporary file meets the preset file rewriting conditions, the temporary file is rewritten into an flog file. The preset file rewriting conditions include at least the following: the file size of the temporary file is greater than a preset threshold, and the difference between the creation time of the temporary file and the current time is greater than a preset time threshold. When it is determined that the flog file exists in the error directory, the flog file is parsed according to a preset parsing method, and the parsed data is stored in the first memory queue.

2. The method according to claim 1, characterized in that, The step of storing the collected business data in the first memory queue in the business module includes: When the business module calls the preset software development kit, the business data generated by the business module is cached sequentially into the first memory queue.

3. The method according to claim 1, characterized in that, The step of writing the business data in the first memory queue to the target database according to the corresponding database type includes: When the database type is a relational database, the Hibernate framework is invoked to save the business data and commit transactions, so as to store the business data in the target database corresponding to the database type.

4. The method according to claim 1, characterized in that, The step of writing the business data in the first memory queue to the target database according to the corresponding database type includes: When the database type is ElasticSearch, reflection is performed on the Hibernate annotation of the entity class to parse the table name and field name corresponding to the database table obtained by reflection, and the parsed data is stored in the ElasticSearch database using the Spring-Data framework.

5. The method according to claim 1, characterized in that, The step of writing the business data in the first memory queue to the target database according to the corresponding database type includes: When the database type is Kafka, reflection is performed on the Hibernate annotation of the entity class to parse the table name corresponding to the database table obtained by reflection, and then the application programming interface of the Kafka database is called to push the parsed data to the message queue named after the table name.

6. The method according to claim 1, characterized in that, The parsing of the flog file according to a preset parsing method includes: Read the data from the flog file as a JSON string; The JSON string is deserialized into an entity, which contains the full path of the class and the corresponding data; An object of the class is generated by reflection using the full path of the class, and then the data is assigned to the object by reflection.

7. A data storage device, characterized in that, The device includes: The queue storage module is used to store the collected business data in the first memory queue in the business module. The data storage module is used to store the business data in the first memory queue into the target database according to the corresponding database type. The data storage device further includes: The failure data queue storage module is used to save the current failure data to the second memory queue after the business data fails to be written to the database. A temporary file creation module is used to save the failure data as a temporary file in tmp format when the failure data exists in the second memory queue; The flog file creation module is used to convert the temporary file into an flog file when the temporary file meets the preset file conversion conditions. The preset file conversion conditions include at least: the file size of the temporary file is greater than a preset threshold, and the difference between the creation time of the temporary file and the current time is greater than a preset time threshold. The file parsing module is used to parse the flog file according to a preset parsing method when it is determined that the flog file exists in the error directory, and to store the parsed data in the first memory queue.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data storage method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the data storage method according to any one of claims 1-6.

Images