Data micro-batch falling table storage method and system

Abstract

The invention provides a data micro-batch table storage method and system, and the method comprises the steps: obtaining to-be-processed target data, transmitting the target data to a local cache memory queue after determining that the target data meets a preset condition, and judging whether the local cache memory queue is full or not; after determining that the local cache memory queue is not full, storing the target data into the local cache memory queue; according to the maximum number of single-time pulling tasks of the local cache memory queue, pulling to-be-stored to-be-consumed data from the local cache memory queue at regular time through a scheduling thread, distributing the to-be-consumed data to each working thread according to parameters of a working thread pool, and storing the to-be-consumed data to a database after the working threads process the to-be-consumed data; after determining that the local cache memory queue is full, performing cache compensation processing on the target data; according to the method, the table storage efficiency of the buried point data is effectively improved, the reusability is good, and the table storage requirements of different data can be met.

Description

Technical Field

[0001] The present application belongs to the field of data technology, and relates to a data processing technology, and in particular to a data micro-batch table storage method and system. Background Art

[0002] Usually, in order to provide users with a better APP experience and more accurate services, it is often necessary to collect user behavior data. This data helps to understand the shortcomings of the product, improve user portraits, support operational decisions, and calculate core indicators to evaluate the product's operating status and market performance.

[0003] When users browse and operate the app, such as opening a page, clicking on page elements, sliding, etc., corresponding behavioral event tracking data will be generated. The client app reports the collected tracking data to the server through the interface for data analysis. Conventional tracking data processing is implemented through hard coding, which is not flexible enough. Any logical changes require code development, the process is complicated, the labor cost is high, and the iteration speed is slow. The database write pressure is increasing, and the database has performance bottlenecks.

[0004] As the number of users grows, the tracking data generated by the client becomes larger and larger, and the number of requests for clients to upload tracking data increases, which puts increasing pressure on the server and database.

[0005] The pressure on the server side can generally be shared by deploying multiple server nodes through horizontal expansion. Based on cost and architectural complexity, most companies generally adopt a master-slave database architecture. The data write traffic cannot be expanded horizontally, and the master data is subject to very high concurrency, which will cause a sudden increase in the pressure on the master database, reduced database performance, slow response, and blocked connections, and even downtime in severe cases. The main reason is that the iteration of buried data generates complex and changeable buried data processing requirements; and when the amount of buried data is very large, the database write concurrency is very high, resulting in reduced data write efficiency. Summary of the invention

[0006] The purpose of the present application is to provide a data micro-batch table storage method and system to solve the problem of low efficiency of buried data writing in the prior art.

[0007] In a first aspect, the present application provides a data micro-batch table storage method, the method comprising: Obtaining target data to be processed, and determining whether the target data meets preset conditions; After determining that the target data meets the preset condition, sending the target data to the local cache memory queue, and determining whether the local cache memory queue is full; After determining that the local cache memory queue is not full, storing the target data in the local cache memory queue; According to the maximum number of tasks pulled at a time by the local cache memory queue, the data to be stored and consumed are pulled from the local cache memory queue by scheduling threads, and the data to be consumed are distributed to each worker thread according to the parameters of the worker thread pool, and the data to be consumed are stored in the database after being processed by the worker thread; After determining that the local cache memory queue is full, cache compensation processing is performed on the target data.

[0008] In an implementation of the first aspect, the performing cache compensation processing on the target data includes: Determining whether the target data fails to be sent to the local cache memory queue for the first time; If the target data fails to be sent to the local cache memory queue for the first time, a scheduling message is sent to the scheduling thread, and the scheduling thread immediately pulls data from the local cache memory queue according to the scheduling message according to the maximum number of tasks pulled at a time by the local cache memory queue and sends it to the working thread pool; The target data is resent to the local cache memory queue, and it is re-determined whether the local cache memory queue is full.

[0009] In an implementation of the first aspect, if the target data fails to be sent to the local cache memory queue for the first time, the target data is stored in an MQ distributed queue, and after a preset time, the target data is sent to a failure compensation handler, and the target data is resubmitted and sent to the local cache memory queue through the failure compensation handler.

[0010] In an implementation of the first aspect, the work thread pool includes core work threads and backup work threads. When the core work threads in the work thread pool are all occupied, the backup work threads are enabled to process data. When the core work threads and the backup work threads in the work thread pool are all occupied, the work thread pool stores the remaining data scheduled by the scheduling thread in the task warehouse.

[0011] In an implementation of the first aspect, allocating the to-be-consumed data to each worker thread according to the parameters of the worker thread pool includes: Obtaining the number of unoccupied working threads in the working thread pool, and allocating data to the working threads according to the single working capacity of each unoccupied working thread; If there is remaining data after the data to be consumed is allocated, the remaining data is stored in the task warehouse, and the remaining data is submitted to the working thread that has completed the data processing in sequence according to the time sequence in which each working thread completes the data processing; The amount of the remaining data submitted to the working thread that completes data processing each time does not exceed the single working capacity of the working thread.

[0012] In an implementation manner of the first aspect, the preset condition includes at least one of a screening condition and a default rule, and the screening condition includes at least one of a page name, an event type, and an event name.

[0013] In an implementation of the first aspect, the method further includes: If the target data does not meet the preset condition, the target data is abandoned from being stored, and it is continued to be determined whether the next target data meets the preset condition.

[0014] In an implementation of the first aspect, the target data includes at least any one of buried point data and device information data. When the target data is buried point data, the method further includes: preprocessing the buried point data to standardize the format and content of the buried point data.

[0015] In an implementation of the first aspect, preprocessing the buried point data includes: After the user generates the buried point data, the buried point data is regularly reported to the server through the client; After receiving the buried data, the server writes the buried data into the message middleware, and asynchronously reads the buried data from the message middleware; The read buried point data is parsed to obtain the buried point data in a standard format, and the buried point data in the standard format is used as the target data.

[0016] In a second aspect, the present invention provides a data micro-batch table storage system, the system comprising: A judgment module, used to obtain target data to be processed and judge whether the target data meets preset conditions; A sending module, configured to send the target data to a local cache memory queue after determining that the target data meets a preset condition, and determine whether the local cache memory queue is full; A first processing module, configured to store the target data into the local cache memory queue after determining that the local cache memory queue is not full; The scheduling module is used to pull the data to be stored and consumed from the local cache memory queue according to the maximum number of tasks pulled at a time by the local cache memory queue through the scheduling thread, and distribute the data to be consumed to each worker thread according to the parameters of the worker thread pool, and store the data to be consumed in the database after the worker thread processes it; The second processing module is used to perform cache compensation processing on the target data after determining that the local cache memory queue is full.

[0017] As described above, the data micro-batch table storage method and system described in this application have the following beneficial effects: The present application performs batch saving processing on the target data to be stored by adopting micro-batch processing, by placing the target data into the local cache memory queue in sequence, and by scheduling the data in the local cache memory queue in a regular manner through the scheduling thread and distributing it to the working threads in the working thread pool for processing, and storing the scheduled data in the database in sequence after the working threads have processed it, and when the local cache memory queue is full, performing cache compensation processing on the target data to ensure that the target data can be cached in the distributed queue after the second submission fails to prevent data loss, and resubmitting the target data to the local cache memory queue after the preset time is reached. Through the clever combination of micro and batch, it has the characteristics of low latency and high performance, which not only meets the real-time requirements of data table entry, but also reduces the concurrency and pressure of database operations, and the method of the present application scheme has high versatility and reusability, and can be applied to data table entry processing in different task business scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Shown is a flow chart of the data micro-batch table storage method described in an embodiment of the present application.

[0019] Figure 2 Shown is a schematic diagram of the execution process of the data micro-batch table storage method described in an embodiment of the present application.

[0020] Figure 3 Shown is a schematic diagram of the process of preprocessing point data in the data micro-batch table storage method described in an embodiment of the present application.

[0021] Figure 4 Shown is a structural block diagram of the data micro-batch table storage system described in the embodiment of the present application DETAILED DESCRIPTION

[0022] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0023] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0024] See also Figures 1 to 3 The following embodiments of the present application provide a data micro-batch table storage method and system described in the present application, which has the following beneficial effects: The data micro-batch table storage method of the present application performs batch preservation processing on the target data to be stored by adopting a micro-batch processing method, by placing the target data into the local cache memory queue in turn, and scheduling the data in the local cache memory queue through a scheduling thread and distributing it to the working threads in the working thread pool for processing, and storing the scheduled data in the database in turn after the working threads are processed, and when the local cache memory queue is full, cache compensation processing is performed on the target data to ensure that the target data can be cached in the distributed queue after the second submission fails to prevent data loss, and resubmit the target data to the local cache memory queue after the preset time is reached. Through the clever combination of micro and batch, it has the characteristics of low latency and high performance, which not only meets the real-time requirements of data table drop, but also reduces the concurrency and pressure of database operations, and the method of the present application scheme has high versatility and reusability, and can be applied to data table drop processing in different task business scenarios.

[0025] The technical solutions in the embodiments of the present application will be described in detail below in conjunction with the drawings in the embodiments of the present application.

[0026] like Figure 1 As shown, this embodiment provides a data micro-batch table storage method, the method comprising the following steps: S101, obtaining target data to be processed, and determining whether the target data meets a preset condition.

[0027] In this embodiment, for the target data that needs to be processed, it is first determined whether the target data meets the preset conditions. Considering that not all data needs to be stored in the database, a preliminary judgment is made on the target data by setting preset conditions to ensure that only the target data that meets the preset conditions will be stored in the database, thereby reducing the storage pressure of the database.

[0028] In some embodiments, the preset condition includes at least one of a filtering condition and a default rule, and the filtering condition includes at least one of a page name, an event type, and an event name.

[0029] Specifically, the preset conditions include at least one of filtering conditions and default rules. The filtering conditions are whitelist rules, and the default rules are blacklist rules. Both can be configured according to actual conditions. For example, the filtering conditions include at least one of page name, event type and event name. By judging whether the target data falls into the filtering conditions, the target data is filtered in a whitelist manner to facilitate processing of the target data.

[0030] Exemplarily, for example, the page name includes "borrow cash_advance loan", when there is data in the target data whose page name includes "borrow cash_advance loan", it is determined that the target data meets the preset condition, otherwise it does not meet the condition.

[0031] S102: After determining that the target data meets a preset condition, the target data is sent to a local cache memory queue, and it is determined whether the local cache memory queue is full.

[0032] In this embodiment, reference Figure 2 After determining that the target data meets the preset conditions, it can be determined that the target data needs to be stored in the database. After that, the target data is sent to the local cache memory queue, and it is determined whether the data in the local cache memory queue is full. Since a large amount of data is stored in the local cache memory queue, in order to avoid the situation where the target data to be stored has no storage location and cannot be stored, by determining whether the local cache memory queue is full, the target data is processed differently to ensure that the target data can be added to the local cache memory queue in time when the local cache memory queue is not full, and can be processed in time when the local cache memory queue is full, to prevent data loss.

[0033] In another aspect, the method further comprises: If the target data does not meet the preset condition, the target data is abandoned from being stored, and it is continued to be determined whether the next target data meets the preset condition.

[0034] In this embodiment, when it is determined that the target data does not meet the preset conditions, it is determined that the current target data does not need to be stored, so it can be directly abandoned and the next target data can be processed to determine whether the next target data meets the preset conditions, thereby achieving continuous processing of the target data.

[0035] S103: After determining that the local cache memory queue is not full, storing the target data into the local cache memory queue.

[0036] S104, according to the maximum number of tasks that can be pulled at a time from the local cache memory queue, the data to be stored and consumed is pulled from the local cache memory queue periodically by scheduling a thread, and the data to be consumed is distributed to each working thread according to the parameters of the working thread pool, and the data to be consumed is stored in the database after being processed by the working thread.

[0037] In this embodiment, when it is determined that the local cache memory queue is not full, the target data is directly submitted and stored in the local cache memory queue. Since the scheduling thread pulls the stored data to be consumed from the local cache memory queue at a preset time, and distributes the pulled data to be consumed to each worker thread in the worker thread pool, the worker thread processes the distributed data to be consumed and stores it in the database.

[0038] In some further embodiments, the work thread pool includes core work threads and backup work threads. When the core work threads in the work thread pool are all occupied, the backup work threads are enabled to process data. When the core work threads and the backup work threads in the work thread pool are all occupied, the work thread pool stores the remaining data scheduled by the scheduling thread in the task warehouse.

[0039] In this embodiment, the worker thread pool includes a core worker thread and a standby worker thread. The core worker thread is used as a default thread to process data and save it to a database, while the standby worker thread is used as a standby thread. When the core worker threads are occupied, the standby worker threads are used to process the remaining data and send it to the database. When the core worker thread and the standby worker thread are occupied and there is still remaining data to be consumed currently scheduled by the scheduling thread, the remaining data is stored in the task warehouse in the scheduling thread, and the remaining data continues to be allocated after the core worker thread or the standby worker thread completes the task processing, thereby effectively preventing accumulation in the worker thread pool.

[0040] In some further embodiments, allocating the to-be-consumed data to each worker thread according to the parameters of the worker thread pool includes: Obtaining the number of unoccupied working threads in the working thread pool, and allocating data to the working threads according to the single working capacity of each unoccupied working thread; If there is remaining data after the data to be consumed is allocated, the remaining data is stored in the task warehouse, and the remaining data is submitted to the working thread that has completed the data processing in sequence according to the time sequence in which each working thread completes the data processing; The amount of the remaining data submitted to the working thread that completes data processing each time does not exceed the single working capacity of the working thread.

[0041] In this embodiment, after the scheduling thread calls the data to be consumed from the local cache memory queue, the number of unoccupied working threads in the working thread pool is first obtained, including core working threads and spare working threads, and the core working threads have a higher allocation priority than the spare working threads. Data is allocated to each working thread according to the single working capacity of each unoccupied working thread, so that the working thread processes the data and saves it to the database.

[0042] In the process of allocating data, if there is still remaining data after the data to be consumed is allocated, the remaining data will be stored in the task warehouse in the working thread pool. Subsequently, as the working threads complete data processing, the remaining data will be submitted to the working threads that have completed data processing in sequence, ensuring that when the capacity of the data to be consumed is greater than the current processing capacity of the working thread pool, there will be no data accumulation.

[0043] It should be noted that the capacity of the remaining data stored in the task warehouse does not exceed the upper capacity limit of the task warehouse. Data that exceeds the upper capacity limit of the task warehouse will be placed in the working thread after the data stored in the task warehouse is processed or after the working thread completes data processing.

[0044] Exemplarily, the maximum capacity of the local cache memory queue is 10,000, the time interval for the scheduled scheduling of the scheduling thread is 50ms, the maximum number of data pulled from the local cache memory queue by the scheduling thread each time is 2,000, the maximum number of work threads in the work thread pool that are saved in batches to the database each time is 100, the maximum number of core work threads in the work thread pool is 10, the number of standby work threads is 10, and the maximum capacity of the character warehouse in the work thread pool is 200.

[0045] S105: After determining that the local cache memory queue is full, perform cache compensation processing on the target data.

[0046] When it is determined that the local cache memory queue is full, in order to prevent the target data from being lost, the target data is compensated so that the target data can be submitted and stored again when the data in the local cache memory queue is not full, thereby ensuring the integrity of the target data and preventing data loss.

[0047] In some embodiments, performing cache compensation processing on the target data includes: Determining whether the target data fails to be sent to the local cache memory queue for the first time; If the target data fails to be sent to the local cache memory queue for the first time, a scheduling message is sent to the scheduling thread, and the scheduling thread immediately pulls data from the local cache memory queue according to the scheduling message according to the maximum number of tasks pulled at a time by the local cache memory queue and sends it to the working thread pool; The target data is resent to the local cache memory queue, and it is re-determined whether the local cache memory queue is full.

[0048] In this embodiment, when the target data sends a request to the local cache memory queue, and the local cache memory queue is full, in order to ensure that the target data will not be lost, it is first determined whether the current target data has failed to be submitted to the local cache memory queue for the first time, so that when the local cache memory queue is still full after multiple submissions, the target data is saved in the distributed queue and waits for the next submission.

[0049] Specifically, after the local cache memory queue is full and the target data fails to be submitted, if it is determined that the target data fails to be submitted for the first time, a scheduling message is sent to the scheduling thread. After receiving the scheduling message, the scheduling thread immediately pulls data from the local cache memory queue according to the maximum number of tasks that can be pulled at a time from the local cache memory queue and sends it to the working thread pool, thereby reducing the data in the local cache memory queue. The target data that failed to be submitted for the first time is then resent to the local cache memory queue, and it is re-determined whether the local cache memory queue is full. If after resubmission, it is determined that the local cache memory queue is not full, the resubmitted target data is directly stored in the local cache memory queue in order. If it is determined that the local cache memory queue is still full, it is further determined whether the local cache memory queue is full.

[0050] Furthermore, if the target data fails to be sent to the local cache memory queue for the first time, the target data is stored in the MQ distributed queue, and after a preset time, the target data is sent to the failure compensation handler, and the target data is resubmitted and sent to the local cache memory queue through the failure compensation handler.

[0051] In this embodiment, when it is determined that the target data has not failed to be sent to the local cache memory queue for the first time, for example, after the initial sending fails, the scheduling thread receives the scheduling information and immediately schedules the local cache memory queue once, and after the target data that failed to be submitted is sent to the local cache memory queue again, if the local cache memory queue is still full, it is determined that the current target data has failed to be submitted for the second time. In order to avoid repeated submissions that affect the efficiency of data table storage and prevent the loss of target data, after the target data fails to be submitted for the second time, the target data is cached in a distributed queue, so that after a preset time is reached, the target data cached in the distributed queue is sent to the failure compensation handler, and the target data cached in the distributed queue is resent to the local cache memory queue through the failure compensation handler, to ensure that the target data that failed to be submitted can also be stored in the database through the local cache memory queue.

[0052] It should be noted that the preset time is artificially set data, mainly to ensure that the failure compensation processing program sends the target data when the local cache memory queue is not full, and to ensure that the target data that fails to be submitted can also be saved normally.

[0053] In some other embodiments, the target data includes at least one of the buried point data and the equipment information data.

[0054] In the above method, the target data can be either time-point data or device information data, and the above method can be used to implement batch table storage.

[0055] It should be noted that the target data can also be other types of data, and this solution does not specifically limit this. For different data, it is only necessary to adjust the preset conditions to achieve preliminary screening, and then directly use the above method to store it in the table, which has good reusability and can meet the storage requirements of different types of data.

[0056] When the target data is buried point data, the method further includes: preprocessing the buried point data to standardize the format and content of the buried point data.

[0057] In some embodiments, the preprocessing of the buried point data is performed by referring to Figure 3 ,include: After the user generates the buried point data, the buried point data is regularly reported to the server through the client; After receiving the buried data, the server writes the buried data into the message middleware, and asynchronously reads the buried data from the message middleware; The read buried point data is parsed to obtain the buried point data in a standard format, and the buried point data in the standard format is used as the target data.

[0058] In this embodiment, when the target data is the data of the buried point, since the buried point data is getting larger and larger with the increasing number of users, it is easy to cause great pressure on the database. After the buried point data is collected, the client reports the buried point data to the server at a regular time. After receiving the buried point data, the server first writes the buried point data to the message middleware Kafka in sequence, and then asynchronously reads the buried point data from the message middleware Kafka. In this way, the peak shaving and valley leveling effect can be achieved according to the server resources.

[0059] Since there are differences between the buried point data, in order to further ensure the efficiency of table storage, the buried point data is parsed to obtain the buried point data in a standardized format to facilitate the subsequent storage of the buried point data.

[0060] Specifically, since the tracking data reported by clients of different versions may be different, such as single reporting, batch reporting, inconsistent structure, inconsistent field naming, inconsistent field content code table, dirty data tracking, default tracking values, additional information supplementation, etc., tracking analysis is to convert these differentially structured tracking data into standard data formats and contents to facilitate subsequent analysis and use.

[0061] It should be noted that the parsing of buried point data in different formats to convert them into a standard format adopts the content of the existing technology. This solution does not involve improvements to the specific parsing process. Any technical means that can convert buried point data into a standard format can be applied to the parsing process of the present application solution, and this solution does not make any special limitations on this.

[0062] The data micro-batch table storage method of the present application performs batch preservation processing on the target data to be stored by adopting a micro-batch processing method, by placing the target data into the local cache memory queue in turn, and scheduling the data in the local cache memory queue through a scheduling thread and distributing it to the working threads in the working thread pool for processing, and storing the scheduled data in the database in turn after the working threads are processed, and when the local cache memory queue is full, cache compensation processing is performed on the target data to ensure that the target data can be cached in the distributed queue after the second submission fails to prevent data loss, and resubmit the target data to the local cache memory queue after the preset time is reached. Through the clever combination of micro and batch, it has the characteristics of low latency and high performance, which not only meets the real-time requirements of data table drop, but also reduces the concurrency and pressure of database operations, and the method of the present application scheme has high versatility and reusability, and can be applied to data table drop processing in different task business scenarios.

[0063] The protection scope of the data micro-batch table storage method described in the embodiment of the present application is not limited to the execution order of the steps listed in the present embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present application are included in the protection scope of the present application.

[0064] This application also provides a data micro-batch table storage system, refer to Figure 4 , the system comprising: The judgment module 401 is used to obtain the target data to be processed and judge whether the target data meets the preset conditions; The sending module 402 is used to send the target data to the local cache memory queue after determining that the target data meets the preset condition, and determine whether the local cache memory queue is full; A first processing module 403 is used to store the target data into the local cache memory queue after determining that the local cache memory queue is not full; The scheduling module 404 is used to pull the to-be-consumed data to be stored from the local cache memory queue regularly through the scheduling thread according to the maximum number of tasks pulled at a time by the local cache memory queue, and distribute the to-be-consumed data to each worker thread according to the parameters of the worker thread pool, and store the to-be-consumed data in the database after being processed by the worker thread; The second processing module 405 is used to perform cache compensation processing on the target data after determining that the local cache memory queue is full.

[0065] The embodiment of the present application also provides a data micro-batch storage system, which can implement the data micro-batch storage method described in the present application. However, the implementation device of the data micro-batch storage method described in the present application includes but is not limited to the structure of the data micro-batch storage system listed in the present embodiment. All structural deformations and replacements of the prior art made according to the principles of the present application are included in the protection scope of the present application.

[0066] It should be noted that the structure and principle of each of the above modules correspond one-to-one to the steps in the above data micro-batch table storage method. The specific working principle can also be referred to the introduction of the data micro-batch table storage method in the above embodiment, so it will not be repeated here.

[0067] An embodiment of the present invention further provides an electronic device, comprising: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program stored in the memory, so that the electronic device executes the above-mentioned data micro-batch table storage method.

[0068] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by an electronic device, the above-mentioned data micro-batch table storage method is implemented.

[0069] A person of ordinary skill in the art can understand that all or part of the steps in the method for implementing the above embodiment can be completed by instructing a processor through a program, and the program can be stored in a computer-readable storage medium, and the storage medium is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid-state hard disk, a magnetic tape, a floppy disk, an optical disc, and any combination thereof. The above storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid-state disk (SSD)), etc.

[0070] In the several embodiments provided in the present application, it should be understood that the disclosed system, device or method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules / units is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules or units, which can be electrical, mechanical or other forms.

[0071] The modules / units described as separate components may or may not be physically separated, and the components displayed as modules / units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules / units may be selected according to actual needs to achieve the purpose of the embodiments of the present application. For example, the functional modules / units in the various embodiments of the present application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0072] Those of ordinary skill in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0073] The descriptions of the processes or structures corresponding to the above-mentioned figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.

[0074] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A data micro-batch table storage method, characterized in that: The method comprises: Obtaining target data to be processed, and determining whether the target data meets preset conditions; After determining that the target data meets the preset condition, sending the target data to the local cache memory queue, and determining whether the local cache memory queue is full; After determining that the local cache memory queue is not full, storing the target data in the local cache memory queue; According to the maximum number of tasks pulled at a time by the local cache memory queue, the data to be stored and consumed are pulled from the local cache memory queue by scheduling threads, and the data to be consumed are distributed to each worker thread according to the parameters of the worker thread pool, and the data to be consumed are stored in the database after being processed by the worker thread; After determining that the local cache memory queue is full, cache compensation processing is performed on the target data.

2. The data micro-batch table storage method according to claim 1 is characterized in that: The performing cache compensation processing on the target data includes: Determining whether the target data fails to be sent to the local cache memory queue for the first time; If the target data fails to be sent to the local cache memory queue for the first time, a scheduling message is sent to the scheduling thread, and the scheduling thread immediately pulls data from the local cache memory queue according to the scheduling message according to the maximum number of tasks pulled at a time by the local cache memory queue and sends it to the worker thread pool; The target data is resent to the local cache memory queue, and it is re-determined whether the local cache memory queue is full.

3. The data micro-batch table storage method according to claim 2 is characterized in that: If the target data fails to be sent to the local cache memory queue for the first time, the target data will be stored in the MQ distributed queue, and after a preset time, the target data will be sent to the failure compensation handler, and the target data will be resubmitted and sent to the local cache memory queue through the failure compensation handler.

4. The data micro-batch table storage method according to claim 1 is characterized in that: The work thread pool includes core work threads and backup work threads. When the core work threads in the work thread pool are all occupied, the backup work threads are enabled to process data. When the core work threads and the backup work threads in the work thread pool are all occupied, the work thread pool stores the remaining data scheduled by the scheduling thread in the task warehouse.

5. The data micro-batch table storage method according to claim 4 is characterized in that: The allocating the to-be-consumed data to each worker thread according to the parameters of the worker thread pool includes: Obtaining the number of unoccupied working threads in the working thread pool, and allocating data to the working threads according to the single working capacity of each unoccupied working thread; If there is remaining data after the data to be consumed is allocated, the remaining data is stored in the task warehouse, and the remaining data is submitted to the working thread that has completed the data processing in sequence according to the time sequence in which each working thread completes the data processing; The amount of the remaining data submitted to the working thread that completes data processing each time does not exceed the single working capacity of the working thread.

6. The data micro-batch table storage method according to claim 1 is characterized in that: The preset condition includes at least one of a screening condition and a default rule, and the screening condition includes at least one of a page name, an event type, and an event name.

7. The data micro-batch table storage method according to claim 6 is characterized in that: The method further comprises: If the target data does not meet the preset condition, the target data is abandoned from being stored, and it is continued to be determined whether the next target data meets the preset condition.

8. The data micro-batch table storage method according to any one of claims 1 to 7, characterized in that: The target data includes at least any one of buried point data and device information data. When the target data is buried point data, the method further includes: preprocessing the buried point data to standardize the format and content of the buried point data.

9. The data micro-batch table storage method according to claim 8 is characterized in that: The preprocessing of the buried point data includes: After the user generates the buried point data, the buried point data is regularly reported to the server through the client; After receiving the buried data, the server writes the buried data into the message middleware, and asynchronously reads the buried data from the message middleware; The read buried point data is parsed to obtain the buried point data in a standard format, and the buried point data in the standard format is used as the target data.

10. A data micro-batch table storage system, characterized in that: The system comprises: A judgment module, used to obtain target data to be processed and judge whether the target data meets preset conditions; A sending module, configured to send the target data to a local cache memory queue after determining that the target data meets a preset condition, and determine whether the local cache memory queue is full; A first processing module, configured to store the target data into the local cache memory queue after determining that the local cache memory queue is not full; The scheduling module is used to pull the data to be stored and consumed from the local cache memory queue according to the maximum number of tasks pulled at a time by the local cache memory queue through the scheduling thread, and distribute the data to be consumed to each worker thread according to the parameters of the worker thread pool, and store the data to be consumed in the database after the worker thread processes it; The second processing module is used to perform cache compensation processing on the target data after determining that the local cache memory queue is full.

Images