Big data analysis processing method, apparatus, device, and medium

Abstract

The application relates to the field of big data, and discloses a big data analysis processing method, device, equipment and medium, which comprises the following steps: acquiring time slices of label data in an enterprise storage big disk constructed based on a preset algorithm; based on the time slices, determining storage media for storing the label data according to a preset cold and hot backup data storage mode; calling the label data in the storage media based on a preset label preloading mechanism; and performing data analysis on the label data to acquire target customer group data. The enterprise storage big disk and the label data for storing data are compressed and converted through the preset algorithm, the storage space for storing data is saved, the efficiency of system query is improved through the cold and hot backup data and the preset label preloading mechanism, and the label and the label value are dynamically increased, so that the current real-time growth data, such as big data analysis processing requirements under the condition of 100 million data, are solved.

Description

Technical Field

[0001] This invention relates to the field of big data, and in particular to a big data analysis and processing method, apparatus, equipment, and medium. Background Technology

[0002] With the rapid development of the market economy and the continuous advancement of digital transformation in banks, the use of big data analytics to identify high-quality and potential customers online, as well as to assess customer risk, has become increasingly important, leading to a growing demand for online big data marketing. Specifically, one existing method to meet the big data marketing needs of banks' corporate banking sector is to aggregate and map massive amounts of corporate business, transaction, and asset information into corporate profiles. Banks not only use these profiles to identify corporate risk information but also conduct targeted marketing based on the corresponding corporate characteristics. Furthermore, targeted marketing has greatly facilitated banks' intelligent investment, digital risk control, and precision marketing efforts.

[0003] However, existing methods for identifying enterprise risk information based on enterprise profiles and analyzing and processing enterprise characteristics based on such risk information still suffer from the problem that the storage methods of enterprise data and enterprise tag data lead to a surge in the number of enterprises and the corresponding increase in the tag dimensions of enterprise tag data. Under the premise of the original data storage method, the data processing methods for enterprise data and enterprise tag data cannot meet the requirements of real-time data analysis and processing for hundreds of millions of data points.

[0004] Specifically, on the one hand, the current storage of enterprise data and enterprise tag data is mainly done in databases with enterprise data as rows and enterprise tag data as columns. This storage method consumes a lot of resources and space, makes it difficult to store historical data, has slow query performance, and makes it difficult to dynamically adjust and modify enterprise data and enterprise tag data in the table, resulting in poor flexibility. On the other hand, with the explosive growth of tag dimensions in enterprise tag data, storing enterprise tag data in databases will cause the time to query enterprise data through enterprise tag data to increase exponentially. This storage and retrieval method is increasingly unable to meet the requirements of real-time data growth, such as big data analysis and processing requirements with hundreds of millions of data points. Summary of the Invention

[0005] The main objective of this invention is to propose a big data analysis and processing method, apparatus, device, and medium, aiming to solve the problem that data analysis and processing methods caused by data storage methods cannot meet the requirements of real-time data analysis and processing under hundreds of millions of data volumes.

[0006] To achieve the above objectives, the present invention provides a big data analysis and processing method, which includes the following steps:

[0007] Obtain time slices of tag data in an enterprise storage disk constructed based on a preset algorithm;

[0008] Based on the time slice, the storage medium for storing the tag data is determined according to the preset cold and hot backup data storage method;

[0009] Based on a preset tag preloading mechanism, the tag data in the storage medium is retrieved;

[0010] Data analysis is performed on the label data to obtain target customer group data.

[0011] Preferably, the enterprise storage disk includes at least a full-data-volume intra-row enterprise disk, a full-data-volume inter-row enterprise disk, and a time-slice enterprise disk. Before the step of obtaining the time slices of tagged data in the enterprise storage disk constructed based on a preset algorithm, the big data analysis and processing method further includes:

[0012] Using the enterprise customer number as the primary key, mapping it to the enterprise ID within the row, and persistently using the enterprise ID within the row to reserve space in the storage area, the full-scale enterprise dashboard within the row is constructed.

[0013] Using the enterprise identifier as the primary key, mapping it to the off-line enterprise ID, and persistently using the off-line enterprise ID to reserve space in the storage area, the full off-line enterprise dashboard is constructed.

[0014] Obtain time slices within a preset time period, and construct the time slice enterprise dashboard based on the enterprises whose activity level within the time slices reaches a preset standard in the full-scale intra-industry enterprise dashboard and the full-scale inter-industry enterprise dashboard.

[0015] Preferably, the step of determining the storage medium for storing the tag data based on the time slice and according to a preset cold / hot backup data storage method includes:

[0016] If the time slice is of the first preset type, then the tag data corresponding to the time slice of the first preset type is used as cold backup data, and the cold backup data is stored in the cloud server ECS.

[0017] If the time slice is of the second preset type, then the tag data corresponding to the second preset type time slice is used as hot backup data, and the hot backup data is stored in the memory storage Redis.

[0018] Preferably, before the step of calling the tag data in the storage medium based on the preset tag preloading mechanism, the big data analysis and processing method further includes:

[0019] Obtain the usage frequency of the tags in the tag data;

[0020] The cold and hot backup data corresponding to the tag data are stored according to the preset arrangement order corresponding to the usage frequency.

[0021] Preferably, the step of calling the tag data in the storage medium based on the preset tag preloading mechanism includes:

[0022] TopN tag application based on the preset tag preloading mechanism, setting the TopN algorithm execution parameters of the preloading mechanism;

[0023] According to the TopN algorithm execution parameters, the corresponding tag data is retrieved from the storage medium.

[0024] Preferably, the step of performing data analysis on the tag data to obtain target customer group data includes:

[0025] Obtain the basic customer group data corresponding to the enterprise data;

[0026] The basic customer data is traversed and drilled down to obtain customer data that meets the query conditions.

[0027] The data of customer groups that meet the query conditions are traversed through the filtering conditions to obtain the corresponding target customer group data.

[0028] Preferably, after the step of performing data analysis on the tag data to obtain target customer group data, the big data analysis and processing method further includes:

[0029] Obtain the tag data to be analyzed;

[0030] Perform an AND operation between the tag data and the target customer group to calculate the customer group data in the target customer group that corresponds to the tag value of the tag data;

[0031] Based on the customer group data, identify the non-customer group data in the target customer group that corresponds to the tag value of the tag data.

[0032] Furthermore, to achieve the above objectives, embodiments of the present invention also propose a big data analysis and processing device, the big data analysis and processing device comprising:

[0033] The routing module is used to obtain time slices of tag data in the enterprise storage disk constructed based on a preset algorithm;

[0034] The storage module is used to determine the storage medium for storing the enterprise data based on the time slice and according to a preset cold and hot backup data storage method;

[0035] The data loading module is used to call the tag data in the storage medium based on a preset tag preloading mechanism;

[0036] The analysis module is used to perform data analysis on the enterprise data to obtain target customer group data.

[0037] Preferably, the storage module is used for:

[0038] If the time slice is of a first preset type, then the tag data corresponding to the time slice of the first preset type is used as cold backup data, and the cold backup data is stored in the cloud server ECS.

[0039] If the time slice is of the second preset type, then the tag data corresponding to the second preset type time slice is used as hot backup data, and the hot backup data is stored in the memory storage Redis.

[0040] Preferably, the data loading module is used for:

[0041] TopN tag application based on the preset tag preloading mechanism, setting the TopN algorithm execution parameters of the preloading mechanism;

[0042] According to the TopN algorithm execution parameters, the corresponding tag data is retrieved from the storage medium.

[0043] Preferably, the analysis module is used for:

[0044] Obtain the basic customer group data corresponding to the enterprise data;

[0045] The basic customer data is traversed and drilled down to obtain customer data that meets the query conditions.

[0046] The data of customer groups that meet the query conditions are traversed through the filtering conditions to obtain the corresponding target customer group data.

[0047] Furthermore, to achieve the above objectives, this invention also proposes an apparatus comprising a memory, a processor, and a big data analysis and processing program stored in the memory and executable on the processor. The big data analysis and processing program is executed by the processor to implement the big data analysis and processing method steps described above.

[0048] In addition, to achieve the above objectives, the present invention also provides a medium, which is a computer-readable storage medium storing a big data analysis and processing program, wherein the big data analysis and processing program, when executed by a processor, implements the steps of the big data analysis and processing method described above.

[0049] This invention proposes a big data analysis and processing method, system, device, equipment, and medium. The steps of the big data analysis and processing method include: acquiring time slices of tag data in an enterprise storage disk constructed based on a preset algorithm; determining the storage medium for storing the tag data based on the time slices and a preset cold / hot backup data storage method; calling the tag data in the storage medium based on a preset tag preloading mechanism; and performing data analysis on the tag data to obtain target customer group data. The preset algorithm compresses and transforms the enterprise storage disk and tag data, saving storage space. The cold / hot backup data and preset tag preloading mechanism achieve precise storage of tag data, improving the accuracy of tag data retrieval. Furthermore, it accelerates the response speed of frequently used data in the system, saving resource consumption in the main task process. Finally, by performing data analysis on the tag data, real-time calculation of tag data "drill-down" and "filtering" is achieved, allowing for dynamic addition of tags and tag values ​​at any time. This solves the problem that the data analysis and processing methods caused by the data storage method cannot meet the real-time growth demands of big data analysis and processing. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the big data analysis and processing method of the present invention;

[0051] Figure 2 This is a flowchart illustrating the first embodiment of the big data analysis and processing method of the present invention;

[0052] Figure 3 This is a schematic diagram of the storage format of tag data in the first embodiment of the big data analysis and processing method of the present invention;

[0053] Figure 4 This is a schematic diagram of the specific process of step S100 in the second embodiment of the big data analysis and processing method of the present invention;

[0054] Figure 5 This is a flowchart illustrating the second embodiment of the big data analysis and processing method of the present invention;

[0055] Figure 6 This is a flowchart illustrating the third embodiment of the big data analysis and processing method of the present invention;

[0056] Figure 7 This is a flowchart illustrating the fourth embodiment of the big data analysis and processing method of the present invention;

[0057] Figure 8 This is a schematic diagram illustrating the specific process of the TopN preloading mechanism for preloading the cold and hot backup data in the fourth embodiment of the big data analysis and processing method of the present invention.

[0058] Figure 9 This is a flowchart illustrating the fifth embodiment of the big data analysis and processing method of the present invention;

[0059] Figure 10 This is a schematic diagram of the functional modules of the big data analysis and processing device of the big data analysis and processing method of the present invention.

[0060] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0061] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0062] Specifically, refer to Figure 1 , Figure 1 This is a schematic diagram of the hardware operating environment involved in the embodiment of the big data analysis and processing method of the present invention.

[0063] The equipment in this embodiment of the invention can be a welding device.

[0064] like Figure 1 As shown, the device may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0065] Those skilled in the art will understand that Figure 1 The device structure shown does not constitute a limitation on the device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0066] To better understand the above technical solutions, exemplary embodiments of this disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art.

[0067] Based on, but not limited to, the above-described terminal device architecture, this invention proposes an embodiment of the big data analysis and processing method.

[0068] Specifically, refer to Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the big data analysis and processing method of the present invention, which includes:

[0069] Step S10: Obtain time slices of tag data in the enterprise storage disk constructed based on a preset algorithm;

[0070] Step S20: Based on the time slice, determine the storage medium for storing the tag data according to the preset cold and hot backup data storage method;

[0071] Step S30: Based on the preset tag preloading mechanism, retrieve the tag data in the storage medium;

[0072] Step S40: Perform data analysis on the tag data to obtain target customer group data.

[0073] The big data analysis and processing method in this application uses a preset algorithm to compress and convert the enterprise storage disk and tag data used for data storage, saving storage space. It achieves accurate storage of tag data through cold and hot backup data and preset tag preloading mechanism, and realizes real-time calculation of tag data "drill-down" and "filtering" through data analysis, and can dynamically add tags and tag values ​​at any time.

[0074] The following will provide a detailed explanation of each step:

[0075] Step S10: Obtain time slices of tag data in the enterprise storage disk constructed based on a preset algorithm;

[0076] In one specific embodiment, the BitMap algorithm is used to mark the existence of a certain enterprise by using a bit to store the data. The transaction time between the enterprise and the bank is used as a time slice in the tag data, which effectively marks the enterprise information, compresses the data size of the enterprise data, and saves the storage space.

[0077] Furthermore, by constructing an enterprise storage disk to store enterprise data, the enterprise data is sorted and persistently allocated according to the differences in the enterprise's industry-wide data, transaction data, and transaction time data, and stored in different enterprise storage disks. For the enterprise tag data corresponding to each day of the above time slice, it is stored through the above enterprise storage disk. Unlike the traditional database storage method that uses enterprise data as the primary key and tags as columns, this embodiment uses "time slice-tag-tag value" as the primary key and uses the BitMap algorithm to transform and store the data, thereby improving the efficiency of data storage.

[0078] Reference Figure 3 , Figure 3 This is a schematic diagram illustrating the storage format of enterprise tag data in this embodiment.

[0079] Step S20: Based on the time slice, determine the storage medium for storing the tag data according to the preset cold and hot backup data storage method;

[0080] In one specific embodiment, based on the time slice of the enterprise tag data, it is determined whether the data of the enterprise corresponding to the tag data is cold backup data or hot backup data, and based on the difference between cold backup data and hot backup data, the storage medium for storing the tag data of these enterprises is determined.

[0081] Specifically, based on the time slices mentioned above, the data can be divided into two categories. One category is where enterprises conduct transactions at the end of the month. The tag data for these enterprises can be judged as infrequently used and is called cold backup data. The other category is where enterprises conduct transactions in the next few days. The tag data for these enterprises can be judged as frequently used and is called hot backup data.

[0082] Furthermore, the aforementioned cold backup data is stored on a cloud server ECS, which employs a distributed file storage method. The infrequently used enterprise tag data is distributed across multiple servers, and these distributed storage resources form a virtual storage device for data storage, reducing the storage cost of cold backup data. The aforementioned hot backup data is stored in memory storage Redis, which caches frequently used enterprise tag data from the past few days in memory. This enables the synchronization of enterprise tag data from the master server to any number of slave servers, improving the retrieval efficiency of hot backup data.

[0083] Step S30: Based on the preset tag preloading mechanism, retrieve the tag data in the storage medium;

[0084] In one specific embodiment, enterprise tag data is stored on different storage media. Applications that call enterprise tag data load and use the data through a pseudo-cluster. Based on the Top N tag preloading mechanism, hot backup data in Redis is loaded and used through a hot cluster, and cold backup data on ECS is loaded and used through a cold cluster.

[0085] Furthermore, the main application that retrieves enterprise tag data uses TopN tags. Based on the specific parameters of the TopN tags, the corresponding enterprise tag data is preloaded from the data warehouse. This preloaded enterprise tag data is obtained by filtering the existing enterprise tag data. For example, if Redis can retrieve the Top 10,000 tags with the tag "average monthly revenue," the tag data of the top 10,000 enterprises with average monthly revenue from the hot backup data is preloaded and ready to be retrieved, thus speeding up the system's response time.

[0086] Step S40: Perform data analysis on the tag data to obtain target customer group data.

[0087] In one specific embodiment, the "drill-down" and "filter" functions are used to achieve precise customer group analysis for enterprise profiling. By performing conditional traversal on the tags in the tag data and simplifying the tag data according to the filtering conditions, the filtered target customer group BitMap data is obtained.

[0088] In this embodiment, the BitMap algorithm is used to compress and convert the enterprise storage disk and tag data, saving storage space. Based on the cold and hot backup data storage method and the TopN tag preloading mechanism, the tag data is stored accurately, which speeds up the response speed of frequently used data in the system and saves the resource consumption of the main task process. By performing data analysis on the tag data, real-time calculation of tag data "drill-down" and "filtering" is realized to obtain the corresponding target customer group data.

[0089] Furthermore, based on the first embodiment of the big data analysis and processing method of this application, a second embodiment of the big data analysis and processing method of this application is proposed.

[0090] The difference between the second embodiment of the big data analysis and processing method and the first embodiment is that, in this embodiment, in step S10, the big data analysis and processing method further includes a scheme for constructing an enterprise storage disk based on the BitMap algorithm. Before step S10, the method also includes:

[0091] Step S100: Construct an enterprise storage disk based on the BitMap algorithm.

[0092] Reference Figure 4 , Figure 4This is a schematic diagram of the specific process of step S100 in the big data analysis and processing method of this embodiment. Before obtaining the time slice of the tag data in the enterprise storage disk constructed based on the preset algorithm, step S100 also requires constructing the enterprise storage disk through the BitMap algorithm. The enterprise storage disk includes at least the full intra-row enterprise disk, the full extra-row enterprise disk, and the time slice enterprise disk.

[0093] Reference Figure 5 Step S100 specifically includes:

[0094] Step S101 uses the enterprise customer number as the primary key, maps it to the enterprise ID in the row, persists the enterprise ID in the row to occupy storage space, and constructs the full-scale enterprise dashboard in the row.

[0095] In one specific embodiment, when a corporate customer has already transacted with the enterprise, a corporate customer number has been created for that corporate customer. This corporate customer number is a unique intra-industry enterprise ID created by the corporate customer by mapping the enterprise name and organization name to intra-industry enterprises. The corporate customer number is used as the primary key to map intra-industry enterprises. Intra-industry enterprise IDs are sorted starting from 1 in the enterprise storage disk and are persistently used as placeholders in the storage database to generate an intra-industry enterprise BitMap disk, such as [1, 2, 3, ...]. For subsequently added intra-industry enterprises, if they are not in the current disk, they are added to the intra-industry disk by placing placeholders with their enterprise IDs.

[0096] Step S102: Use the enterprise identifier as the primary key to map the off-line enterprise ID, persist the off-line enterprise ID to the storage space, and construct the full off-line enterprise dashboard.

[0097] In one specific embodiment, when an external enterprise does not transact with this enterprise, the external enterprise ID is mapped using the enterprise name or enterprise registration number, which are unique enterprise identifiers within the industry. The external enterprise IDs are sorted starting from 50 million in the enterprise storage disk, and are persistently used to place space in the storage database, generating an external enterprise BitMap disk, [50000001, 50000002, 50000003, ...]. For subsequently added external enterprises, if they are not in the current disk, they are added to the external enterprise disk by placing space with their external enterprise IDs.

[0098] Step S103: Obtain time slices within a preset time period. Based on the enterprises in the full-scale intra-industry enterprise dashboard and the full-scale inter-industry enterprise dashboard whose activity level reaches a preset standard within the time slice, construct the time slice enterprise dashboard.

[0099] In one specific embodiment, the corresponding intra-industry enterprise ID and extra-industry enterprise ID are queried through the full intra-industry enterprise dashboard and the full extra-industry enterprise dashboard to obtain the enterprise ID and the enterprise tag data corresponding to the enterprise ID within a preset time slice. The preset time slice may be the enterprise data of a certain day. Based on the above time slice and the enterprise tag data corresponding to the enterprise ID, the intra-industry enterprise dashboard of the time slice is constructed as [1, 3, 4, ...], the extra-industry enterprise dashboard of the time slice is [50000001, 50000003, ...], and the full enterprise dashboard of the time slice is [1, 3, 4, ..., 50000001, 50000003, ...].

[0100] This embodiment uses the BitMap algorithm to construct an enterprise storage disk, which uses "time slice-tag-tag value" as the primary key to compress and convert enterprise tag data within and across the industry, greatly saving data storage space.

[0101] Furthermore, based on the first and second embodiments of the big data analysis and processing method of this application, a third embodiment of the big data analysis and processing method of this application is proposed.

[0102] The difference between the third embodiment of the big data analysis and processing method and the first, second, and third embodiments of the diagnostic teaching method is that this embodiment refines the storage medium for storing the tag data based on step S20, according to the time slice and a preset cold / hot backup data storage method, referring to... Figure 6 Specifically, it includes:

[0103] Step S21: If the time slice is a first preset type, then the tag data corresponding to the first preset type time slice is used as cold backup data, and the cold backup data is stored in the cloud server ECS.

[0104] In one specific embodiment, the time type of the tag data is determined to be a first preset type time slice based on the time slice of the tag data. Specifically, when the time when the enterprise in the above tag data transacts with this enterprise is at the end of the month or further away, the time slice of the tag data corresponding to the enterprise belongs to the first preset type time slice. The tag data corresponding to the first preset type time slice is cold backup data, and the above enterprise tag data, which is cold backup data, will be stored in the cloud server ECS.

[0105] Furthermore, the aforementioned ECS adopts a distributed file storage method, distributing the enterprise tag data that is not frequently used across multiple servers, and using these distributed storage resources to form a virtual storage device for data storage, thereby reducing the storage cost of cold backup data.

[0106] Step S22: If the time slice is a second preset type, then the tag data corresponding to the second preset type time slice is used as hot backup data, and the hot backup data is stored in the memory storage Redis.

[0107] In one specific embodiment, the time type of the tag data is determined to be a second preset type time slice based on the time slice of the tag data. Specifically, when the time when the enterprise in the above tag data transacts with this enterprise is a preset number of days in advance or closer to the current time, the time slice of the tag data corresponding to the enterprise belongs to the second preset type time slice. The tag data corresponding to the second preset type time slice is hot backup data, and the above enterprise tag data, which is hot backup data, will be stored in the memory storage Redis.

[0108] Furthermore, the aforementioned memory-based Redis storage method caches frequently used enterprise tag data in memory over the past few days, enabling the synchronization of enterprise tag data from the master server to any number of slave servers, thus improving the efficiency of hot backup data retrieval.

[0109] In this embodiment, tag data is classified into cold backup data and hot backup data by time slicing. By classifying and storing enterprise tag data through cold and hot backup data storage methods, the response speed of hot backup data during the retrieval process is accelerated, the consumption of process resources during the main task is saved, and the efficiency of querying hundreds of millions of data in real time is improved.

[0110] Furthermore, based on the first, second, and third embodiments of the big data analysis and processing method of this application, a fourth embodiment of the big data analysis and processing method of this application is proposed.

[0111] The fourth embodiment of the big data analysis and processing method differs from the first, second, and third embodiments of the diagnostic teaching method in that this embodiment is a refinement of step S30, which involves calling the tag data in the storage medium based on a preset tag preloading mechanism, referring to... Figure 7 Specifically, it includes:

[0112] Step S31: Based on the preset tag preloading mechanism, the Top N tag application sets the execution parameters of the Top N algorithm of the preloading mechanism;

[0113] Step S32: According to the Top N algorithm execution parameters, retrieve the corresponding tag data from the storage medium.

[0114] In one specific embodiment, based on customer drill-down, preview, and download requests, the execution parameters of the Top N preloading mechanism are determined on the main application. Depending on the different execution parameters of the Top N preloading mechanism, dimension tables and tag data are retrieved from the cloud host of the data warehouse. When it is determined that the data retrieved by the execution parameters is cold backup data, the corresponding tag data is retrieved in the ECS according to the tag slice of the cold backup data; when it is determined that the data retrieved by the execution parameters is hot backup data, the corresponding tag data is retrieved in Redis according to the tag slice of the hot backup data.

[0115] Reference Figure 8 , Figure 8 This embodiment describes the specific process of preloading the cold and hot backup data using the Top N preloading mechanism.

[0116] Before the step of calling the tag data in the storage medium based on the preset tag preloading mechanism, the big data analysis and processing method further includes:

[0117] Obtain the usage frequency of the tags in the tag data;

[0118] The cold and hot backup data corresponding to the tag data are stored according to the preset arrangement order corresponding to the usage frequency.

[0119] In one specific embodiment, whether it is cold backup data stored in ECS or hot backup data stored in Redis, they are stored in the corresponding storage media in a preset order.

[0120] Furthermore, the aforementioned preset sorting order is based on the frequency of use of the tag data. The more frequently a company uses its tag data, the higher its sorting order will be. The tag data is loaded using a Top N preloading mechanism according to the aforementioned preset sorting order.

[0121] This embodiment uses a Top N tag preloading mechanism to preload customer group data that meets preset standards, which speeds up the response speed of system requests and improves the query performance of tag data.

[0122] Furthermore, based on the first, second, third, and fourth embodiments of the big data analysis and processing method of this application, a fifth embodiment of the big data analysis and processing method of this application is proposed.

[0123] The fifth embodiment of the big data analysis and processing method differs from the first, second, third, and fourth embodiments of the diagnostic teaching method in that this embodiment performs data analysis on the tag data in step S40 to obtain detailed target customer group data, referring to... Figure 9 Specifically, it includes:

[0124] Step S41: Obtain the basic customer group data corresponding to the enterprise data;

[0125] In one specific embodiment, a basic customer group BitMap is obtained by selecting from the full-scale in-line enterprise dashboard BitMap, the full-scale out-of-line enterprise dashboard BitMap, and the full-scale time-slice enterprise dashboard BitMap.

[0126] Step S42: Iterate through the basic customer group data according to the drill-down conditions to obtain customer group data that meets the query conditions;

[0127] In one specific embodiment, the "drill-down" function is used to achieve precise customer group analysis of enterprise profiles. The "AND" operation is used between different tags in the tag data, and the "OR" operation is used for the same tags in the tag data. The basic customer group BitMap obtained above is further simplified to obtain the final drill-down target customer group.

[0128] Furthermore, the obtained basic customer group BitMap is further simplified. By traversing each preset "drill-down" condition through these basic customer group BitMaps, the specific target customer group for drilling down is obtained. For example, traversing each "drill-down" condition "A1 AND B1 AND (C1 OR C2)", obtaining BitMap (A1), and performing an AND operation with the original customer group BitMap to obtain the latest customer group BitMap; then traversing the next condition and performing an AND operation to obtain a more concise target customer group BitMap for drilling down.

[0129] Step S43: Iterate through the customer group data that meets the query conditions and filter the data to obtain the corresponding target customer group data.

[0130] In one specific embodiment, the "filtering" function enables precise customer segment analysis for enterprise profiling. By performing a "NOT" operation on the obtained original customer segment Bitmap, customer segment data that does not meet the criteria can be filtered out, thus identifying more accurate target customer segment data, such as: "(A1 OR B1) AND C1 OR (NOT D1)". Furthermore, the aforementioned "filtering" function can be performed simultaneously with the "drill-down" function, achieving a two-level nested structure.

[0131] After the step of performing data analysis on the tag data to obtain target customer group data, the big data analysis and processing method further includes:

[0132] Obtain the tag data to be analyzed;

[0133] Perform an AND operation between the tag data and the target customer group to calculate the customer group data in the target customer group that corresponds to the tag value of the tag data;

[0134] Based on the customer group data, identify the non-customer group data in the target customer group that corresponds to the tag value of the tag data.

[0135] In one specific embodiment, after obtaining the target customer group data BitMap, the distribution of tags in the target customer group data BitMap can be statistically analyzed to determine the distribution of customer group data under different tags in the target customer group data BitMap.

[0136] Specifically, when you need to check the label distribution of labels LAB1 and LAB2, you can iterate through each label that needs to be counted, such as LAB1, get its label value, and obtain A1, A2, etc. Then, obtain the BitMap of its label value ("time slice-LAB1-A1"), perform an AND operation with the BitMap of the target customer group, and calculate to get the number of companies in the target customer group whose label LAB1 has the value A1. Similarly, you can get the number of companies with other label values ​​that do not belong to label LAB1.

[0137] This embodiment calculates enterprise profiles in real time by drilling down and filtering the tag data, which accelerates the acquisition of precise customer groups. Furthermore, it performs real-time analysis of target customer groups based on enterprise profiles, enhances the visualization effect of data during the data query process, and improves the flexibility of tags and tag values ​​in the database.

[0138] Furthermore, embodiments of the present invention also propose a big data analysis and processing device, referring to... Figure 10 , Figure 10 This is a schematic diagram of the functional modules of the big data analysis and processing device involved in an embodiment of the big data analysis and processing method of the present invention. Figure 10 As shown, the big data analysis and processing device includes:

[0139] The routing module 10 is used to obtain time slices of tag data in the enterprise storage disk constructed based on a preset algorithm;

[0140] Storage module 20 is used to determine the storage medium for storing the enterprise data based on the time slice and according to a preset cold and hot backup data storage method;

[0141] The data loading module 30 is used to call the tag data in the storage medium based on a preset tag preloading mechanism;

[0142] Analysis module 40 is used to perform data analysis on the enterprise data to obtain target customer group data.

[0143] The principles and implementation process of data analysis and processing in this embodiment are described in the above embodiments and will not be repeated here.

[0144] Furthermore, this invention also proposes an apparatus comprising a memory, a processor, and a big data analysis and processing program stored in the memory and executable on the processor. When executed by the processor, the big data analysis and processing program implements the steps of the big data analysis and processing method described in the above embodiments.

[0145] In addition, to achieve the above objectives, the present invention also provides a medium, which is a computer-readable storage medium storing a big data analysis and processing program, wherein the big data analysis and processing program, when executed by a processor, implements the steps of the big data analysis and processing method described above.

[0146] Since this big data analysis and processing program employs all the technical solutions of all the aforementioned embodiments when executed by the processor, it possesses at least all the beneficial effects brought about by all the technical solutions of all the aforementioned embodiments, which will not be elaborated upon here.

[0147] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0148] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0149] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0150] The above are merely preferred embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A big data analysis and processing method, characterized in that, The big data analysis and processing method includes the following steps: Obtain time slices of tag data in an enterprise storage dashboard constructed based on a preset algorithm; the time slice refers to the transaction time during which the enterprise conducts transactions. Based on the time slice, and according to the preset cold and hot backup data storage method, the storage medium for storing the tag data is determined; the data storage is performed using the time slice, tag, and tag value as the primary key. Based on a preset tag preloading mechanism, the tag data in the storage medium is retrieved; Data analysis is performed on the tag data to obtain target customer group data; The enterprise storage disk includes at least a full-row in-row enterprise disk, a full-row out-of-row enterprise disk, and a time-slice enterprise disk. The construction of the enterprise storage disk is based on the Bitmap algorithm. Prior to the step of obtaining time slices of tag data in an enterprise storage disk constructed based on a preset algorithm, the big data analysis and processing method further includes: Using the enterprise customer number as the primary key, mapping it to the enterprise ID within the row, and persistently using the enterprise ID within the row to reserve space in the storage area, the full-scale enterprise dashboard within the row is constructed. Using the enterprise identifier as the primary key, mapping it to the off-line enterprise ID, and persistently using the off-line enterprise ID to reserve space in the storage area, the full off-line enterprise dashboard is constructed. By querying the full-scale enterprise dashboard within the industry and the full-scale enterprise dashboard outside the industry, the corresponding enterprise IDs within the industry and the enterprise IDs outside the industry are obtained, and the enterprise IDs and the corresponding enterprise tag data within the preset time slice are obtained; the time slice enterprise dashboard is constructed based on the time slice and the enterprise tag data corresponding to the enterprise ID. The step of performing data analysis on the tag data to obtain target customer group data includes: Select from the full-scale enterprise dashboard within the industry, the full-scale enterprise dashboard outside the industry, and the full-scale time-slice enterprise dashboard to obtain the basic customer group data corresponding to the tag data; The basic customer data is traversed and drilled down to obtain customer data that meets the query conditions. Drilling down refers to using AND operations between different tags in the tag data and using OR operations on the same tags in the tag data. The data of customer groups that meet the query conditions are traversed through the filtering conditions to obtain the corresponding target customer group data.

2. The big data analysis and processing method as described in claim 1, characterized in that, The step of determining the storage medium for storing the tag data based on the time slice and according to a preset cold / hot backup data storage method includes: If the time slice is of the first preset type, then the tag data corresponding to the time slice of the first preset type is used as cold backup data, and the cold backup data is stored in the cloud server ECS. If the time slice is of the second preset type, then the tag data corresponding to the second preset type time slice is used as hot backup data, and the hot backup data is stored in the memory storage Redis.

3. The big data analysis and processing method as described in claim 2, characterized in that, Before the step of calling the tag data in the storage medium based on the preset tag preloading mechanism, the big data analysis and processing method further includes: Obtain the usage frequency of the tags in the tag data; The cold and hot backup data corresponding to the tag data are stored according to the preset arrangement order corresponding to the usage frequency.

4. The big data analysis and processing method as described in claim 3, characterized in that, The step of calling tag data in the storage medium based on the preset tag preloading mechanism includes: Based on the preset tag preloading mechanism Top N tag application, set the execution parameters of the Top N algorithm of the preloading mechanism; According to the Top N algorithm execution parameters, the corresponding tag data is retrieved from the storage medium.

5. The big data analysis and processing method as described in claim 1, characterized in that, After the step of performing data analysis on the tag data to obtain target customer group data, the big data analysis and processing method further includes: Obtain the tag data to be analyzed; Perform an AND operation between the tag data and the target customer group to calculate the customer group data in the target customer group that corresponds to the tag value of the tag data; Based on the customer group data, identify the non-customer group data in the target customer group that corresponds to the tag value of the tag data.

6. A big data analysis and processing device, characterized in that, The big data analysis and processing device includes: The routing module is used to obtain time slices of tag data in the enterprise storage dashboard constructed based on a preset algorithm; the time slice refers to the transaction time when the enterprise conducts transactions. The storage module is used to determine the storage medium for storing the tag data based on the time slice and according to a preset cold and hot backup data storage method; the data storage is performed using the time slice, tag, and tag value as the primary key. The data loading module is used to call the tag data in the storage medium based on a preset tag preloading mechanism; The analysis module is used to perform data analysis on the tag data to obtain target customer group data; The enterprise storage disk includes at least a full-row in-row enterprise disk, a full-row out-of-row enterprise disk, and a time-slice enterprise disk. The construction of the enterprise storage disk is based on the Bitmap algorithm. Before obtaining the time slice of tag data in the enterprise storage disk constructed based on a preset algorithm, the following steps are also included: Using the enterprise customer number as the primary key, mapping it to the enterprise ID within the row, and persistently using the enterprise ID within the row to reserve space in the storage area, the full-scale enterprise dashboard within the row is constructed. Using the enterprise identifier as the primary key, mapping it to the off-line enterprise ID, and persistently using the off-line enterprise ID to reserve space in the storage area, the full off-line enterprise dashboard is constructed. By querying the full-scale enterprise dashboard within the industry and the full-scale enterprise dashboard outside the industry, the corresponding enterprise IDs within the industry and the enterprise IDs outside the industry are obtained, and the enterprise IDs and the corresponding enterprise tag data within the preset time slice are obtained; the time slice enterprise dashboard is constructed based on the time slice and the enterprise tag data corresponding to the enterprise ID. The step of performing data analysis on the tag data to obtain target customer group data includes: Select from the full-scale enterprise dashboard within the industry, the full-scale enterprise dashboard outside the industry, and the full-scale time-slice enterprise dashboard to obtain the basic customer group data corresponding to the tag data; The basic customer data is traversed and drilled down to obtain customer data that meets the query conditions. Drilling down refers to using AND operations between different tags in the tag data and using OR operations on the same tags in the tag data. The data of customer groups that meet the query conditions are traversed through the filtering conditions to obtain the corresponding target customer group data.

7. A device, characterized in that, The device includes a memory, a processor, and a big data analysis and processing program stored in the memory and executable on the processor. When executed by the processor, the big data analysis and processing program implements the steps of the big data analysis and processing method as described in any one of claims 1 to 5.

8. A medium, said medium being a computer-readable storage medium, characterized in that, The computer-readable storage medium stores a big data analysis and processing program, which, when executed by a processor, implements the steps of the big data analysis and processing method as described in any one of claims 1 to 5.

Images