Business data processing method, computer equipment and readable storage medium
By identifying and utilizing cold and hot data warehouses based on time ranges and request types, the method enhances data processing efficiency and accuracy, addressing performance bottlenecks in big data clusters.
Patent Information
- Application Number
- CN202411180986.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-12-03
AI Technical Summary
The existing big data cluster architecture faces performance bottlenecks in data warehouses, leading to inefficient access and processing of business data, regardless of node expansion or resource augmentation.
A method that identifies cold and hot data warehouses based on time ranges and request types to efficiently select and store business data, optimizing data access and storage through separate data warehouses for timely availability.
Improves data processing efficiency and accuracy by ensuring timely access to relevant data, optimizing resource utilization, and maintaining data consistency and security.
Smart Images

Figure CN119066110A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of computer technology, and in particular, relates to a business data processing method, a computer device, and a computer-readable storage medium. Background Art
[0002] In the current big data cluster architecture of various business systems, the data warehouse in the cluster can be divided into an offline data warehouse and a real-time data warehouse according to the different uses of business data. After the cluster performance reaches a bottleneck, this data warehouse structure can no longer significantly improve the cluster performance, whether by increasing the number of nodes or increasing the processing and storage resources of a single node, resulting in the inability to access business data in a timely manner and poor access efficiency. Summary of the invention
[0003] In view of this, the embodiments of the present application provide a business data processing method, a computer device, and a computer-readable storage medium, which can improve the efficiency and accuracy of data processing and ensure the timely availability of data.
[0004] A first aspect of an embodiment of the present application provides a business data processing method, including:
[0005] Obtaining a service data request for a specified service, wherein the service data request carries a time range of required service data;
[0006] Based on the time range of the required business data, cold and hot data identification is performed to obtain a target data warehouse storing the required business data, wherein the target data warehouse is a cold data warehouse or a hot data warehouse associated with the specified business;
[0007] Based on the request type of the business data request, the required business data is selected from an offline data warehouse or a real-time data warehouse associated with the target data warehouse.
[0008] In an implementation of the first aspect, performing hot and cold identification based on the time range of the required business data to obtain a target data warehouse storing the required business data includes:
[0009] Obtaining the effective time range of the hot data warehouse associated with the specified business;
[0010] Comparing the effective time range with the time range of the required service data to obtain a comparison result;
[0011] Based on the comparison result, a target data warehouse for storing the required business data is determined.
[0012] In an implementation of the first aspect, determining, based on the comparison result, a target data warehouse storing the required business data includes:
[0013] If the comparison result indicates that the time range of the required business data belongs to the valid time range, the hot data warehouse associated with the specified business is determined as the target data warehouse storing the required business data;
[0014] If the comparison result indicates that part of the time range of the required business data belongs to or does not belong to the valid time range, the cold data warehouse associated with the designated business is determined as the target data warehouse storing the required business data.
[0015] In an implementation of the first aspect, the selecting the required business data from an offline data warehouse or a real-time data warehouse associated with the target data warehouse based on the request type of the business data request includes:
[0016] If the request type of the business data request belongs to the first type, selecting the required business data from the real-time data warehouse associated with the target data warehouse, the first type indicating a query type request for business data;
[0017] If the request type of the business data request belongs to the second type, the required business data is selected from the offline data warehouse associated with the target data warehouse, and the second type indicates a statistical request for business data.
[0018] In an implementation manner of the first aspect, the request type of the service data request belongs to the second type, and the method further includes:
[0019] Obtaining statistical results determined based on the required business data;
[0020] If the time range of the required business data belongs to the valid time range of the hot data warehouse, the statistical results are written into the hot data warehouse and the cold data warehouse respectively;
[0021] If part of the time range of the required business data belongs to or does not belong to the effective time range of the hot data warehouse, the statistical result is written into the cold data warehouse.
[0022] In an implementation of the first aspect, the method further includes:
[0023] Acquire the business data to be stored associated with the specified business, and write the business data to be stored into the offline data warehouse of the cold data warehouse associated with the specified business;
[0024] Creating a data copy of the business data to be stored, and migrating the data copy to a specified directory;
[0025] If changes in the content of the designated directory are detected, the business data in the data copy is written into the real-time data warehouse of the cold data warehouse associated with the designated business.
[0026] In an implementation of the first aspect, the method further includes:
[0027] Periodically testing the business data in the hot data warehouse of the specified business to obtain a test result;
[0028] If the detection result indicates that the last update time of any designated business data does not fall within the valid time range of the hot data warehouse, the designated business data is deleted from the hot data warehouse.
[0029] In an implementation of the first aspect, if the offline data warehouse is deployed in an independent manner, the cold data warehouse associated with the specified business corresponds to the first offline data warehouse, and the hot data warehouse associated with the specified business corresponds to the second offline data warehouse;
[0030] If the offline data warehouse is deployed in a shared manner, the cold data warehouse and the hot data warehouse share the third offline data warehouse.
[0031] A second aspect of an embodiment of the present application provides a service data processing device, including:
[0032] An acquisition module, used to acquire a service data request for a specified service, wherein the service data request carries a time range of required service data;
[0033] An identification module, configured to perform cold and hot identification based on the time range of the required business data, and obtain a target data warehouse storing the required business data, wherein the target data warehouse is a cold data warehouse or a hot data warehouse associated with the specified business;
[0034] A selection module is used to select the required business data from an offline data warehouse or a real-time data warehouse associated with the target data warehouse based on the request type of the business data request.
[0035] A third aspect of an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the computer program.
[0036] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.
[0037] In a first aspect of an embodiment of the present application, by obtaining business data requests for specified businesses and the time range of the business data required, cold and hot identification is performed to determine a target data warehouse (cold data warehouse or hot data warehouse) for storing the required business data, and the required business data is selected from an offline data warehouse or real-time data warehouse associated with the target data warehouse according to the type of business data request, thereby improving the efficiency and accuracy of data processing and ensuring the timely availability of data.
[0038] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0040] Figure 1 is a structural diagram of a business data processing system provided in an embodiment of the present application;
[0041] Figure 2 It is a schematic diagram of the implementation flow of the business data processing method provided in the embodiment of the present application;
[0042] Figure 3 is a flowchart of a method for determining a target data warehouse based on time provided in an embodiment of the present application;
[0043] Figure 4 is another flowchart of a method for determining a target data warehouse provided in an embodiment of the present application;
[0044] Figure 5 is a flowchart of a method for obtaining required business data provided by an embodiment of the present application;
[0045] Figure 6 is a flowchart of a method for writing statistical results provided in an embodiment of the present application;
[0046] Figure 7 is a flowchart of a method for writing business data to be stored provided in an embodiment of the present application;
[0047] Figure 8 is another structural diagram of the data warehouse provided in the embodiment of the present application;
[0048] Fig. 9 It is a structural diagram of a data warehouse provided in the related art;
[0049] Fig.10is a schematic diagram of a business data processing device provided in an embodiment of the present application;
[0050] Fig.11 It is a schematic diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0051] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0052] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.
[0053] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0054] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.
[0055] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.
[0056] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.
[0057] In one embodiment, Figure 1 The business data processing system structure diagram of the business system shown includes a data processing unit 100, a data storage unit 200, a data application unit 300, a metadata management unit 400, and a monitoring unit 500.
[0058] In the application, the data processing unit 100 is used to obtain raw data from various data sources associated with the specified business. The data sources may be from various business systems within the enterprise, or may be external sources, such as social media, sensor data, etc. The data processing unit 100 also includes a business tracking subunit 110, a data acquisition subunit 120, and a data cleaning subunit 130. Among them, the business tracking subunit 110 is used to generate business tracking data through multiple business tracking points preset in the business data during the operation of the business system. The data acquisition subunit 120 is used to collect business tracking data and raw business data from other data sources. The data cleaning subunit 130 is used to extract (Extract), transform (Transform) and load (Load) the raw business data of various different data sources collected by the data acquisition subunit 120, obtain the business data to be stored suitable for storage and analysis, and then load the business data to be stored into the data storage unit 300.
[0059] In the application, the data storage unit 200 includes a cold data warehouse and a hot data warehouse, the cold data warehouse is used to store cold data, and the hot data warehouse is used to store hot business data. Both the cold data warehouse and the hot data warehouse are associated with an offline data warehouse and a real-time data warehouse, the offline data warehouse is used to store business data that supports statistical business data requests, and the real-time data warehouse is used to store business data that supports non-statistical query business data requests.
[0060] In the application, the data application unit 300 is used to receive business data requests and access business data in the cold data warehouse or the hot data warehouse through the data source switching unit 310. The monitoring unit 500 is used to monitor the status of the service node where the big data cluster of the entire business application is located together with the deployment monitoring component. The metadata management unit 400 includes a cold data metadata unit 410 for independently managing the metadata of the cold data warehouse and a hot data metadata unit 420 for independently managing the metadata of the hot data warehouse.
[0061] In one embodiment, Figure 2 As shown, a business data processing method is provided. The method is based on Figure 1 The business data processing system in the embodiment is implemented by taking the method applied to a computer device (which may be a service node in a big data cluster associated with a specified business system) as an example, and includes the following steps S101 to S103. Among them:
[0062] Step S101: obtaining a service data request for a specified service, wherein the service data request carries a time range of required service data.
[0063] In the application, during the operation of the specified business system, when the upper-level business application needs to use the business data in the data warehouse, the computer device obtains the business data request initiated through the data request interface of the application program, and the business data request carries the time range of the business data required by the upper-level business application. The time range refers to the time period to which the business data is written into the data warehouse. Business data requests may include but are not limited to submitting real-time data query requests, submitting offline data statistics task requests, submitting report data display requests, etc.
[0064] Step S102, based on the time range of the required business data, perform cold and hot identification to obtain a target data warehouse storing the required business data, wherein the target data warehouse is a cold data warehouse or a hot data warehouse associated with the specified business.
[0065] In the application, the computer device obtains the time range of the required business data, and combines the effective time range of the hot data warehouse to perform hot and cold identification on the required business data, so as to determine the target data warehouse for storing the learned business data based on the identification results. The target data warehouse is a hot data warehouse or a cold data warehouse associated with the specified business system. For example, the specified business is a game application, and the business data request is to obtain user login information for the game application within the past month. The business data required by the business data request is the login information of the past month, and the time range of the required business data is the past month (i.e., one month from the current time point). The hot data warehouse in the data warehouse of the game application is used to store the business data of the past three months, and the cold data warehouse is used to store the full amount of business data (or to store various business data that can be statistically analyzed). The computer device performs hot and cold identification based on the time range of the required business data, and obtains that the target data warehouse storing the required business data is the hot data warehouse of the game application.
[0066] Step S103: based on the request type of the business data request, select the required business data from the offline data warehouse or the real-time data warehouse associated with the target data warehouse.
[0067] In the application, the computer device determines that the data warehouse storing the required business data is the cold data warehouse or hot data warehouse of the specified business based on the time range of the required business data, and then further selects the required business data from the real-time data warehouse or offline data warehouse of the target data warehouse according to the request type of the business data request.
[0068] In the application, the request type of the service data request includes a first type for indicating a query type service request, and a second type for indicating a statistics type service request.
[0069] Continuing with the above example, after the computer device determines that the target data warehouse to which the required business data belongs is the hot data warehouse, it further determines that the request type of the business data request is a non-statistical query request, and the server further obtains the required business data from the real-time data warehouse associated with the hot data warehouse.
[0070] In this embodiment, by obtaining business data requests for specified businesses and the time range of their required business data, cold and hot identification is performed to determine the target data warehouse (cold data warehouse or hot data warehouse) that stores the required business data, and the required business data is selected from the offline data warehouse or real-time data warehouse associated with the target data warehouse according to the type of business data request, thereby improving the efficiency and accuracy of data processing and ensuring the timely availability of data.
[0071] In one embodiment, Figure 3 As shown, the implementation process of step S102 includes the following steps S201 to S203. Among them:
[0072] Step S201, obtaining the effective time range of the hot data warehouse associated with the specified business.
[0073] In the application, the hot data warehouse has the attribute of effective time range, which refers to the time span of the writing time of the business data of the hot data warehouse. The effective time range is configured according to the specific application scenario and business needs. For example, the effective time range can be the last month, the last three months, the last six months, etc.
[0074] Step S202: compare the effective time range with the time range of the required service data to obtain a comparison result.
[0075] In the application, the computer device compares the time range of the required business data with the effective time range of the hot data warehouse to obtain a comparison result. The comparison result is used to indicate whether the time range of the required business data belongs to the effective time range of the hot data warehouse.
[0076] Step S203: Based on the comparison result, determine the target data warehouse storing the required business data.
[0077] In the application, the computer device determines the target data warehouse storing the required business data from the cold data warehouse or the hot data warehouse according to the comparison result. The comparison result includes whether the time range of the required business data belongs to the valid time range, whether the time range deployment of the required business data belongs to or does not belong to the valid time range at all.
[0078] In this embodiment, by obtaining the effective time range of the hot data warehouse associated with the specified business and comparing it with the time range of the required business data, the target data warehouse for storing the required business data can be determined based on the comparison result, ensuring effective storage and efficient access to data.
[0079] In one embodiment, Figure 4 As shown, the implementation process of step S203 includes the following steps S301 to S302. Among them:
[0080] Step S301: If the comparison result indicates that the time range of the required business data belongs to the valid time range, the hot data warehouse associated with the specified business is determined as the target data warehouse storing the required business data.
[0081] In the application, if the comparison result indicates that the time range of the required business data belongs to the valid time range, the hot data warehouse associated with the specified business is determined as the target data warehouse storing the required business data, that is, the computer device selects the required business data from the hot data warehouse.
[0082] Step S302: If the comparison result indicates that part of the time range of the required business data belongs to or does not belong to the valid time range, the cold data warehouse associated with the designated business is determined as the target data warehouse storing the required business data.
[0083] In the application, if the comparison result indicates that the time range of the required business data partially belongs to or does not belong to the effective time range, the cold data warehouse associated with the specified business is determined as the target data warehouse for storing the required business data. That is, the computer device selects the required business data from the cold data warehouse.
[0084] In this embodiment, by comparing the time range of the required business data with the effective time range, the target data warehouse storing the required business data is quickly determined, thereby ensuring efficient storage and access to data.
[0085] In one embodiment, Figure 5 As shown, the implementation process of step S103 includes the following steps S401 to S402. Among them:
[0086] Step S401: If the request type of the business data request belongs to the first type, the required business data is selected from the real-time data warehouse associated with the target data warehouse, and the first type indicates a query type request for business data.
[0087] In the application, the request types of business data requests include at least a first type and a second type. The first type indicates query requests for business data, such as obtaining user login information for the most recent month, etc., and the second type indicates statistical requests for business data, such as counting the total purchase amount of game equipment in the most recent month, etc.
[0088] Step S402: If the request type of the business data request belongs to the second type, the required business data is selected from the offline data warehouse associated with the target data warehouse, and the second type indicates a statistical request for business data.
[0089] In the application, the offline data warehouse stores business data for statistics. When the request type of the business data request belongs to the second type, the computer device selects the required business data from the offline data warehouse associated with the target data warehouse (cold data warehouse or hot data warehouse).
[0090] In this embodiment, by selecting a suitable data source according to the type of business data request, the required business data is obtained from the real-time data warehouse for query requests, and from the offline data warehouse for statistical requests, thereby improving data access efficiency and accuracy and optimizing resource utilization.
[0091] In one embodiment, Figure 6As shown, when the request type of the service data request is the second type, the implementation process of the write operation based on the statistical results determined based on the required service data includes the following steps S501 to S503. Among them:
[0092] Step S501: Obtain statistical results determined based on the required business data.
[0093] In the application, for the second type of business data request, the computer device searches for the required business data from the offline data warehouse associated with the cold data warehouse or the hot data warehouse, and obtains the statistical results determined based on the required business data. The computer device can write the statistical results into the data warehouse.
[0094] Step S502: If the time range of the required business data belongs to the valid time range of the hot data warehouse, the statistical results are written into the hot data warehouse and the cold data warehouse respectively.
[0095] In the application, the computer device determines whether the time range of the required business data belongs to the effective time range of the hot data warehouse, and writes the statistical results into the hot data warehouse and the cold data warehouse respectively.
[0096] Step S503: If part of the time range of the required business data belongs to or does not belong to the effective time range of the hot data warehouse, the statistical result is written into the cold data warehouse.
[0097] In the application, the computer device determines whether the time range of the required business data partially belongs to or does not belong to the effective time range of the hot data warehouse, and writes the statistical results only into the cold data warehouse.
[0098] In this embodiment, whether the data falls within the valid time range of the hot data warehouse is determined based on the time range of the data, and the statistical results are written into the hot data warehouse and the cold data warehouse respectively, or only into the cold data warehouse, thereby ensuring efficient storage and management of the data while improving the speed and accuracy of data access.
[0099] In one embodiment, Figure 7 As shown, the implementation process of writing business data into the data warehouse includes the following steps S601 to S603. Among them:
[0100] Step S601: Acquire the business data to be stored associated with the specified business, and write the business data to be stored into the offline data warehouse of the cold data warehouse associated with the specified business.
[0101] In the application, during the operation of the specified business system, the original business data is collected through the business points in the specified business system. Business points refer to the data collection points pre-set in the business system. When a specific event occurs, these points will trigger the corresponding data recording action. Business points can collect user behavior data, system performance data, business logic execution status, etc. Then the original business data is extracted (Extract), transformed (Transform) and loaded (Load) to obtain the business data to be stored associated with the specified business. The computer equipment directly writes the business data to be stored into the offline data warehouse of the cold data warehouse associated with the specified business.
[0102] Step S602: create a data copy of the business data to be stored, and migrate the data copy to a designated directory.
[0103] In the application, the storage devices of the cold data warehouse and the hot data warehouse are different, and the storage devices corresponding to the offline data warehouse and the real-time data warehouse are also different. The computer device creates a data copy of the business data to be stored and migrates the data copy to the specified directory associated with the real-time data warehouse of the cold data warehouse.
[0104] Step S603: If a change in the content of the designated directory is detected, the business data in the data copy is written into the real-time data warehouse of the cold data warehouse associated with the designated business.
[0105] In the application, the computer device listens to the specified directory through the running listening service. When the content of the specified directory is changed, the business data in the data copy is written into the real-time data of the cold data warehouse associated with the specified business.
[0106] It should be noted that the computer equipment will also write the business data to be stored into the hot data warehouse at the same time. Specifically, the business data to be stored is first written into the offline data warehouse of the hot data warehouse, a data copy of the business data to be stored is created, and the data copy is migrated to the designated directory associated with the real-time data warehouse of the hot data warehouse. The listening service listens to the content changes of the designated directory and writes the business data to be stored in the data copy into the real-time data warehouse of the hot data warehouse.
[0107] In this embodiment, the business data to be stored of the specified business is written into the offline data warehouse of the cold data warehouse, and a data copy is created and migrated to the specified directory. When changes in the directory content are detected, the business data in the data copy is written into the real-time data warehouse of the cold data warehouse, thereby achieving efficient storage and updating of data and ensuring the timeliness and consistency of the data.
[0108] In one embodiment, the computer device performs the following operations: periodically detects the business data in the hot data warehouse of the specified business to obtain a detection result; if the detection result indicates that the last update time of any specified business data does not fall within the valid time range of the hot data warehouse, the specified business data is deleted from the hot data warehouse.
[0109] In the application, the computer device will also delete the business data in the hot data warehouse. Specifically, the computer device periodically detects the business data in the hot data warehouse and obtains the detection result, which is used to indicate whether to delete the corresponding business data. If the detection result indicates that the last update time of the specified business data does not belong to the valid time range of the hot data warehouse (that is, there is expired business data in the hot data warehouse), the expired business data will be deleted from the hot data warehouse.
[0110] In this embodiment, by periodically detecting business data in the hot data warehouse and deleting data that is no longer within the valid time range based on the detection results, it can be ensured that only the latest and relevant data is retained in the hot data warehouse, thereby improving the timeliness and accuracy of the data while optimizing the use of storage resources.
[0111] In one embodiment, if the offline data warehouse is deployed in an independent manner, the cold data warehouse associated with the specified business corresponds to the first offline data warehouse, and the hot data warehouse associated with the specified business corresponds to the second offline data warehouse; if the offline data warehouse is deployed in a shared manner, the cold data warehouse and the hot data warehouse share the third offline data warehouse.
[0112] In the application, the offline data warehouse can be deployed in an independent manner, that is, the cold data warehouse corresponds to the first offline data warehouse, the hot data warehouse corresponds to the second offline data warehouse, and the first offline data warehouse and the second offline data warehouse are completely independent. The write time of the business data of the first offline data warehouse does not belong to the effective time range of the hot data warehouse. The write time of the business data of the second offline data warehouse belongs to the effective time range of the hot data warehouse. The offline data warehouse can also be deployed in a shared manner, that is, the cold data warehouse and the hot data warehouse share the same offline data warehouse. The shared offline data warehouse stores all business data used to support statistical business data requests.
[0113] For example, the offline data warehouse is independently deployed as follows: Figure 1 The data warehouse structure shown in the figure, in which the data warehouse is divided into a cold data warehouse and a hot data warehouse. The cold data warehouse includes an independent offline data warehouse (which can be called the first offline data warehouse) and an independent real-time data warehouse, and the hot data warehouse includes an independent offline data warehouse (which can be called the second offline data warehouse) and an independent real-time data warehouse. The offline data warehouse shared deployment method is as follows Figure 8The data warehouse structure shown in the figure, in which the cold data warehouse and the hot data warehouse are respectively associated with their own real-time data warehouses, and the cold data warehouse and the hot data warehouse share the same offline data warehouse.
[0114] Relative to Fig. 9 The data warehouse in the system does not separate the hot and cold data warehouses. Figure 1 or Figure 8 The data warehouse in the data warehouse is divided into a cold data warehouse and a hot data warehouse according to the cold-hot separation index, which can include the time when the business data is written into the data warehouse, the frequency of access to the business data, etc., where the cold-hot separation index can be the time when the business data is written into the data warehouse, or the frequency of access to the business data, etc. This separate storage method helps to improve the efficiency of data reading, because the storage strategy can be adjusted according to the popularity and access frequency of the data, such as placing hot data in the hot data warehouse for fast access; and placing historical data in the cold data warehouse to save storage space and access costs. In addition, deploying corresponding metadata management modules on different servers to independently manage the metadata of the cold data warehouse and the hot data warehouse can not only ensure the stability of the big data cluster of the business system and the security of the data, but also prevent data loss caused by a single server failure, thereby improving the security of the data. All historical data are retained in the cold data warehouse, while the hot data warehouse can retain the business data within the effective time range of the hot data warehouse. Such a data retention strategy can balance the data storage cost and query efficiency.
[0115] In this embodiment, different deployment modes of offline data warehouses can flexibly manage offline data warehouses according to deployment strategies, improve resource utilization and simplify data management processes.
[0116] The present application also provides a business data processing device for executing the steps in the above business data processing method embodiment. Fig.10 As shown, the service data processing device 900 provided in the embodiment of the present application includes:
[0117] The acquisition module 910 is used to acquire a service data request for a specified service, where the service data request carries a time range of required service data.
[0118] The identification module 920 is used to perform cold and hot identification based on the time range of the required business data to obtain a target data warehouse storing the required business data, wherein the target data warehouse is a cold data warehouse or a hot data warehouse associated with the specified business.
[0119] The selection module 930 is used to select the required business data from the offline data warehouse or the real-time data warehouse associated with the target data warehouse based on the request type of the business data request.
[0120] In one embodiment, the identification module is also used to obtain the effective time range of the hot data warehouse associated with the specified business; compare the effective time range with the time range of the required business data to obtain a comparison result; and based on the comparison result, determine the target data warehouse storing the required business data.
[0121] In one embodiment, the identification module is also used to determine the hot data warehouse associated with the specified business as the target data warehouse storing the required business data if the comparison result indicates that the time range of the required business data belongs to the valid time range; if the comparison result indicates that the time range of the required business data partially belongs to or does not belong to the valid time range, determine the cold data warehouse associated with the specified business as the target data warehouse storing the required business data.
[0122] In one embodiment, the selection module is also used to select the required business data from the real-time data warehouse associated with the target data warehouse if the request type of the business data request belongs to the first type, and the first type indicates a query type request for business data; if the request type of the business data request belongs to the second type, the required business data is selected from the offline data warehouse associated with the target data warehouse, and the second type indicates a statistical type request for business data.
[0123] In one embodiment, the request type of the business data request belongs to the second type, and the selection module is further used to obtain statistical results determined based on the required business data; if the time range of the required business data belongs to the effective time range of the hot data warehouse, the statistical results are written into the hot data warehouse and the cold data warehouse respectively; if the time range of the required business data partially belongs to or does not belong to the effective time range of the hot data warehouse, the statistical results are written into the cold data warehouse.
[0124] In one embodiment, the acquisition module is also used to acquire the business data to be stored associated with the specified business, and write the business data to be stored into the offline data warehouse of the cold data warehouse associated with the specified business; create a data copy of the business data to be stored, and migrate the data copy to a specified directory; if a change in the content of the specified directory is detected, the business data in the data copy is written into the real-time data warehouse of the cold data warehouse associated with the specified business.
[0125] In one embodiment, the acquisition module is also used to periodically detect the business data in the hot data warehouse of the specified business to obtain a detection result; if the detection result indicates that the last update time of any specified business data does not fall within the valid time range of the hot data warehouse, the specified business data is deleted from the hot data warehouse.
[0126] In one embodiment, if the offline data warehouse is deployed in an independent manner, the cold data warehouse associated with the specified business corresponds to the first offline data warehouse, and the hot data warehouse associated with the specified business corresponds to the second offline data warehouse; if the offline data warehouse is deployed in a shared manner, the cold data warehouse and the hot data warehouse share the third offline data warehouse.
[0127] In application, each module in the data transmission device may be a software program module, or may be implemented by different logic circuits integrated in a processor, or may be implemented by multiple distributed processors.
[0128] Fig.11 A schematic diagram of the structure of a computer device for implementing a business data processing method provided in an embodiment of the application. Fig.11 As shown, the computer device 1000 of this embodiment includes: at least one processor 1010 ( Fig.11 Only one is shown in the figure) a processor, a memory 1020, and a computer program 1030 stored in the memory 1020 and executable on the at least one processor 1010, and when the processor 1010 executes the computer program 1030, the steps in any of the above-mentioned data transmission method embodiments are implemented.
[0129] The computer device 1000 may include, but is not limited to, a processor 1010 and a memory 1020. Those skilled in the art will appreciate that Fig.11 The computer device 1000 is merely an example and does not constitute a limitation on the computer device 1000 . The computer device 1000 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device 1000 may also include input and output devices, network access devices, etc.
[0130] The processor 1010 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0131] In some embodiments, the memory 1020 may be an internal storage unit of the computer device 1000, such as a hard disk or memory of the computer device 1000. In other embodiments, the memory 1020 may also be an external storage device of the computer device 1000, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 1000. Further, the memory 1020 may also include both an internal storage unit of the computer device 1000 and an external storage device. The memory 1020 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory 1020 may also be used to temporarily store data that has been output or is to be output.
[0132] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.
[0133] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0134] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.
[0135] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.
[0136] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.
[0137] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0138] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0139] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0140] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0141] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A business data processing method, characterized in that: include: Obtaining a service data request for a specified service, wherein the service data request carries a time range for required service data; Based on the time range of the required business data, cold and hot data identification is performed to obtain a target data warehouse storing the required business data, wherein the target data warehouse is a cold data warehouse or a hot data warehouse associated with the specified business; Based on the request type of the business data request, the required business data is selected from an offline data warehouse or a real-time data warehouse associated with the target data warehouse.
2. The business data processing method according to claim 1, characterized in that: The step of performing hot and cold identification based on the time range of the required business data to obtain a target data warehouse storing the required business data includes: Obtaining the effective time range of the hot data warehouse associated with the specified business; Comparing the effective time range with the time range of the required service data to obtain a comparison result; Based on the comparison result, a target data warehouse storing the required business data is determined.
3. The business data processing method according to claim 2, characterized in that: The determining, based on the comparison result, a target data warehouse storing the required business data comprises: If the comparison result indicates that the time range of the required business data belongs to the valid time range, the hot data warehouse associated with the specified business is determined as the target data warehouse storing the required business data; If the comparison result indicates that part of the time range of the required business data belongs to or does not belong to the valid time range, the cold data warehouse associated with the designated business is determined as the target data warehouse storing the required business data.
4. The business data processing method according to claim 1, characterized in that: The selecting the required business data from an offline data warehouse or a real-time data warehouse associated with the target data warehouse based on the request type of the business data request includes: If the request type of the business data request belongs to the first type, selecting the required business data from the real-time data warehouse associated with the target data warehouse, the first type indicating a query type request for business data; If the request type of the business data request belongs to the second type, the required business data is selected from the offline data warehouse associated with the target data warehouse, and the second type indicates a statistical request for business data.
5. The business data processing method according to claim 4, characterized in that: The request type of the service data request belongs to the second type, and the method further includes: Obtaining statistical results determined based on the required business data; If the time range of the required business data belongs to the valid time range of the hot data warehouse, the statistical results are written into the hot data warehouse and the cold data warehouse respectively; If part of the time range of the required business data belongs to or does not belong to the effective time range of the hot data warehouse, the statistical result is written into the cold data warehouse.
6. The business data processing method according to claim 1, characterized in that: The method further comprises: Acquire the business data to be stored associated with the specified business, and write the business data to be stored into the offline data warehouse of the cold data warehouse associated with the specified business; Creating a data copy of the business data to be stored, and migrating the data copy to a specified directory; If changes in the content of the designated directory are detected, the business data in the data copy is written into the real-time data warehouse of the cold data warehouse associated with the designated business.
7. The business data processing method according to claim 1, characterized in that: The method further comprises: Periodically testing the business data in the hot data warehouse of the specified business to obtain a test result; If the detection result indicates that the last update time of any designated business data does not fall within the valid time range of the hot data warehouse, the designated business data is deleted from the hot data warehouse.
8. The business data processing method according to any one of claims 1 to 7, characterized in that: If the offline data warehouse is deployed in an independent manner, the cold data warehouse associated with the specified business corresponds to the first offline data warehouse, and the hot data warehouse associated with the specified business corresponds to the second offline data warehouse; If the offline data warehouse is deployed in a shared manner, the cold data warehouse and the hot data warehouse share the third offline data warehouse.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.
