A data warehouse-based data operation method, device, medium and product

Abstract

The present specification discloses a data operation method and device based on a data warehouse, a medium and a product, characterized in that the data warehouse is used to store real-time data and offline data in the same data table; the method comprises: receiving a first data operation request for the data table, the first data operation request containing generation time information and operation description information, wherein the generation time information is used to represent the time of generating to-be-processed data, the operation description information is used to represent the processing operation that needs to be performed on the to-be-processed data, and the to-be-processed data contains real-time data and / or offline data; and in response to the first data operation request, performing the processing operation on the to-be-processed data corresponding to the generation time information in the data table.

Description

Technical Field

[0001] This specification relates to the field of data warehouse technology, and more particularly to a data operation method, device, medium and product based on a data warehouse. Background Technology

[0002] With the increasing data analytics needs of enterprises, traditional databases face performance bottlenecks when processing large-scale, diverse data, failing to efficiently support complex data analysis and mining tasks. To address these issues, data warehouses have been proposed as a solution specifically designed for storing and managing large amounts of historical data. However, current data warehouses employ different technical architectures for managing and maintaining real-time and offline data, requiring the simultaneous consideration of different data operation logics when performing related data operations. This results in complex operation processes and low execution efficiency. Summary of the Invention

[0003] In view of the above, one or more embodiments of this specification provide the following technical solutions:

[0004] According to a first aspect of one or more embodiments of this specification, a data manipulation method based on a data warehouse is proposed, wherein the data warehouse is used to store real-time data and offline data in the same data table; the method includes:

[0005] Receive a first data operation request for the data table. The first data operation request includes generation time information and operation description information, wherein the generation time information is used to characterize the time when the data to be processed was generated, and the operation description information is used to characterize the processing operation to be performed on the data to be processed. The data to be processed includes real-time data and / or offline data.

[0006] In response to the first data operation request, the processing operation is performed on the data to be processed corresponding to the generation time information in the data table.

[0007] Optionally, the processing operation is a data writing operation, and the first data operation request includes the data to be processed; performing the processing operation on the data to be processed corresponding to the generation time information in the data table includes:

[0008] Obtain the data to be processed from the first data operation request and write it into the data table;

[0009] The data type of the data to be processed is set according to the generation time information. The data type is used to indicate that the data to be processed belongs to real-time data and / or offline data in the data table.

[0010] Optionally, the processing operation is a data update operation, the data table maintains the data to be processed, and the first data operation request includes update information and data identification information for the data to be processed; performing the processing operation on the data to be processed corresponding to the generation time information in the data table includes:

[0011] If the generation time information does not belong to the update information, the generation time information is used as the data identification information, and the data update operation is performed on the data to be processed in the data table that matches the data identification information according to the update information;

[0012] If the generation time information belongs to the update information, the data update operation is performed on the data to be processed in the data table that matches the data identification information according to the update information, and the data type of the data to be processed is set according to the generation time information. The data type is used to indicate that the data to be processed belongs to real-time data and / or offline data in the data table.

[0013] Optionally, setting the data type of the data to be processed according to the generation time information includes:

[0014] The data type of the data to be processed corresponding to the generation time information is determined according to the data type judgment rules;

[0015] Set the value of the field corresponding to the data to be processed in the data table to a specific data type.

[0016] Optionally, the data type determination rule is used to indicate:

[0017] When the time represented by the corresponding generation time information is greater than the preset data type conversion time, the data to be processed is real-time data; when the time represented by the corresponding generation time information is less than the preset data type conversion time, the data to be processed is offline data.

[0018] Optionally, the data table maintains generation time information for each of the real-time data and the offline data; the method further includes:

[0019] Receive a second data operation request for the data table;

[0020] In response to the second data operation request, the real-time data in the data table that meets the data type conversion conditions is converted into the offline data according to the generation time information in the data table.

[0021] Optionally, the data type conversion condition is: the time represented by the corresponding generation time information is less than the preset data type conversion time.

[0022] According to a second aspect of one or more embodiments of this specification, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in the first aspect by executing the executable instructions.

[0023] According to a third aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0024] According to a fourth aspect of one or more embodiments of this specification, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the method as described in the first aspect.

[0025] In the technical solution provided in this specification, real-time data and offline data are stored in the same data table in the data warehouse, achieving a unified data structure. Simultaneously, when the data warehouse receives a first data operation request targeting the aforementioned data table and containing generation time information and operation description information, it can execute the processing operation represented by the operation description information on the data to be processed corresponding to the generation time information within that data table. Since the data to be processed may include real-time data and / or offline data, only the same set of operational logic needs to be followed when performing the above processing operations, resulting in a simple operation process and high execution efficiency. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the present invention, the accompanying drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0027] Figure 1 This is a schematic diagram of an architecture for a data operating system based on a data warehouse, provided as an exemplary embodiment.

[0028] Figure 2 This is a flowchart illustrating a data manipulation method based on a data warehouse, provided in an exemplary embodiment.

[0029] Figure 3 This is a flowchart illustrating another data manipulation method based on a data warehouse, provided in an exemplary embodiment.

[0030] Figure 4 This is a schematic structural diagram of an electronic device provided in an exemplary embodiment;

[0031] Figure 5 This is a schematic diagram of the structure of a data operation device based on a data warehouse, provided in an exemplary embodiment. Detailed Implementation

[0032] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present invention.

[0033] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” used in this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0034] It should be understood that although the terms first, second, third, etc., may be used in this invention to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of this invention, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0035] Figure 1 This is a schematic diagram of an architecture for a data operating system based on a data warehouse, provided as an exemplary embodiment. Figure 1 As shown, the system includes a data warehouse 11.

[0036] Data warehouse 11, as a centralized repository, can be used to integrate data from different data sources. Data warehouse 11 can maintain n data tables, each corresponding to different functions and uses according to actual needs, for example... Figure 1Table 1 can be used to store warehouse entry and exit records, Table 2 can be used to store warehouse employee personal identification information, etc., where n is a positive integer. Each table can store both real-time and offline data. Specifically, real-time data refers to data that needs to be processed, analyzed, and fed back immediately after its generation. This type of data is typically used in applications requiring immediate response, such as online transactions, real-time monitoring, and personalized recommendations. Offline data refers to data that does not require immediate processing and can be accumulated for a period of time before being processed in batches. This type of data is often used for historical data analysis, report generation, and trend analysis.

[0037] During system operation, when data warehouse 11 receives a first data operation request for the aforementioned data table, it can perform the processing operation represented by the operation description information on the data to be processed corresponding to the aforementioned generation time information in the aforementioned data table, based on the generation time information and operation description information contained in the first data operation request. The so-called data to be processed, for the aforementioned data table, can be real-time data, offline data, or both, and all of these can be processed through the first data operation request.

[0038] Those skilled in the art will understand that, in the architecture design of a data warehouse, real-time or offline data between the above different data tables can be organized through specific design patterns to further optimize query performance and data analysis efficiency, such as star schema, snowflake schema, etc. The comparison in this specification is not intended to be limiting.

[0039] Figure 2 This is a flowchart illustrating a data manipulation method based on a data warehouse, provided in an exemplary embodiment. The data warehouse is used to store real-time and offline data in the same data table. The method may include the following steps:

[0040] Step S202: Receive a first data operation request for the data table. The first data operation request includes generation time information and operation description information. The generation time information is used to characterize the time when the data to be processed was generated, and the operation description information is used to characterize the processing operation to be performed on the data to be processed. The data to be processed includes real-time data and / or offline data.

[0041] When a user needs to perform data-related operations on a data table in a data warehouse, they can initiate a first data operation request for the aforementioned data table through command-line tools, client software associated with the data warehouse, or business intelligence (BI) systems that utilize the application programming interface (API) provided by the data warehouse. This first data operation request may include generation time information and operation description information. Specifically, the generation time information can be represented by a timestamp or other preset time representations to indicate the moment the data to be processed was generated. The operation description information can be represented by different preset operation description identifiers such as numbers and strings to indicate the corresponding processing operation. These processing operations can be categorized into data write operations, data update operations, data deletion operations, and data query operations based on the specific execution process.

[0042] It should be noted that the real-time data and / or offline data included in the above-mentioned data to be processed are defined from the perspective of the data warehouse. For example, in the data tables of the above-mentioned data warehouse, all data whose generation time information corresponds to the current date T can be defined as real-time data, and the remaining data whose generation time information corresponds to the current date T-1...TN (i.e., the previous day to the previous N days, where N is a positive integer) can be defined as offline data. The purpose is to prevent interference caused by the difference in the definition of real-time data and offline data in different scenarios, and to avoid situations such as the data to be processed corresponding to the first data operation request being real-time data for the user, other data warehouses, or other data tables, but being offline data for the data table corresponding to the first data operation request.

[0043] Furthermore, if it is predetermined that the data to be processed is real-time or offline, the tool initiating the first data operation request can employ a targeted technical framework. For example, if the data to be processed is real-time, technologies such as Change Data Capture (CDC) or Debezium can be used to capture the first data operation request in real time, and combined with high-performance stream processing frameworks such as Flink and Spark to achieve near real-time data updates and analysis. Alternatively, if the data to be processed is offline, managed data warehouse solutions provided by cloud service providers, such as MaxCompute, Redshift, and BigQuery, can be used to perform complex batch data processing tasks at low cost and high efficiency.

[0044] Step S204: In response to the first data operation request, perform the processing operation on the data to be processed corresponding to the generation time information in the data table.

[0045] Once the data warehouse successfully receives the first data operation request, it can respond to it and execute the processing operation on the data to be processed, as represented by the operation description information in the first data operation request. As mentioned earlier, this processing operation can be divided into four basic operations based on the specific content of the operation description information: data write operation, data update operation, data deletion operation, and data query operation. In different processing operations, the generation time information of the first data operation request also has different effects.

[0046] Those skilled in the art will understand that, regardless of the processing operation, it only needs to be executed once on the aforementioned data table. Compared to the traditional approach of attempting to perform processing operations on the data tables of the data warehouses where real-time data and offline data reside, the method in this application of storing real-time data and offline data in the same data table achieves their isomorphism. That is, real-time data and offline data are physically integrated, while the operation logic for processing real-time data or offline data is unified, eliminating the underlying technical differences under different processing logics, simplifying the amount of operations required to perform processing operations, and improving the overall efficiency of the execution operation.

[0047] The following will be combined with this instruction manual Figure 3 The examples will introduce each basic operation in turn.

[0048] The purpose of the aforementioned data write operation is to write the data to be processed contained in the first data operation request into the corresponding data table of the data warehouse. The data warehouse can then retrieve the data to be processed from the first data operation request and write it into the data table. The data table being written to does not actually know whether the data to be processed belongs to real-time or offline data. Therefore, it can set the data type of the data to be processed based on the generation time information in the first data operation request. This data type can be used to indicate whether the data to be processed belongs to real-time and / or offline data in the data table. In other words, whether writing real-time or offline data, it can be achieved through the first data operation request, simplifying the overall data write process. Of course, since the data to be processed here is assumed to be a combination of multiple data entries, there is a possibility that the data to be processed belongs to both real-time and offline data. If the data to be processed is determined to be a single data entry, then it will belong to either real-time or offline data.

[0049] The following is based on Figure 3For example, the above data writing operation involves the following steps:

[0050] In step S302, the data warehouse receives a first data operation request for a data table and determines the processing operation represented by the operation description information therein.

[0051] In one embodiment, suppose a user wants to add an inbound record to a goods inbound / outbound record table in a data warehouse. The user can then initiate a first data operation request for this inbound record, which includes time information representing the record's creation time, such as a second-level timestamp "1705008000", corresponding to 00:00:00 on November 11, 2024. The first data operation request itself can also include an operation description representing a data write operation to be performed on the data to be processed, such as the value "creat" in the custom request field "data-type". Upon receiving the first data operation request, the data warehouse can extract and determine this operation description information.

[0052] Step S304: Determine whether the processing operation is a data write operation.

[0053] In one embodiment, the processing operation corresponding to the operation description information can be determined by the correspondence between the preset operation description information and the processing operation or other rules. If the above correspondence indicates that the operation description information in the previous embodiment corresponds to the data writing operation, then step S306 can be executed; otherwise, step S310 can be executed.

[0054] Step S306: Obtain the data to be processed from the first data operation request and write it into the data table.

[0055] In one embodiment, the inbound records included in the first data operation request can be written into a data table named Goods Inbound / Outbound Record Table to generate new data in the data table.

[0056] Step S308: Set the data type of the data to be processed according to the generation time information.

[0057] In one embodiment, the aforementioned goods in / out record table may record the data type of each data entry to indicate whether the corresponding data belongs to real-time data or offline data in the table. For the inbound record written in the previous step, the corresponding data type can be set according to the generation time information therein. The specific process of setting this data type will be described in detail below, and will not be elaborated on here.

[0058] The purpose of the aforementioned data update operation is to update the update information contained in the first data operation request into the pending data maintained in the data table. In addition to the update information, the first data operation request may also include data identification information for the pending data to identify it in the data table. This data identification information can be used to identify the pending data. In short, both real-time and offline data updates can be achieved through the aforementioned first data operation request, simplifying the overall data update process. In different situations, the aforementioned generation time information can be used either as part of the update information to update the original generation time information in the pending data, or as the aforementioned data identification information to identify the pending data in the data table. The following sections describe these two situations separately:

[0059] If the aforementioned generation time information does not belong to the aforementioned update information, it can be used as the aforementioned data identifier information, and the aforementioned data update operation can be performed on the pending data in the aforementioned data table that matches the aforementioned data identifier information according to the aforementioned update information. Here, the generation time information can be a specific timestamp; when the generation time information corresponding to any pending data in the aforementioned data table is consistent with this timestamp, the pending data can be considered to match the aforementioned generation time information. Alternatively, the generation time information can also be a time range consisting of two different timestamps; when the generation time information corresponding to any pending data in the aforementioned data table falls within this time range, the pending data can be considered to match the aforementioned generation time information. Of course, the above matching logic can be adjusted according to the actual scenario, and this specification does not impose any restrictions on it.

[0060] When the aforementioned generation time information belongs to the aforementioned update information, the aforementioned data update operation can be performed on the pending data in the aforementioned data table that matches the aforementioned data identifier information, and the data type of the pending data can be set according to the aforementioned generation time information. Since the original generation time information in the pending data will also be updated, this means that the original data type of the pending data may change, for example, from real-time data to offline data, or from offline data to real-time data. Therefore, after the aforementioned data update operation is completed, the data type of the pending data can be reset according to the updated generation time information. Furthermore, in this case, the data identifier information can be a unique in-table identifier for the pending data, and this specification does not impose any restrictions on this.

[0061] The following is based on Figure 3 For example, the above data update operation involves the following steps:

[0062] In step S302, the data warehouse receives a first data operation request for a data table and determines the processing operation represented by the operation description information therein.

[0063] In one embodiment, suppose a user wants to update the content of an existing inbound record in a goods inbound / outbound record table in a data warehouse. A first data operation request containing update information can be made for this record table. This update information might include time information representing the record's creation time, such as a second-level timestamp "1705008000". The first data operation request itself can also include an operation description indicating that a data update operation needs to be performed on the data to be processed, such as the value "update" in the custom request field "data-type". Upon receiving the first data operation request, the data warehouse can extract and determine this operation description information.

[0064] Step S304: Determine whether the processing operation is a data write operation.

[0065] Step S310: Determine whether the processing operation is a data update operation.

[0066] In one embodiment, the processing operation corresponding to the operation description information can be determined by the pre-defined correspondence between the operation description information and the processing operation or other rules. If the above correspondence indicates that the operation description information in the previous embodiment corresponds to the data update operation, then step S312 can be executed after step S304 and S310 are executed in sequence; otherwise, step S320 is executed.

[0067] Step S312: Determine whether the generation time information in the first data operation request belongs to update information.

[0068] In one embodiment, it can be determined whether the above-mentioned generation time information is included in the update information. If it is included, it is determined that the generation time information belongs to the update information and step S316 is executed; otherwise, step S314 is executed.

[0069] Step S314: Use the generation time information as data identification information, and perform data update operation on the data to be processed in the data table that matches the data identification information according to the update information.

[0070] In one embodiment, since the generation time information is not part of the update information, it does not participate in the actual data update operation. Instead, it is used to determine the data to be updated in the goods entry and exit record table. Taking the above-mentioned generation time information "1705008000" as an example, if there is data with the same generation time information in the goods entry and exit record table, this data can be used as the above-mentioned data to be updated, and the data in the above-mentioned update information can be updated into this data.

[0071] Step S316: Perform a data update operation on the data to be processed that matches the data identifier information in the data table according to the update information.

[0072] In one embodiment, since the generation time information is update information, it will participate in the actual data update operation. It is necessary to determine the data to be updated in the goods entry / exit record table based on additional data identification information, such as the unique ID corresponding to each record in the goods entry / exit record table. Assuming the data identification information is the number "12", the record with ID "12" in the goods entry / exit record table can be queried, and the retrieved data can be used as the data to be processed that matches the data identification information, so that the data in the update information, including the generation time information, can be updated in that data.

[0073] Step S318: Set the data type of the data to be processed according to the generation time information.

[0074] In one embodiment, after the original pending data in the goods entry and exit record table is updated with the generation time information, it is also necessary to set the data type of the pending data according to the generation time information. This step is basically the same as the execution process of step S308, and will not be described in detail in this specification.

[0075] Regarding the data type determination process in steps S308 and S318, the data type of the data to be processed corresponding to the aforementioned generation time information is first determined according to the data type judgment rules. Then, the field value of the type field corresponding to the aforementioned data to be processed in the data table is set to the determined data type. The type field conforms to the structure of the data table, facilitating the persistence of the last determined data type for each piece of data. The data type judgment rules can be used to indicate: if the time represented by the corresponding generation time information is greater than the preset data type conversion time, the data to be processed is real-time data; if the time represented by the corresponding generation time information is less than the preset data type conversion time, the data to be processed is offline data. The preset data type conversion time serves to distinguish between real-time data and offline data and can be expressed using a timestamp or other format identical to the generation time information for easy time comparison. The type field corresponding to the data to be processed in the aforementioned data table can be a pre-defined dedicated field in the data table. Its value is used to indicate whether the corresponding data is real-time data or offline data. It can be implemented using Boolean or numeric field types, for example, a value of "0" represents real-time data, a value of "1" represents real-time data, etc. This specification does not impose any restrictions on this.

[0076] The purpose of the aforementioned data generation operation is to delete the data to be processed specified in the first data operation request from the data table. This data to be processed is actually maintained in the data table, and the first data operation request contains data identification information for that data. This data identification information can be the aforementioned generation time information, used to perform a data deletion operation on the data to be processed in the data table that matches the data identification information.

[0077] The following is based on Figure 3 For example, the process of deleting data as described above involves the following steps:

[0078] In step S302, the data warehouse receives a first data operation request for a data table and determines the processing operation represented by the operation description information therein.

[0079] In one embodiment, suppose a user wants to delete an existing inbound record in the goods inbound / outbound record table of the data warehouse. A first data operation request can be made for the aforementioned goods inbound / outbound record table. This request might include update information representing the time the record was created, such as a second-level timestamp "1705008000". The first data operation request itself could also include an operation description representing the data deletion operation to be performed on the data to be processed, such as the value "delete" in the custom request field "data-type". Upon receiving the first data operation request, the data warehouse can extract and determine this operation description information.

[0080] Step S304: Determine whether the processing operation is a data write operation.

[0081] Step S310: Determine whether the processing operation is a data update operation.

[0082] Step S320: Determine whether the processing operation is a data deletion operation.

[0083] In one embodiment, the processing operation corresponding to the operation description information can be determined by the pre-defined correspondence between the operation description information and the processing operation or other rules. If the above correspondence indicates that the operation description information in the previous embodiment corresponds to a data deletion operation, rather than a data writing operation or a data update operation, then step S322 can be executed after executing steps S304, S310 and S320 in sequence; otherwise, step S324 can be executed.

[0084] Step S322: Perform a data deletion operation on the data to be processed that matches the data identification information in the data table.

[0085] In one embodiment, the generation time information can be used as the aforementioned data identification information to identify the data to be updated in the goods entry and exit record table. Taking the aforementioned generation time information "1705008000" as an example, if a query finds data with the same generation time information in the goods entry and exit record table, a deletion operation can be performed on it.

[0086] The purpose of the above data query operation is to provide the ability to retrieve specified data to be processed from the data table. The first data operation request contains data identification information for the data to be processed. At this time, the data identification information can be the above generation time information, which is used to perform a data query operation on the data to be processed that matches the data identification information in the data table, and then return the queried data to be processed as a data result set.

[0087] The following is based on Figure 3For example, the above data query operation involves the following steps:

[0088] In step S302, the data warehouse receives a first data operation request for a data table and determines the processing operation represented by the operation description information therein.

[0089] In one embodiment, suppose a user wants to query all inbound and outbound records for November 11, 2024, in the goods inbound / outbound record table of the data warehouse. A first data operation request can be made for the aforementioned goods inbound / outbound record table. This update information includes time information representing the generation time of the record itself, such as a combination of second-level timestamps "1705008000" and "1705094399," corresponding to the time period between 00:00:00 on November 11, 2024, and 23:59:59 on November 11, 2024. Simultaneously, the first data operation request itself can include operation description information representing the data query operation to be performed on the data to be processed, such as the value "query" in the custom request field "data-type." After receiving the first data operation request, the data warehouse can extract and determine this operation description information.

[0090] Step S304: Determine whether the processing operation is a data write operation.

[0091] Step S310: Determine whether the processing operation is a data update operation.

[0092] Step S320: Determine whether the processing operation is a data deletion operation.

[0093] Step S324: Determine whether the processing operation is a data query operation.

[0094] In one embodiment, the processing operation corresponding to the operation description information can be determined by the pre-defined correspondence between the operation description information and the processing operation or other rules. Assuming that the above correspondence indicates that the operation description information in the previous embodiment corresponds to a data query operation, rather than a data write operation, a data update operation, or a data deletion operation, then step S326 can be executed after executing steps S304, S310, S320, and S324 in sequence; otherwise, no actual response is given to the first data operation request.

[0095] Step S326: Perform a data query operation on the data to be processed that matches the data identification information in the data table.

[0096] In one embodiment, the generation time information can be used as the aforementioned data identification information to determine the data to be processed in the goods entry and exit record table. Taking the time period formed by the aforementioned generation time information "1705008000" and "1705094399" as an example, if the query finds that there is data in the goods entry and exit record table whose generation time information is in this time period, a data query operation can be performed on it.

[0097] Step S328: Return the queried data to be processed as a data result set.

[0098] In one embodiment, one or more pieces of data to be processed that have been queried can be returned to the initiator of the first data operation request in the form of a data result set.

[0099] It is worth mentioning that, Figure 3 The logical implementation order of steps S304, S310, S320 and S324 can be adjusted according to the actual situation, and the result does not have a substantial impact on the overall processing effect of the first data operation request.

[0100] For any data table in a data warehouse, even without executing any processing operations triggered by the aforementioned first data operation request, its stored real-time and offline data do not remain static. This is because real-time data, as data that needs to be processed, analyzed, and fed back immediately after its generation, will lose its original intended effect over time and transform into offline data used for complex analysis, mining, and modeling. Therefore, this specification can also provide other requests, different from the first data operation request, to convert real-time data and offline data in the data table.

[0101] In one embodiment, the aforementioned data table can maintain generation time information for each piece of real-time data and offline data. Upon receiving a second data operation request for the aforementioned data table, the data warehouse can respond to the request by converting the real-time data in the data table that meets the data type conversion conditions into the offline data, based on the generation time information in the data table. The data type conversion condition can be set to a condition where the time represented by the corresponding generation time information is less than a preset data type conversion time. The conversion method between the real-time data and the offline data can be found in the process of setting the data type through the data table's type field described above, and will not be repeated here.

[0102] In addition, the aforementioned second data operation request can be initiated not only by the user but also periodically and automatically by devices such as timers to ensure that real-time data in the data table is converted into offline data in a timely manner. This avoids misjudgment of real-time or offline data in data analysis scenarios, which could lead to a decrease in the accuracy of the analysis results.

[0103] Figure 4 This is a schematic structural diagram of a device provided in an exemplary embodiment. Please refer to... Figure 4 At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, memory 408, and non-volatile memory 410, and may also include other hardware required for its functions. One or more embodiments of this specification can be implemented in software, for example, the processor 402 reads the corresponding computer program from the non-volatile memory 410 into memory 408 and then runs it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0104] Corresponding to the aforementioned embodiments of the data manipulation method based on a data warehouse, this specification also provides an embodiment of a data manipulation apparatus based on a data warehouse. Please refer to... Figure 5 The data warehouse is used to store real-time data and offline data in the same data table; the apparatus includes:

[0105] The first request receiving unit 502 is configured to receive a first data operation request for the data table. The first data operation request includes generation time information and operation description information, wherein the generation time information is used to characterize the moment when the data to be processed is generated, and the operation description information is used to characterize the processing operation to be performed on the data to be processed. The data to be processed includes real-time data and / or offline data.

[0106] The operation execution unit 504 is used to perform the processing operation on the data to be processed corresponding to the generation time information in the data table in response to the first data operation request.

[0107] Optionally, the processing operation is a data writing operation, and the first data operation request includes the data to be processed; the operation execution unit 504 is specifically used for:

[0108] Obtain the data to be processed from the first data operation request and write it into the data table;

[0109] The data type of the data to be processed is set according to the generation time information. The data type is used to indicate that the data to be processed belongs to real-time data and / or offline data in the data table.

[0110] Optionally, the processing operation is a data update operation, the data table maintains the data to be processed, and the first data operation request includes update information and data identification information for the data to be processed; the operation execution unit 504 is specifically used for:

[0111] If the generation time information does not belong to the update information, the generation time information is used as the data identification information, and the data update operation is performed on the data to be processed in the data table that matches the data identification information according to the update information;

[0112] If the generation time information belongs to the update information, the data update operation is performed on the data to be processed in the data table that matches the data identification information according to the update information, and the data type of the data to be processed is set according to the generation time information. The data type is used to indicate that the data to be processed belongs to real-time data and / or offline data in the data table.

[0113] Optionally, the operation execution unit 504 is specifically used for:

[0114] The data type of the data to be processed corresponding to the generation time information is determined according to the data type judgment rules;

[0115] Set the value of the field corresponding to the data to be processed in the data table to a specific data type.

[0116] Optionally, the data type determination rule is used to indicate:

[0117] When the time represented by the corresponding generation time information is greater than the preset data type conversion time, the data to be processed is real-time data; when the time represented by the corresponding generation time information is less than the preset data type conversion time, the data to be processed is offline data.

[0118] Optionally, the data table maintains generation time information for each of the real-time data and the offline data; the device further includes:

[0119] The second request receiving unit is configured to receive a second data operation request for the data table.

[0120] A data type conversion unit is used to respond to the second data operation request and convert real-time data that meets the data type conversion conditions in the data table into offline data according to the generation time information in the data table.

[0121] Optionally, the data type conversion condition is: the time represented by the corresponding generation time information is less than the preset data type conversion time.

[0122] Based on the same concept as the methods described above, this specification also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in any of the above embodiments by executing the executable instructions.

[0123] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0124] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0125] While this invention contains numerous specific details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of particular inventions. Certain features described in the multiple embodiments of this invention may also be implemented in combination in a single embodiment. On the other hand, various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation of a sub-combination.

[0126] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0127] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0128] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method of data manipulation based on a data warehouse, characterized by, The data warehouse is used to store real-time and offline data in the same data table; the method includes: A first data operation request is received for the data table. The first data operation request includes generation time information and operation description information. The generation time information is used to characterize the time when the data to be processed was generated. The operation description information is used to characterize the processing operation to be performed on the data to be processed. The data to be processed includes real-time data and / or offline data. The processing operation is a data update operation. The data table maintains the data to be processed. The first data operation request includes update information and data identification information for the data to be processed. In response to the first data operation request, the processing operation is performed on the data to be processed corresponding to the generation time information in the data table, including: If the generation time information does not belong to the update information, the generation time information is used as the data identification information, and the data update operation is performed on the data to be processed in the data table that matches the data identification information according to the update information; If the generation time information belongs to the update information, the data update operation is performed on the data to be processed in the data table that matches the data identification information according to the update information, and the data type of the data to be processed is set according to the generation time information. The data type is used to indicate that the data to be processed belongs to real-time data and / or offline data in the data table.

2. The method according to claim 1, characterized in that, Setting the data type of the data to be processed according to the generation time information includes: The data type of the data to be processed corresponding to the generation time information is determined according to the data type judgment rules; Set the value of the field corresponding to the data to be processed in the data table to a specific data type.

3. The method according to claim 2, characterized in that, The data type determination rule is used to indicate: When the time represented by the corresponding generation time information is greater than the preset data type conversion time, the data to be processed is real-time data; when the time represented by the corresponding generation time information is less than the preset data type conversion time, the data to be processed is offline data.

4. The method according to claim 1, characterized in that, The data table maintains generation time information for each piece of real-time data and each piece of offline data; the method further includes: Receive a second data operation request for the data table; In response to the second data operation request, the real-time data in the data table that meets the data type conversion conditions is converted into the offline data according to the generation time information in the data table.

5. The method according to claim 4, characterized in that, The data type conversion condition is: the time represented by the corresponding generation time information is less than the preset data type conversion time.

6. An electronic device, characterized in that, include: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method as described in any one of claims 1-5 by executing the executable instructions.

7. A computer-readable storage medium, characterized in that, It stores computer instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1-5.

8. A computer program product, characterized in that, Includes a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1-5.

Images