Data operation method and device based on data warehouse, medium and product

Abstract

The invention discloses a data operation method and device based on a data warehouse, a medium and a product, and is characterized in that the data warehouse is used for storing real-time data and offline data in the same data table; the method comprises the steps that a first data operation request for the data table is received, the first data operation request comprises generation time information and operation description information, and the generation time information is used for representing the moment of generating data to be processed; the operation description information is used for representing a processing operation needing to be executed for the to-be-processed data, and the to-be-processed data comprises real-time data and / or offline data; and in response to the first data operation request, executing the processing operation on the to-be-processed data corresponding to the generation time information in the data table.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of data warehouse technology, and more particularly to a data operation method, device, medium, and product based on a data warehouse. Background Art

[0002] With the growing demand for data analysis in enterprises, traditional databases face performance bottlenecks when processing large-scale and diversified data, and are unable to efficiently support complex data analysis and mining tasks. To solve the above problems, data warehouses have been proposed as a solution specifically for storing and managing large amounts of historical data. However, current data warehouses use different technical architectures to manage and maintain real-time data and offline data, which means that when performing related data operations, the operation logic of different data needs to be considered at the same time, which in turn leads to complex operation processes and low execution efficiency. Summary of the invention

[0003] In view of this, one or more embodiments of this specification provide the following technical solutions:

[0004] According to a first aspect of one or more embodiments of this specification, a data operation method based on a data warehouse is proposed, wherein the data warehouse is used to store real-time data and offline data in the same data table; the method comprises:

[0005] receiving a first data operation request for the data table, wherein the first data operation request includes generation time information and operation description information, wherein the generation time information is used to characterize the time when the data to be processed is generated, and the operation description information is used to characterize the processing operation to be performed on the data to be processed, wherein the data to be processed includes real-time data and / or offline data;

[0006] In response to the first data operation request, the processing operation is performed on the to-be-processed data corresponding to the generation time information in the data table.

[0007] Optionally, the processing operation is a data writing operation, and the first data operation request includes the data to be processed; and performing the processing operation on the data to be processed corresponding to the generation time information in the data table includes:

[0008] Obtaining the data to be processed in the first data operation request and writing the data into the data table;

[0009] The data type of the data to be processed is set according to the generation time information, and the data type is used to indicate that the data to be processed is real-time data and / or offline data in the data table.

[0010] Optionally, the processing operation is a data update operation, the data table maintains the data to be processed, and the first data operation request includes update information and data identification information for the data to be processed; performing the processing operation on the data to be processed corresponding to the generation time information in the data table includes:

[0011] In the case where the generation time information does not belong to the update information, the generation time information is used as the data identification information, and the data update operation is performed on the to-be-processed data in the data table that matches the data identification information according to the update information;

[0012] In the case where the generation time information belongs to the update information, the data update operation is performed on the data to be processed that matches the data identification information in the data table according to the update information, and the data type of the data to be processed is set according to the generation time information, and the data type is used to characterize that the data to be processed belongs to real-time data and / or offline data in the data table.

[0013] Optionally, setting the data type of the to-be-processed data according to the generation time information includes:

[0014] Determine the data type of the to-be-processed data corresponding to the generation time information according to a data type judgment rule;

[0015] The field value of the type field corresponding to the data to be processed in the data table is set to the determined data type.

[0016] Optionally, the data type determination rule is used to indicate:

[0017] When the moment represented by the corresponding generation time information is greater than the preset data type conversion moment, the data to be processed is real-time data; when the moment represented by the corresponding generation time information is less than the preset data type conversion moment, the data to be processed is offline data.

[0018] Optionally, the data table maintains generation time information of each of the real-time data and the offline data; the method further includes:

[0019] receiving a second data operation request for the data table;

[0020] In response to the second data operation request, the real-time data in the data table that meets the data type conversion condition is converted into the offline data according to the generation time information in the data table.

[0021] Optionally, the data type conversion condition is: the moment represented by the corresponding generation time information is less than a preset data type conversion moment.

[0022] According to a second aspect of one or more embodiments of this specification, an electronic device is proposed, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements the steps of the method described in the first aspect by running the executable instructions.

[0023] According to a third aspect of one or more embodiments of the present specification, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0024] According to a fourth aspect of one or more embodiments of the present specification, a computer program product is provided, comprising a computer program / instruction, which implements the steps of the method described in the first aspect when executed by a processor.

[0025] In the technical solution provided in this specification, real-time data and offline data are stored in the same data table of the data warehouse, so that the two realize the unification of data structure; at the same time, when the data warehouse receives the first data operation request for the above data table and contains the generation time information and the operation description information, the processing operation represented by the above operation description information can be performed on the data to be processed corresponding to the above generation time information in the data table. Among them, since the data to be processed can include real-time data and / or offline data, it is only necessary to follow the same set of operation logic when performing the above processing operation, that is, the operation process is simple and the execution efficiency is high. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments or prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 It is a schematic diagram of the architecture of a data operation system based on a data warehouse provided by an exemplary embodiment;

[0028] Figure 2 It is a flowchart of a data operation method based on a data warehouse provided by an exemplary embodiment;

[0029] Figure 3 It is a flowchart diagram of another data operation method based on a data warehouse provided by an exemplary embodiment;

[0030] Figure 4 is a schematic structural diagram of an electronic device provided by an exemplary embodiment;

[0031] Figure 5 It is a structural diagram of a data operation device based on a data warehouse provided by an exemplary embodiment. DETAILED DESCRIPTION

[0032] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are only examples of devices and methods consistent with some aspects of the present invention.

[0033] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0034] It should be understood that although the terms first, second, third, etc. may be used in the present invention to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0035] Figure 1 FIG. 1 is a schematic diagram of an architecture of a data operation system based on a data warehouse provided by an exemplary embodiment. Figure 1 As shown, the system includes a data warehouse 11.

[0036] As a centralized storage repository, the data warehouse 11 can be used to integrate data from different data sources. The data warehouse 11 can maintain n data tables, each of which can correspond to different functions and uses according to actual needs, such as Figure 1Data table 1 can be used to store the in and out records of goods, data table 2 can be used to store the personal identity information of warehouse employees, etc., and n is a positive integer. For each data table, real-time data and offline data can be stored in it at the same time. Specifically, the real-time data refers to data that needs to be processed, analyzed and fed back immediately after it is generated. This type of data is usually used in application scenarios that require immediate response, such as online transactions, real-time monitoring, personalized recommendations, etc. The offline data refers to data that does not need to be processed immediately and can be accumulated for a period of time before batch processing. This type of data is often used for historical data analysis, report generation, trend analysis, etc.

[0037] During the operation of the system, when the data warehouse 11 receives the first data operation request for the above data table, it can perform the processing operation represented by the above operation description information on the to-be-processed data corresponding to the above generation time information in the above data table according to the generation time information and operation description information contained in the first data operation request. Wherein, the so-called to-be-processed data for the above data table can be real-time data, offline data or both, and the corresponding processing operation can be performed on the above to-be-processed data through the first data operation request.

[0038] Those skilled in the art will appreciate that, in the architectural design of a data warehouse, the real-time data or offline data between the above different data tables can be organized through specific design patterns to further optimize query performance and data analysis efficiency, such as star schema, snowflake schema, etc., and the comparison in this specification is not limited thereto.

[0039] Figure 2 The present invention is a flowchart of a data operation method based on a data warehouse provided by an exemplary embodiment. The data warehouse is used to store real-time data and offline data in the same data table. The method may include the following steps:

[0040] Step S202: receiving a first data operation request for the data table, wherein the first data operation request includes generation time information and operation description information, wherein the generation time information is used to characterize the moment when the data to be processed is generated, and the operation description information is used to characterize the processing operation that needs to be performed on the data to be processed, and the data to be processed includes real-time data and / or offline data.

[0041] When a user needs to perform data-related operations on a data table in a data warehouse, a first data operation request for the data table can be initiated through a command line tool, client software associated with the data warehouse, or a business intelligence (BI) system that uses an application programming interface (API) provided by the data warehouse. The first data operation request may include generation time information and operation description information. Specifically, for the generation time information, the time when the data to be processed is generated may be represented by a timestamp or other preset time expressions; for the operation description information, the corresponding processing operation may be represented by different preset operation description identifiers such as numbers and strings. The processing operation may be divided into data write operation, data update operation, data delete operation, and data query operation according to the execution process of the specific operation.

[0042] It should be noted that the real-time data and / or offline data contained in the above-mentioned data to be processed are defined based on the perspective of the above-mentioned data warehouse. For example, in the data table in the above-mentioned data warehouse, all data whose generation time information corresponds to the current date T can be defined as real-time data, and the remaining data whose generation time information corresponds to the current date T-1...TN (that is, the previous day to the previous N days, N is a positive integer) can be defined as offline data. The purpose is to prevent interference caused by differences in the definitions of real-time data and offline data in different scenarios, and to avoid similar situations such as the data to be processed corresponding to the above-mentioned first data operation request being real-time data for users, other data warehouses or other data tables, but being offline data for the data table corresponding to the above-mentioned first data operation request.

[0043] In addition, when it is predetermined that the data to be processed is real-time data or offline data, the tool that initiates the above-mentioned first data operation request can adopt a targeted technical framework. For example, when the data to be processed is real-time data, technologies such as Change Data Capture (CDC) or Debezium can be used to capture the first data operation request in real time, and cooperate with technical frameworks such as Flink and Spark that support high-performance stream processing to achieve near-real-time data update and analysis. Or when the data to be processed is offline data, managed data warehouse solutions provided by cloud service providers such as MaxCompute, Redshift and BigQuery can be used to perform complex batch data processing tasks at low cost and high efficiency.

[0044] Step S204: In response to the first data operation request, the processing operation is performed on the to-be-processed data corresponding to the generation time information in the data table.

[0045] When the data warehouse successfully receives the first data operation request, it can respond to it to execute the processing operation on the data to be processed represented by the operation description information in the first data operation request. As mentioned above, the processing operation can be divided into four basic operations, namely, the data writing operation, data updating operation, data deleting operation and data query operation, according to the specific content of the operation description information. In different processing operations, the generation time information of the first data operation request also has different effects.

[0046] Those skilled in the art can understand that no matter what kind of processing operation is performed, it only needs to be executed once on the above-mentioned data table. Compared with the traditional method of trying to execute processing operations on the data tables of the data warehouse where the real-time data and the offline data are located respectively, the present application realizes the isomorphism of the real-time data and the offline data by storing them in the same data table, that is, the real-time data and the offline data are integrated with each other at the physical level, and at the same time, the operation logic of the processing operations for the real-time data or the offline data is unified, eliminating the underlying technical differences under different processing logics, simplifying the amount of operations required to execute the processing operations, and improving the overall execution efficiency of the operation.

[0047] The following in this manual will be combined Figure 3 The embodiments of the present invention introduce each basic operation in turn.

[0048] For the above-mentioned data writing operation, its purpose is to write the to-be-processed data contained in the first data operation request as new data into the data table corresponding to the data warehouse. Then the above-mentioned data warehouse can obtain the to-be-processed data in the above-mentioned first data operation request and write it into the above-mentioned data table. For the data table into which the data is written, it is actually unknown whether the to-be-processed data belongs to real-time data or offline data in this data table. Therefore, the data type of the to-be-processed data can be set according to the generation time information in the above-mentioned first data operation request. The data type can be used to characterize that the above-mentioned to-be-processed data belongs to real-time data and / or offline data in the above-mentioned data table. In other words, whether writing real-time data or offline data, it can be achieved through the above-mentioned first data operation request, which simplifies the overall process of data writing. Of course, since the to-be-processed data here defaults to a combination of multiple data, there is a situation where the to-be-processed data belongs to real-time data and offline data. If it is determined that the to-be-processed data is a single data, then the to-be-processed data will belong to one of real-time data or offline data.

[0049] Below Figure 3Taking the above data writing operation as an example, the following steps are introduced:

[0050] Step S302: the data warehouse receives a first data operation request for a data table, and determines a processing operation represented by the operation description information therein.

[0051] In one embodiment, assuming that a user wants to add an entry record to the goods entry and exit record table in the data warehouse, a first data operation request containing the entry record can be initiated for the above goods entry and exit record table, and the entry record contains time information used to characterize the generation time of the record itself, such as the second-level timestamp "1705008000", which corresponds to 0:0:0:0 on November 11, 2024. At the same time, the first data operation request itself can contain operation description information used to characterize the need to perform a data write operation on the data to be processed, such as the value "creat" in the custom request field "data-type". Then, after receiving the above first data operation request, the data warehouse can extract and determine the operation description information.

[0052] Step S304, determining whether the processing operation is a data writing operation.

[0053] In one embodiment, the processing operation corresponding to the operation description information can be determined through the correspondence between the preset operation description information and the processing operation or other rules. Assuming that the above correspondence indicates that the operation description information in the previous embodiment corresponds to the data writing operation, then step S306 can be executed, otherwise step S310 is executed.

[0054] Step S306, obtaining the data to be processed in the first data operation request, and writing it into the data table.

[0055] In one embodiment, the warehouse entry record included in the first data operation request may be written into a data table named goods warehouse entry and exit record table to generate new data in the data table.

[0056] Step S308: setting the data type of the to-be-processed data according to the generation time information.

[0057] In one embodiment, the above-mentioned goods in and out of the warehouse record table may record the data type of each piece of data to indicate whether the corresponding data in the table is real-time data or offline data. For the warehouse entry record written in the previous step, the corresponding data type may be set according to the generation time information therein, wherein the specific setting process of the data type will be specifically introduced below, and this specification will not be described in detail here.

[0058] For the above-mentioned data update operation, its purpose is to update the update information contained in the first data operation request to the pending data maintained in the data table. In addition to the update information, the above-mentioned first data operation request may also include data identification information for the above-mentioned data to be processed so as to determine the pending data in the data table, wherein the data identification information can be used to identify the above-mentioned data to be processed. In short, whether it is updating real-time data or offline data, it can be achieved through the above-mentioned first data operation request, which simplifies the overall process of data update. In different cases, the above-mentioned generation time information can be used as part of the above-mentioned update information to update the original generation time information in the above-mentioned data to be processed, and can also be used as the above-mentioned data identification information to determine the above-mentioned data to be processed in the data table. The above two situations are introduced respectively below:

[0059] In the case where the above-mentioned generation time information does not belong to the above-mentioned update information, the generation time information can be used as the above-mentioned data identification information, and the above-mentioned data update operation can be performed on the to-be-processed data in the above-mentioned data table that matches the above-mentioned data identification information according to the above-mentioned update information. Among them, the generation time information can be a specific timestamp. When the generation time information corresponding to any to-be-processed data in the above-mentioned data table is consistent with the timestamp, it can be considered that the to-be-processed data matches the above-mentioned generation time information; or, the generation time information can also be a time range composed of two different timestamps. When the generation time information corresponding to any to-be-processed data in the above-mentioned data table is within the time range, it can be considered that the to-be-processed data matches the above-mentioned generation time information. Of course, the above matching logic can be adjusted according to the actual scenario, and this specification does not limit this.

[0060] In the case where the above-mentioned generation time information belongs to the above-mentioned update information, the above-mentioned data update operation can be performed on the to-be-processed data in the above-mentioned data table that matches the above-mentioned data identification information according to the above-mentioned update information, and the data type of the above-mentioned to-be-processed data can be set according to the above-mentioned generation time information. Among them, since the original generation time information in the above-mentioned data to be processed will also be updated, this means that the original data type of the to-be-processed data may change, for example, from real-time data to offline data, or from offline data to real-time data. Therefore, after the above-mentioned data update operation is completed, the data type of the above-mentioned data to be processed can also be reset according to the updated generation time information. In addition, the data identification information in this case can be a unique in-table identifier of the data to be processed, which is not limited in this specification.

[0061] Below Figure 3 Taking the above data update operation as an example, the following steps are introduced:

[0062] Step S302: the data warehouse receives a first data operation request for a data table, and determines a processing operation represented by the operation description information therein.

[0063] In one embodiment, assuming that a user wants to update the content of an existing entry record in the goods entry and exit record table in the data warehouse, a first data operation request containing update information can be made for the above-mentioned goods entry and exit record table. Assume that the update information contains time information used to characterize the generation time of the record itself, such as the second-level timestamp "1705008000". At the same time, the first data operation request itself can contain operation description information used to characterize the need to perform a data update operation on the data to be processed, such as the value "update" in the custom request field "data-type". Then, after receiving the above-mentioned first data operation request, the data warehouse can extract and determine the operation description information.

[0064] Step S304, determining whether the processing operation is a data writing operation.

[0065] Step S310, determining whether the processing operation is a data update operation.

[0066] In one embodiment, the processing operation corresponding to the operation description information can be determined through the correspondence between the preset operation description information and the processing operation or other rules. Assuming that the above correspondence indicates that the operation description information in the previous embodiment corresponds to the data update operation, then step S312 can be executed after executing steps S304 and S310 in sequence, otherwise step S320 is executed.

[0067] Step S312: determine whether the generation time information in the first data operation request belongs to update information.

[0068] In one embodiment, it may be determined whether the generation time information is included in the update information. If so, it is determined that the generation time information is part of the update information and step S316 is executed; otherwise, step S314 is executed.

[0069] Step S314: Use the generation time information as data identification information, and perform a data update operation on the to-be-processed data in the data table that matches the data identification information according to the update information.

[0070] In one embodiment, since the generation time information does not belong to the update information, it does not participate in the actual data update operation, but is used to determine the pending data to be updated in the goods entry and exit record table. Taking the above-mentioned generation time information "1705008000" as an example, if there is data with the same generation time information in the goods entry and exit record table, the data can be used as the above-mentioned pending data, and the data in the above-mentioned update information can be updated to the data.

[0071] Step S316: performing a data update operation on the to-be-processed data in the data table that matches the data identification information according to the update information.

[0072] In one embodiment, since the generation time information belongs to the update information, it will participate in the actual data update operation, and the pending data to be updated needs to be determined in the goods in and out of the warehouse record table according to the additional data identification information, such as the unique ID corresponding to each record in the goods in and out of the warehouse record table. Assuming that the above data identification information is the number "12", the record with the ID "12" in the goods in and out of the warehouse record table can be queried, and the queried data is used as the above pending data matching the data identification information, so as to update the data in the update information including the above generation time information into the data.

[0073] Step S318, setting the data type of the data to be processed according to the generation time information.

[0074] In one embodiment, after the generation time information of the original data to be processed in the goods in and out record table is updated, the data type of the data to be processed needs to be set according to the generation time information. This step is basically consistent with the execution process of step S308 and will not be elaborated in this manual.

[0075] For the determination process of the data type in step S308 and step S318, the data type of the to-be-processed data corresponding to the above-mentioned generation time information can first be determined according to the data type judgment rule, and then the field value of the type field corresponding to the above-mentioned to-be-processed data in the data table is set to the determined data type. Among them, the type field conforms to the structure of the data table, which is convenient for realizing the persistence of the data type determined for the last time for each data. The data type judgment rule can be used to indicate: when the moment represented by the corresponding generation time information is greater than the preset data type conversion moment, the to-be-processed data is real-time data; when the moment represented by the corresponding generation time information is less than the preset data type conversion moment, the to-be-processed data is offline data, and the above preset data type conversion moment plays a role in distinguishing real-time data from offline data, and can be expressed in the same format as the generation time information, such as a timestamp, so as to facilitate the time comparison between the two. As for the type field corresponding to the to-be-processed data in the above-mentioned data table, it can be a special field preset in the data table, and its value is used to represent whether the corresponding data belongs to real-time data or offline data, which can be implemented by field types such as Boolean values ​​or numbers, for example, the value "0" represents real-time data, the value "1" represents real-time data, etc., which is not limited in this specification.

[0076] The purpose of the data generation operation is to delete the to-be-processed data specified by the first data operation request from the data table. The to-be-processed data is actually maintained in the data table, and the first data operation request includes data identification information for the to-be-processed data. In this case, the data identification information may be the generation time information, so as to perform a data deletion operation on the to-be-processed data in the data table that matches the data identification information.

[0077] Below Figure 3 Take the above data deletion operation as an example to introduce the following steps:

[0078] Step S302: the data warehouse receives a first data operation request for a data table, and determines a processing operation represented by the operation description information therein.

[0079] In one embodiment, assuming that a user wants to delete an existing entry record in the goods entry and exit record table in the data warehouse, a first data operation request can be made for the above goods entry and exit record table. Assume that the update information contains time information for representing the generation time of the record itself, such as the second-level timestamp "1705008000". At the same time, the first data operation request itself can contain operation description information for representing the need to perform a data deletion operation on the data to be processed, such as the value "delete" in the custom request field "data-type". Then, after receiving the above first data operation request, the data warehouse can extract and determine the operation description information.

[0080] Step S304, determining whether the processing operation is a data writing operation.

[0081] Step S310, determining whether the processing operation is a data update operation.

[0082] Step S320, determining whether the processing operation is a data deletion operation.

[0083] In one embodiment, the processing operation corresponding to the operation description information can be determined through the correspondence between the preset operation description information and the processing operation or other rules. Assuming that the above correspondence indicates that the operation description information in the previous embodiment corresponds to a data deletion operation rather than a data writing operation or a data update operation, then step S322 can be executed after executing steps S304, S310 and S320 in sequence, otherwise continue to execute step S324.

[0084] Step S322: performing a data deletion operation on the to-be-processed data in the data table that matches the data identification information.

[0085] In one embodiment, the generation time information can be used as the above-mentioned data identification information to determine the pending data to be updated in the goods in and out record table. Taking the above-mentioned generation time information "1705008000" as an example, if the query finds that the goods in and out record table has data with the same generation time information, the deletion operation can be performed on it.

[0086] The purpose of the above-mentioned data query operation is to provide the ability to obtain specified data to be processed from the data table. The above-mentioned first data operation request includes data identification information for the data to be processed. At this time, the above-mentioned data identification information can be the above-mentioned generation time information, which is used to perform a data query operation on the data to be processed that matches the data identification information in the data table, and then return the queried data to be processed as a data result set.

[0087] Below Figure 3Taking the above data query operation as an example, the following steps are introduced:

[0088] Step S302: the data warehouse receives a first data operation request for a data table, and determines a processing operation represented by the operation description information therein.

[0089] In one embodiment, assuming that a user wishes to query all the inbound and outbound records corresponding to November 11, 2024 in the goods inbound and outbound record table in the data warehouse, a first data operation request may be made for the goods inbound and outbound record table, and the update information may include time information for representing the time when the record itself was generated, such as a combination of second-level timestamps "1705008000" and "1705094399", which corresponds to the time period formed between 0:00:00 on November 11, 2024 and 23:59:59 on November 11, 2024. At the same time, the first data operation request itself may include operation description information for representing the need to perform a data query operation on the data to be processed, such as the value "query" in the custom request field "data-type". After receiving the first data operation request, the data warehouse may extract and determine the operation description information.

[0090] Step S304, determining whether the processing operation is a data writing operation.

[0091] Step S310, determining whether the processing operation is a data update operation.

[0092] Step S320, determining whether the processing operation is a data deletion operation.

[0093] Step S324, determining whether the processing operation is a data query operation.

[0094] In one embodiment, the processing operation corresponding to the operation description information can be determined through the correspondence between the preset operation description information and the processing operation or other rules. Assuming that the above correspondence indicates that the operation description information in the previous embodiment corresponds to a data query operation rather than a data write operation, a data update operation or a data deletion operation, then step S326 can be executed after executing steps S304, S310, S320 and S324 in sequence, otherwise no actual response is made to the first data operation request.

[0095] Step S326: perform a data query operation on the data to be processed that matches the data identification information in the data table.

[0096] In one embodiment, the generation time information can be used as the above-mentioned data identification information to determine the data to be processed in the goods entry and exit record table. Taking the time period formed by the above-mentioned generation time information "1705008000" and "1705094399" as an example, when the query finds that there is data in the goods entry and exit record table with generation time information in this time period, a data query operation can be performed on it.

[0097] Step S328, returning the queried data to be processed as a data result set.

[0098] In one embodiment, the one or more to-be-processed data that are queried may be uniformly returned to the initiator of the first data operation request in the form of a data result set.

[0099] It is worth mentioning that Figure 3 The logical implementation order of step S304, step S310, step S320 and step S324 can be adjusted according to actual conditions, and the result does not substantially affect the overall processing effect of the first data operation request.

[0100] For any data table of the data warehouse, even if no processing operation caused by the above-mentioned first data operation request is performed, the real-time data and offline data stored therein do not remain static, because real-time data is a kind of data that needs to be processed, analyzed and fed back immediately after being generated. As time passes, it will lose the original expected effect of real-time data and turn into offline data for complex analysis, mining and modeling. Therefore, this specification can also provide for converting the real-time data and the offline data in the data table for other requests different from the first data operation request.

[0101] In one embodiment, the data table may maintain generation time information of each real-time data and offline data. After receiving the second data operation request for the data table, the data warehouse may respond to the request and convert the real-time data in the data table that meets the data type conversion condition into the offline data according to the generation time information in the data table. The data type conversion condition may be set to be less than the preset data type conversion time when the time represented by the corresponding generation time information is less than the preset data type conversion time. As for the conversion method between the real-time data and the offline data, please refer to the setting process of the data type through the type field of the data table in the above text, which will not be described in detail in this specification.

[0102] In addition, in addition to being actively initiated by the user, the above-mentioned second data operation request can also be periodically and automatically initiated through devices such as timers to ensure that the real-time data in the data table is converted into offline data in a timely manner, avoiding misjudgment of real-time data or offline data in scenarios such as data analysis, which in turn leads to a decrease in the accuracy of the analysis results.

[0103] Figure 4 is a schematic structural diagram of a device provided by an exemplary embodiment. Figure 4 At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, a memory 408, and a non-volatile memory 410, and may also include hardware required for other functions. One or more embodiments of this specification may be implemented based on software, such as the processor 402 reading the corresponding computer program from the non-volatile memory 410 into the memory 408 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0104] Corresponding to the above-mentioned data warehouse-based data operation method embodiment, this specification also provides an embodiment of a data warehouse-based data operation device. Figure 5 , the data warehouse is used to store real-time data and offline data in the same data table; the device includes:

[0105] A first request receiving unit 502 is configured to receive a first data operation request for the data table, wherein the first data operation request includes generation time information and operation description information, wherein the generation time information is used to indicate the time when the data to be processed is generated, and the operation description information is used to indicate the processing operation to be performed on the data to be processed, and the data to be processed includes real-time data and / or offline data;

[0106] The operation execution unit 504 is used to execute the processing operation on the to-be-processed data corresponding to the generation time information in the data table in response to the first data operation request.

[0107] Optionally, the processing operation is a data writing operation, and the first data operation request includes the data to be processed; the operation execution unit 504 is specifically used to:

[0108] Obtaining the data to be processed in the first data operation request and writing the data into the data table;

[0109] The data type of the data to be processed is set according to the generation time information, and the data type is used to indicate that the data to be processed is real-time data and / or offline data in the data table.

[0110] Optionally, the processing operation is a data update operation, the data table maintains the data to be processed, and the first data operation request includes update information and data identification information for the data to be processed; the operation execution unit 504 is specifically used to:

[0111] In the case where the generation time information does not belong to the update information, the generation time information is used as the data identification information, and the data update operation is performed on the to-be-processed data in the data table that matches the data identification information according to the update information;

[0112] In the case where the generation time information belongs to the update information, the data update operation is performed on the data to be processed that matches the data identification information in the data table according to the update information, and the data type of the data to be processed is set according to the generation time information, and the data type is used to characterize that the data to be processed belongs to real-time data and / or offline data in the data table.

[0113] Optionally, the operation execution unit 504 is specifically used to:

[0114] Determine the data type of the to-be-processed data corresponding to the generation time information according to a data type judgment rule;

[0115] The field value of the type field corresponding to the data to be processed in the data table is set to the determined data type.

[0116] Optionally, the data type determination rule is used to indicate:

[0117] When the moment represented by the corresponding generation time information is greater than the preset data type conversion moment, the data to be processed is real-time data; when the moment represented by the corresponding generation time information is less than the preset data type conversion moment, the data to be processed is offline data.

[0118] Optionally, the data table maintains generation time information of each of the real-time data and the offline data; the device further includes:

[0119] A second request receiving unit, configured to receive a second data operation request for the data table;

[0120] A data type conversion unit is used to respond to the second data operation request and convert the real-time data in the data table that meets the data type conversion condition into the offline data according to the generation time information in the data table.

[0121] Optionally, the data type conversion condition is: the moment represented by the corresponding generation time information is less than a preset data type conversion moment.

[0122] Based on the same concept as the above method, this specification also provides an electronic device, including: a processor; a memory for storing processor executable instructions; wherein the processor implements the steps of the method described in any of the above embodiments by running the executable instructions.

[0123] Based on the same concept as the above method, this specification also provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0124] Based on the same concept as the above method, this specification also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any of the above embodiments when executed by a processor.

[0125] Although the present invention includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of the specific embodiments of specific inventions. Certain features described in multiple embodiments of the present invention may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claimed as such, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of a sub-combination.

[0126] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0127] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0128] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data operation method based on a data warehouse, characterized in that: The data warehouse is used to store real-time data and offline data in the same data table; the method comprises: receiving a first data operation request for the data table, wherein the first data operation request includes generation time information and operation description information, wherein the generation time information is used to characterize the time when the data to be processed is generated, and the operation description information is used to characterize the processing operation to be performed on the data to be processed, wherein the data to be processed includes real-time data and / or offline data; In response to the first data operation request, the processing operation is performed on the to-be-processed data corresponding to the generation time information in the data table.

2. The method according to claim 1, characterized in that The processing operation is a data writing operation, and the first data operation request includes the data to be processed; The performing the processing operation on the to-be-processed data corresponding to the generation time information in the data table includes: Obtaining the data to be processed in the first data operation request and writing the data into the data table; The data type of the data to be processed is set according to the generation time information, and the data type is used to indicate that the data to be processed is real-time data and / or offline data in the data table.

3. The method according to claim 1, characterized in that The processing operation is a data update operation, the data table maintains the data to be processed, and the first data operation request includes update information and data identification information for the data to be processed; The performing the processing operation on the to-be-processed data corresponding to the generation time information in the data table includes: In the case where the generation time information does not belong to the update information, the generation time information is used as the data identification information, and the data update operation is performed on the to-be-processed data in the data table that matches the data identification information according to the update information; In the case where the generation time information belongs to the update information, the data update operation is performed on the data to be processed that matches the data identification information in the data table according to the update information, and the data type of the data to be processed is set according to the generation time information, and the data type is used to characterize that the data to be processed belongs to real-time data and / or offline data in the data table.

4. The method according to claim 2 or 3, characterized in that: The step of setting the data type of the to-be-processed data according to the generation time information includes: Determine the data type of the to-be-processed data corresponding to the generation time information according to a data type judgment rule; The field value of the type field corresponding to the data to be processed in the data table is set to the determined data type.

5. The method according to claim 4, characterized in that The data type judgment rule is used to indicate: When the moment represented by the corresponding generation time information is greater than the preset data type conversion moment, the data to be processed is real-time data; when the moment represented by the corresponding generation time information is less than the preset data type conversion moment, the data to be processed is offline data.

6. The method according to claim 1, characterized in that The data table maintains generation time information of each of the real-time data and the offline data; the method further includes: receiving a second data operation request for the data table; In response to the second data operation request, the real-time data in the data table that meets the data type conversion condition is converted into the offline data according to the generation time information in the data table.

7. The method according to claim 6, characterized in that The data type conversion condition is: the moment represented by the corresponding generation time information is less than the preset data type conversion moment.

8. An electronic device, characterized in that: include: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method according to any one of claims 1 to 7 by executing the executable instructions.

9. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images