Data processing method and device, equipment and medium

Abstract

The invention is suitable for the technical field of databases, and relates to a data processing method and device, equipment and a medium. The method comprises the steps of obtaining target data of a to-be-written data fragment; the tail row group of the data fragment is determined as a target row group in which the target data needs to be written, the tail row group adopts row-column mixed storage, and the non-tail row group of the data fragment adopts column type storage; and writing the target data into the target row group by row units according to the column sequence. When the data processing scheme provided by the invention is applied to the OLAP system, the real-time performance of data writing can be improved on the premise of keeping the high query performance of the OLAP system.

Description

TECHNICAL FIELD

[0001] The present application is suitable for the field of database technology, and particularly relates to a data processing method, a data processing apparatus, a computer device and a computer readable storage medium. BACKGROUND

[0002] A modern database management system (DBMS) is a large software system for efficiently managing data access and facilitating queries. In order to better handle different task requests, it is divided into two types of products, namely, On-Line Transaction Processing (OLTP) and On-Line Analytical Processing (OLAP). In OLTP, a large number of concurrent short transactions are used for data recording and modification, while OLAP is used for data analysis scenarios such as production reports and instant queries, and obtains results by scanning a large amount of data and running complex operations. In the related art, in order to exert the query optimization capabilities such as encoding compression and sparse index, the OLAP system usually relies on the batch writing mode of accumulated data, which results in poor real-time performance of data writing of the OLAP system. SUMMARY

[0003] Embodiments of the present application provide a data processing method, a data processing apparatus, a computer device and a computer readable storage medium, which can be applied to an OLAP system, and can improve the real-time performance of data writing while maintaining high query performance of the OLAP system.

[0004] In a first aspect, embodiments of the present application provide a data processing method, comprising: obtaining target data of a data shard to be written; determining a target row group to which the target data needs to be written as a last row group of the data shard, the last row group being stored in a row-column mixed manner, and a non-last row group of the data shard being stored in a columnar manner; writing the target data into the target row group in a column order in units of rows.

[0005] In a second aspect, embodiments of the present application provide a data processing apparatus, comprising: a data import module configured to obtain target data of a data shard to be written; a row group determination module configured to determine a target row group to which the target data needs to be written as a last row group of the data shard, the last row group being stored in a row-column mixed manner, and a non-last row group of the data shard being stored in a columnar manner; a data writing module configured to write the target data into the target row group in a column order in units of rows.

[0006] Optionally, in an embodiment, the data writing module is configured to write the target data into the target row group in a column order in a row unit if a data amount of the target data is less than a remaining writable data amount of the target row group.

[0007] Optionally, in an embodiment, the data writing module is configured to write part of the target data into the target row group in a row unit if a data amount of the target data is greater than a remaining writable data amount of the target row group; initialize a new end row group using the row-column mixed storage for the data slice, and write the remaining target data into the new end row group in a row unit.

[0008] Optionally, in an embodiment, the data writing module is further configured to convert a storage mode of the target row group into column storage when the target row group is full.

[0009] Optionally, in an embodiment, the data writing module is configured to identify whether the data slice is full; if the data slice is not full, initialize a new end row group using the row-column mixed storage at an end of the data slice, and write the remaining target data into the new end row group in a row unit.

[0010] Optionally, in an embodiment, the data writing module is configured to initialize a new data slice if the data slice is full; if a data amount of the remaining data is equal to a writable data amount of a new row group in the new data slice, initialize a new row group using column storage for the new data slice, and write the remaining data into the new row group in the new data slice in a column unit.

[0011] In a third aspect, a computer device is provided, which includes a processor and a memory. The memory stores a computer program capable of being run on the processor. The processor runs the computer program to implement the data processing method provided in the embodiments of the present application.

[0012] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program. The computer program is run by a processor to implement the data processing method provided in the embodiments of the present application.

[0013] The data processing scheme provided in the application obtains target data to be written in a data slice; a last row group of the data slice is determined as a target row group to which the target data needs to be written, the last row group adopts row-column mixed storage, and a non-last row group of the data slice adopts columnar storage. When the data processing method provided in the application is applied to an OLAP system, all row groups in the data slice except the last row group are configured to adopt columnar storage, so that the query optimization capabilities such as encoding compression and sparse index can be maximized, the last row group in the data slice is configured to adopt row-column mixed storage, so that any batch data can be directly appended without batch writing, and therefore the real-time performance of data writing is improved on the premise of maintaining high query performance of the OLAP system. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0015] Figure 1 is an application environment schematic diagram of the data processing method provided in the embodiments of the present application; Figure 2 is a flow schematic diagram of the data processing method provided in the embodiments of the present application; Figure 3 is an example diagram of storage modes of different row groups of a data slice in the embodiments of the present application; Figure 4 is a first example diagram of writing target data into a target row group in the embodiments of the present application; Figure 5 is a second example diagram of writing target data into a target row group in the embodiments of the present application; Figure 6 is a third example diagram of writing target data into a target row group in the embodiments of the present application; Figure 7 is a structure schematic diagram of the data processing apparatus provided in the embodiments of the present application; Figure 8 is a structure schematic diagram of the computer device provided in the embodiments of the present application. DETAILED DESCRIPTION

[0016] In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular architectures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known methods, devices, circuits, and

[0017] It is to be understood that the terminology "includes", "has", "holds", "contains" and / or "comprising", "including", "having" and their conjugates, as used herein, means "including but not limited to", and not to the exclusion of any other term or aspect.

[0018] It is also to be understood that the terminology "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items, and that the term "at least one of A and B" is equivalent to "A, B, or A and B".

[0019] As used in the description of the application and the appended claims, the term "if' can be interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "if it is determined" or "if [a described condition or event] is detected" can be interpreted to mean "upon determining" or "in response to determining" or "upon [the described condition or event] being detected" or "in response to [the described condition or event] being detected", depending on the context.

[0020] In addition, the terms "first", "second", "third", etc. as used in the description of the application and the appended claims are not used to denote or imply relative importance but are used to distinguish one element from another.

[0021] Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in other embodiments" or "in still other embodiments" in various places throughout this specification are not necessarily all referring to the same embodiment, unless otherwise specified. Furthermore, the terms "comprise", "comprises", "comprising", "include", "includes", "including" and the like are synonymous with the term "contain", "contains", "containing" or "has", "has", "having" and are used in the sense of "including but not limited to", unless otherwise specified.

[0022] It should be understood that the size of the serial number of each step in the following embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0023] In order to illustrate the technical solutions of the present application, the following will be described by specific embodiments.

[0024] Please refer to Figure 1 , Figure 1 is an application environment diagram of the data processing method provided by the present application. As an embodiment, the data processing method provided by the present application can be applied to a server 100, the server 100 and the terminal device 200 are connected through a network, and the server 100 is used to provide database service for the terminal device 200. Among them, the network is used to provide a communication link medium between the server 100 and the terminal device 200, which can include various connection types, such as wired communication link, wireless communication link, etc., and the embodiments of the present application do not limit this.

[0025] It should be pointed out that Figure 1 The server 100, network and terminal device 200 in are only schematic. According to the actual needs, there can be any number of servers 100. For example, the server 100 can be realized by an independent physical server, or can be realized by a server cluster composed of multiple servers or a distributed system, etc., and the terminal device 200 can be any device such as a mobile phone, a tablet, a desktop computer, a notebook computer, etc.

[0026] In some embodiments, the server 100 can obtain target data to be written into a data shard in response to a user operation from the terminal device 200, determine a target row group of the target data as a last row group of the data shard, wherein the last row group of the data shard adopts row-column mixed storage, and the non-last row group of the data shard adopts row-column mixed storage, and finally write the target data into the target row group in units of rows. When the data processing method provided by the present application is applied to an OLAP system, by configuring all row groups in the data shard except the last row group as columnar storage, the query optimization capabilities such as encoding compression and sparse index can be maximized. By configuring the last row group in the data shard as row-column mixed storage, any batch data can be directly appended without batch writing, thereby improving the real-time performance of data writing while maintaining the high query performance of the OLAP system.

[0027] Please refer to Figure 2 , Figure 2 is a flow diagram of a data processing method provided by an embodiment of the present application, as shown in Figure 2 The flow of the data processing method can be as follows: In S110, target data of a data shard to be written is acquired.

[0028] The data shard is a highest-level logical division unit in a storage system in which large-scale data is divided according to a certain rule or strategy. For example, the data can be divided by date, by business type, or by user ID, etc., to ensure high correlation of data in the data shard. For example, when a business system divides data shards by date, transaction data generated on the same day is stored in the same data shard, facilitating efficient query and analysis.

[0029] The row group is a next-level storage unit of the data shard. The row group has no fixed physical structure. In columnar storage, data in the row group is organized by column, facilitating compression and indexing. In row-column hybrid storage, part of the data in the row group is organized by row, facilitating fast appending. In the embodiment of the application, the row group storage mode is flexibly configured, ensuring query efficiency and improving real-time data writing.

[0030] The column block is a columnar storage unit in the row group. Each column block contains data of the same column, facilitating efficient compression and indexing. For example, the ID column block in the row group contains all ID data, and the Name column block contains all Name data. Each column block is independently compressed and encoded using a column-level compression algorithm such as dictionary encoding.

[0031] The business system includes a business system using a micro-service architecture, such as an insurance system, a bank system, an e-commerce system, etc. in the field of financial technology, a diagnosis and treatment system, a hospital information system, a medical image information system, etc. in the field of medical health.

[0032] The following describes an example in which the data processing method provided by the application is executed by a server.

[0033] In the embodiment of the application, the server can acquire the to-be-written data imported by the business system and take the to-be-written data as target data of the data shard to be written. For example, when the business system imports transaction data, the server takes the transaction data as target data of the data shard to be written, when the business system imports diagnosis and treatment data, the server takes the diagnosis and treatment data as target data of the data shard to be written, when the business system imports user behavior data, the server takes the user behavior data as target data of the data shard to be written, and so on.

[0034] In S120, the last row group of the data shard is determined as a target row group in which the target data needs to be written. The last row group uses row-column hybrid storage, and the non-last row group of the data shard uses columnar storage.

[0035] It should be noted that please refer to Figure 3In the embodiment of the present application, the data slice adopts a hybrid file format, in which the end row groups of the data slice are stored in a row-column hybrid manner, and the non-end row groups of the data slice are stored in a columnar manner. Through this hybrid storage strategy, the advantages of high compression rate and fast query of columnar storage are fully utilized, and the flexibility of row storage in data appending is also taken into account.

[0036] In the embodiment of the present application, whenever data is imported, the server first performs an appending write operation to the end row groups of the data slice until the end row groups are full, at which time the server automatically creates new end row groups and continues the appending write.

[0037] Correspondingly, after obtaining the target data of the data slice to be written, the server determines the end row groups of the data slice as the row groups to which the target data needs to be written, denoted as target row groups.

[0038] In S130, the target data is written to the target row groups in units of rows.

[0039] In the embodiment of the present application, after determining the target row groups to which the target data needs to be written, the server writes the target data to the target row groups in units of rows in column order without encoding compression. In this way, sparse indexes (such as Min-Max indexes) can be created for the data in the end row groups and the non-end row groups to speed up range query filtering.

[0040] For example, assuming that a complete piece of data contains all data of three columns of user ID, user age, and transaction amount, please refer to Figure 4 If the obtained target data contains two complete pieces of data, one complete piece of data A: user ID is 123, user age is 25, and transaction amount is 5000; and another complete piece of data B: user ID is 124, user age is 30, and transaction amount is 6000, the server first writes data A to the target row groups in column order, writes “123” to the user ID column, then writes “30” to the user age column, and finally writes “5000” to the transaction amount column, and then writes data B to the target row groups in column order, writes “124” to the user ID column, writes “30” to the user age column, and writes “6000” to the transaction amount column, to complete the writing of the target data.

[0041] Optionally, in an embodiment, writing the target data to the target row groups in units of rows in column order includes: If the data amount of the target data is less than the remaining writable data amount of the target row groups, the target data is written to the target row groups in units of rows in column order.

[0042] In the embodiment of the present application, when the server writes the target data into the target row group in column order in units of rows, first, the server obtains the remaining writable data amount of the data shard and the data amount of the target data, and if it is identified that the data amount of the target data is less than the remaining writable data amount of the target row group, the server writes all the target data into the target row group in column order in units of rows.

[0043] For example, referring to Figure 5 , it is assumed that the remaining writable data amount of the data shard is 2048 rows, and the data amount of the target data is 1024 rows, and it is obvious that the data amount of the target data is less than the remaining writable data amount of the target row group at this time, and the server writes all the 1024 rows of target data into the target row group in column order row by row.

[0044] Optionally, in an embodiment, writing the target data into the target row group in column order in units of rows further includes: If the data amount of the target data is greater than the remaining writable data amount of the target row group, writing part of the data in the target data into the target row group in units of rows; Initializing a new end row group using row-column hybrid storage for the data shard, and writing the remaining data in the target data into the new end row group in units of rows.

[0045] In the embodiment of the present application, when the server writes the target data into the target row group in column order in units of rows, first, the server obtains the remaining writable data amount of the data shard and the data amount of the target data, and if it is identified that the data amount of the target data is greater than the remaining writable data amount of the target row group, the server writes part of the data in the target data into the target row group in column order in units of rows until the target row group is full. In addition, the server initializes a new row group using row-column hybrid storage for the data shard, which is the new end row group of the data shard. Subsequently, the server writes the remaining data in the target data except for the part of the data that has been written into the target row group into the new end row group in column order in units of rows, ensuring data integrity.

[0046] In the embodiment of the present application, the data processing method further includes: When the target row group is full, converting the storage mode of the target row group to columnar storage.

[0047] When the target row group is full, the server automatically triggers a storage conversion mechanism to reorganize the data in the target row group by column, and converts the reorganized data into a columnar storage format through encoding compression (such as dictionary encoding).

[0048] For example, referring to Figure 6, assuming that the remaining writable data amount of the data slice is 2048 rows, and the data amount of the target data is 4096 rows, it is obvious that the data amount of the target data is greater than the remaining writable data amount of the target row group, at this time, the server writes 2048 rows of data in the target data in the order of columns in units of rows into the target row group until the target row group is full, and then converts the storage mode of the target row group from the row-column mixed storage to the columnar storage, in addition, the server also initializes a new end row group using the row-column mixed storage for the data slice, and then writes the remaining 2048 rows of data in the target data in the order of columns in units of rows into the new end row group.

[0049] It should be noted that in other embodiments, if the data amount of the target data is equal to the remaining writable data amount of the target row group, the target data is directly written in the order of columns in units of rows into the target row group, and the storage mode of the target row group is converted to columnar storage. When the next new target data is obtained, the server can create a new end row group with a corresponding storage mode for the data slice according to the data amount of the new target data, wherein if the data amount of the new target data is less than the writable data amount of the row group of the data slice, a new end row group using row-column mixed storage is initialized for the data slice, and the new target data is written in the order of columns in units of rows into the new end row group; if the data amount of the new target data is equal to the writable data amount of the row group of the data slice, a new end row group using columnar storage is initialized for the data slice, and the new target data is written into the new end row group in units of columns; if the data amount of the new target data is greater than the writable data amount of the row group of the data slice, a new row group using columnar storage is initialized, and part of the new target data is written into the row group until it is full, and then a new row group using row-column mixed storage is initialized, and the remaining data in the new target data is written into the row group. This process is repeated to ensure efficient and orderly data writing, improving the overall performance and data processing speed of the storage system.

[0050] Optionally, in an embodiment, before initializing a new end row group using row-column mixed storage at the end of the data slice and writing the remaining data in the target data into the new row group in units of rows, it further comprises: identifying whether the data slice is full; if the data slice is not full, initializing a new end row group using row-column mixed storage at the end of the data slice, and writing the remaining data in the target data into the new end row group in units of rows.

[0051] In the embodiment of the application, a data amount threshold corresponding to the data slice is configured to control the size of the data slice, and the value of the data amount threshold is not limited here and can be configured by a person skilled in the art according to actual needs.

[0052] Correspondingly, before initializing a new end row group adopting the mixed storage of row and column at the end of the data slice and writing the remaining data in the target data into the new row group in units of row, the server acquires the current data amount of the data slice and compares the current data amount of the data slice with the preset data amount threshold, if the current data amount of the data slice has reached the data amount threshold, it is determined that the data slice has been written full, if the current data amount of the data slice has not reached the data amount threshold, it is determined that the data slice has not been written full.

[0053] Wherein, when it is identified that the data slice has not been written full, the server initializes a new end row group adopting the mixed storage of row and column at the end of the data slice and writes the remaining data in the target data into the new end row group in units of row.

[0054] Optionally, in an embodiment, after identifying whether the data slice has been written full, further comprising: if the data slice has been written full, initializing a new data slice; if the data amount of the remaining data is equal to the row group writable data amount in the new data slice, initializing a new row group adopting the column storage for the new data slice and writing the remaining data into the new row group in the new data slice in units of column.

[0055] In the embodiment of the application, the row group writable data amounts in the same data slice are configured to be the same, and the specific value of the row group writable data amount is not limited here and can be flexibly configured according to actual needs. For example, for a data slice, the row group writable data amount of the data slice can be configured to be 4096 rows.

[0056] For the case that the data slice has been written full, the server will automatically initialize a new data slice, if the data amount of the remaining data is equal to the row group writable data amount in the new data slice, the server further initializes a new row group adopting the column storage for the new data slice and writes the remaining data into the new row group in units of column.

[0057] It should be noted that in other embodiments, if the data amount of the remaining data is less than the row group writable data amount in the new data slice, the server further initializes a new row group adopting the mixed storage of row and column for the new data slice and writes the remaining data into the new row group in units of row according to the column order.

[0058] Exemplarily, the data processing method provided by the application is described below by taking a plurality of import events received by the server as an example: A data slice with a row group writable data amount of 4096 rows is initialized.

[0059] Data import event 1: 4096 rows of data is received; Since the data volume of the data import event 1 reaches the writable data volume of the row group, the row group 0 using column storage is initialized for the data shard, and the 4096 rows of data of the data import event 1 are written into the row group 0 after column encoding compression; the transaction is committed, and the data import event 1 ends; The data import event 2 is received: 2048 rows of data; Since the row group 0 is full, and the data volume of the data import event 2 is less than the writable data volume of the row group, the row group 1 using row-column hybrid storage is initialized for the data shard, and the 2048 rows of data of the data import event 2 are written into the row group 1 in column order in units of rows; the transaction is committed, and the data import event 2 ends; The data import event 3 is received: 4096 rows of data; Since the row group 1 is not full, the first 2048 rows of data in the 4096 rows of data of the data import event 3 are written into the row group 1 in column order in units of rows, at this time, the row group 1 is full, the row group 1 is converted into column storage, and the row group 2 using row-column hybrid storage is initialized for the data shard, and the remaining 2048 rows of data are written into the row group 2 in column order in units of rows; the transaction is committed, and the data import event 3 ends.

[0060] As can be seen from the above, the data processing scheme provided in the application obtains target data to be written into a data shard; determines a tail row group of the data shard as a target row group to which the target data needs to be written, and the tail row group uses row-column hybrid storage, and non-tail row groups of the data shard use column storage. When the data processing method provided in the application is applied to an OLAP system, all row groups in the data shard except the tail row group are configured to use column storage, which can maximize the query optimization capabilities such as encoding compression and sparse indexing, and the tail row group in the data shard is configured to use row-column hybrid storage, so that any batch data can be directly appended without batch writing, thereby improving the real-time performance of data writing while maintaining high query performance of the OLAP system.

[0061] In order to better implement the above data processing method, an embodiment of the application further provides a corresponding data processing device. The meanings of the terms are the same as those in the above data processing method, and specific implementation details are referred to the description in the method embodiment.

[0062] Please refer to Figure 7 , Figure 7 The structure schematic diagram of the data processing device provided in the embodiment of the application can include a data import module 210, a row group determination module 220, and a data writing module 230, wherein, The data import module 210 is configured to obtain target data to be written into a data shard; The row group determination module 220 is used to determine the last row group of the data fragment as the target row group to be written to the target data. The last row group is stored in a mixed row and column format, and the non-last row groups of the data fragment are stored in a column format. The data writing module 230 is used to write the target data into the target row group in column order and row unit.

[0063] Optionally, in one embodiment, the data writing module 230 is used to write the target data into the target row group in column order and row unit if the amount of target data is less than the remaining writable data amount of the target row group.

[0064] Optionally, in one embodiment, the data writing module 230 is used to: if the amount of data in the target data is greater than the amount of remaining writable data in the target row group, write a portion of the data in the target data into the target row group in rows; initialize a new last row group using row-column hybrid storage for data fragmentation, and write the remaining data in the target data into the new last row group in rows.

[0065] Optionally, in one embodiment, the data writing module 230 is further configured to convert the storage mode of the target row group to columnar storage when the target row group is full.

[0066] Optionally, in one embodiment, the data writing module 230 is used to identify whether the data fragment is full; if the data fragment is not full, a new end row group with row-column mixed storage is initialized at the end of the data fragment, and the remaining data in the target data is written to the new end row group in rows.

[0067] Optionally, in one embodiment, the data writing module 230 is configured to: initialize a new data fragment if the data fragment is full; and initialize a new row group for the new data fragment using columnar storage if the amount of remaining data is equal to the amount of writable data in the new row group in the new data fragment, and write the remaining data into the new row group in the new data fragment in column units.

[0068] It should be noted that the information interaction and execution process between the above modules are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0069] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 8 As shown, the computer device includes: at least one processor ( Figure 8 Only one is shown in the diagram), a memory, and a computer program stored in the memory and executable on at least one processor, wherein the processor executes the computer program to implement the steps in the above-described data processing method embodiments.

[0070] The computer device can include, but not limited to, a processor, a memory. Those skilled in the art can understand that, Figure 8 The computer device is only an example and does not constitute a limitation on the computer device. The computer device can include more or less components than the illustration, or combine certain components, or different components, for example, it can also include a network interface, a display screen, an input device, etc.

[0071] The processor can be a CPU. The processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0072] The memory includes a readable storage medium, an internal memory, etc. The internal memory can be the memory of the computer device, and the internal memory provides an environment for the operation of the operating system and the computer readable instructions in the readable storage medium. The readable storage medium can be the hard disk of the computer device, and in other embodiments, it can also be the external storage device of the computer device, for example, the plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Further, the memory can include both the internal storage unit of the computer device and the external storage device. The memory is used to store the operating system, application programs, boot loader, data, and other programs, such as program codes of computer programs, etc. The memory can also be used to temporarily store data that has been output or will be output.

[0073] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the above device can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here. If the integrated unit is realized in the form of software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, all or part of the processes in the above embodiment methods can be completed by a computer program instructing related hardware. The computer program can be stored in a computer readable storage medium. When the processor executes the computer program, the steps of the above method embodiments can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium at least includes any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium. For example, U disk, mobile hard disk, magnetic disk or optical disk, etc. In some jurisdictions, according to legislation and patent practice, computer readable medium cannot be electrical carrier signal and telecommunication signal.

[0074] The above embodiment methods can also be completed by a computer program product, which can be run on a computer device to make the computer device execute the steps of the above method embodiments.

[0075] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0076] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0077] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / computer device and method can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely schematic. The division of the modules or units is merely a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0078] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0079] The above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A data processing method, characterized by, The method comprises: obtaining target data to be written into a data slice; determining a last row group of the data slice as a target row group to which the target data needs to be written, the last row group being stored in a row-column mixed manner, and a non-last row group of the data slice being stored in a columnar manner; writing the target data into the target row group in a column order in units of rows.

2. The data processing method according to claim 1, characterized in that, The writing of the target data into the target row group in a column order in units of rows comprises: if an amount of data of the target data is less than a remaining writable amount of the target row group, writing the target data into the target row group in a column order in units of rows.

3. The data processing method of claim 1, wherein, The writing of the target data into the target row group in a column order in units of rows further comprises: if the amount of data of the target data is greater than the remaining writable amount of the target row group, writing part of the target data into the target row group in units of rows; and initializing a new last row group of the data slice in a row-column mixed manner, and writing the remaining data of the target data into the new last row group in units of rows.

4. The data processing method according to claim 3, characterized in that, The method further comprises: when the target row group is full, converting a storage manner of the target row group into a columnar manner.

5. The data processing method according to claim 3, characterized in that, Before the initializing of the new last row group of the data slice in a row-column mixed manner and the writing of the remaining data of the target data into the new row group in units of rows, the method further comprises: identifying whether the data slice is full; if the data slice is not full, initializing a new last row group of the data slice in a row-column mixed manner, and writing the remaining data of the target data into the new last row group in units of rows.

6. The data processing method according to claim 5, characterized in that, After the identifying of whether the data slice is full, the method further comprises: if the data slice is full, initializing a new data slice; if an amount of the remaining data is equal to a writable amount of a new row group in the new data slice, initializing a new row group of the new data slice in a columnar manner, and writing the remaining data into the new row group in the new data slice in units of columns.

7. A data processing apparatus, characterized by The method comprises: a data importing module configured to obtain target data to be written into a data slice; a row group determining module configured to determine a last row group of the data slice as a target row group to which the target data needs to be written, the last row group being stored in a row-column mixed manner, and a non-last row group of the data slice being stored in a columnar manner; a data writing module configured to write the target data into the target row group in a column order in units of rows.

8. The data processing apparatus according to claim 7, characterized in that, The data writing module is configured to, if an amount of data of the target data is less than a remaining writable amount of the target row group, write the target data into the target row group in a column order in units of rows.

9. A computer device, comprising: The computer device comprises a processor and a memory, and the memory stores a computer program which can be run on the processor, and the processor runs the computer program to implement the data processing method in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program which is executed by a processor to implement the data processing method in any one of claims 1 to 6.

Images