Method and device for reading and writing data table
By classifying and identifying data tables and storing data blocks, the problems of low data query efficiency and large storage space in columnar storage are solved, and efficient query and resource conservation are achieved.
Patent Information
- Application Number
- CN202310582131.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-22
- Publication Date
- 2024-11-22
AI Technical Summary
When the prior art improves the efficiency of columnar storage data query, it is difficult to reduce the storage space occupied by the data warehouse, especially when querying data in specific fields, all data needs to be loaded and filtered, resulting in waste of resources.
By obtaining multiple classification identifiers, the field values of the data table to be stored are combined for classification identification marking, and the same category field values are stored in the same data block. Only when querying is found and sent corresponding data according to the classification identifier and field name block, avoid loading all data and pre-computed storage.
Improve data query efficiency, reduce the storage space occupied by the data warehouse, avoid the process of loading all data and pre-calculated results, and improve storage and query speed.
Smart Images

Figure CN119003538A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data processing, and in particular to a method and device for reading and writing a data table. Background Art
[0002] The storage methods of data warehouses can generally be divided into row storage and column storage. Column storage is to store data tables by field. For example, you need to store a student information table. A piece of data in the student information table is the information of a student, including age, height, weight, grade, etc. In this table, age is one field and height is another field. When writing data to the data warehouse, the age data of all students will be written first, and then the height data of all students will be written until the data of all fields in the student information table are written. This storage method is convenient for data analysis. For example, when you need to query the height of all students, you can directly load the height data stored continuously. However, usually, when querying student information, users only need to query the height of some students, such as querying the height of first-grade students. At this time, you still need to load the height data of all students and then filter out the height data of first-grade students.
[0003] In order to improve the query efficiency of column storage, the existing technology will first perform secondary pre-calculation for common query scenarios, store the calculation results in the data warehouse in advance, and directly load the stored calculation results when receiving the query instruction, so as to avoid the data screening process and directly display the query results to the user. However, the secondary calculation results need to be stored in the data warehouse, which occupies the storage space of the data warehouse.
[0004] How to reduce the space occupied by data storage while improving data query efficiency with column-based storage is a problem that people in this field need to solve. Summary of the invention
[0005] In view of the above problems, the present invention proposes a method and device for reading and writing a data table, the main purpose of which is to reduce the occupation of data warehouse storage space while improving data query efficiency.
[0006] In order to achieve the above object, the present invention mainly provides the following technical solutions:
[0007] In a first aspect, the present invention provides a method for reading and writing a data table, comprising:
[0008] Acquire multiple classification identifiers, where the classification identifiers are used to mark field value combinations, where field values in the field value combinations are field values corresponding to different specified fields in the data table to be stored;
[0009] Determine a data block for storing field values corresponding to a target field in the data table to be stored, wherein different data blocks store field values corresponding to different classification identifiers in the target field;
[0010] Writing the field value in the target field into the corresponding data block according to different classification identifiers;
[0011] After receiving a read instruction carrying the field name of the target field and a field value combination, the corresponding data block is searched according to the classification identifier corresponding to the field value combination and the field name of the target field, so as to send the data in the data block.
[0012] In a second aspect, the present invention provides a data reading and writing device based on a data warehouse, the device comprising:
[0013] An acquisition unit, used for acquiring a plurality of classification identifiers, wherein the classification identifiers are used for marking a field value combination, wherein the field values in the field value combination are field values corresponding to different designated fields in the data table to be stored;
[0014] A determination unit, used to determine a data block for storing a field value corresponding to a target field in the data table to be stored, wherein different data blocks store field values corresponding to different classification identifiers in the target field;
[0015] A writing unit, used for writing the field value in the target field into the corresponding data block according to different classification identifiers;
[0016] The reading unit is used to search for the corresponding data block according to the classification identifier corresponding to the field value combination and the field name of the target field after receiving a reading instruction carrying the field name of the target field and the field value combination, so as to send the data in the data block.
[0017] In a third aspect, the present invention further provides a storage medium, wherein the storage medium is used to store a computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute the data table reading and writing method of the first aspect.
[0018] In a fourth aspect, the present invention further provides an electronic device, comprising at least one processor, and at least one memory and a bus connected to the processor; wherein the processor and the memory communicate with each other via the bus; and the processor is used to call program instructions in the memory to execute the data table reading and writing method as described in the first aspect above.
[0019] By means of the above technical scheme, a data table reading and writing method and device provided by the present invention can reduce the occupation of data storage space while improving data query efficiency. The present application first obtains all classification identifiers, each classification identifier refers to a field value combination, the field value combination is composed of the field values of each specified field, and a field value combination corresponds to a type of field value in the target field of the data table to be stored. The present application classifies the field values in the target field, and the field values of the same type correspond to the same classification identifier; secondly, a data block is set for the target field, and different data blocks can store field values corresponding to different classification identifiers in the target field; then, based on the classification identifier, the field value of the target field value is written into the corresponding data block. After receiving a read instruction carrying the field name of the target field and the field value combination, the corresponding data block is searched according to the classification identifier corresponding to the field value combination and the field name of the target field, and the data in the data block is sent to the user. Compared with the prior art, the present application does not need to load all the data in the data table to be stored, nor does it need to store the pre-calculated results in the data warehouse. The present application improves the data query efficiency while reducing the occupation of the storage space in the data warehouse.
[0020] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:
[0022] Figure 1 A flow chart of a method for reading and writing a data table proposed in an embodiment of the present invention is shown;
[0023] Figure 2 A flowchart of another data table reading and writing method proposed in an embodiment of the present invention is shown;
[0024] Figure 3 A schematic diagram showing the structure of a data table reading and writing device proposed in an embodiment of the present invention;
[0025] Figure 4 A schematic diagram showing the structure of another data table reading and writing device proposed in an embodiment of the present invention;
[0026] Figure 5A schematic diagram of the structure of an electronic device proposed in an embodiment of the present invention. DETAILED DESCRIPTION
[0027] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.
[0028] Data warehouses usually store a large number of data tables, and each data table has a large amount of data. For example, a data table may record millions of data. When querying data in a data table, all the data in the data table will be loaded, the required data will be filtered out from all the data, and finally the filtered data will be sent to the client to display the data required by the user. Due to the large amount of data in the data table, it takes too long to load all the data in the data table, resulting in low data query efficiency. In order to improve data query efficiency, after storing the data table, a secondary pre-calculation is usually performed according to common query scenarios, and the pre-calculation results are stored in the data warehouse so that the pre-calculation results can be directly sent when querying. However, the pre-calculation results are stored in the data warehouse, occupying the storage resources of the data warehouse. When improving data query efficiency, how to reduce the occupation of data warehouse storage resources is a problem faced by the domain.
[0029] The inventor found that when a user queries the data in a data table in a data warehouse, the field values of several fields in the data table are usually used for querying, and such fields can be called commonly used fields. For example, the grade field and the class field in the student information table can be considered commonly used fields. Assuming that the field value of the grade field is the first grade and the second grade, and the field value of the class field is the first class, the second class, the third class, and the fourth class, when the height data of all students in the third class of the first grade need to be queried, the field value will be carried in the read instruction: the first grade, the third class, the field: height, which shows that the field value in the commonly used field is usually used as a condition for data query, i.e., a keyword. Therefore, the inventor believes that when the data is stored in a columnar manner, the data can be classified according to the field value in the commonly used field, and the classified data can be stored again. Since the data has been classified according to the field value of the commonly used field when it is stored, the data of the same class is stored together continuously, so when querying, the corresponding data of the same class can be directly obtained according to the field value corresponding to the commonly used field in the read instruction. Therefore, the present application can not only improve the query efficiency, but also does not need to store the pre-calculated results in the data warehouse. The specific implementation steps of the embodiment of this application are as follows: Figure 1 As shown, including:
[0030] 101. Obtain multiple classification identifiers, where the classification identifiers are used to mark field value combinations. Field values in the field value combination are field values corresponding to different specified fields in the data table to be stored.
[0031] The classification identifier is used to refer to a field value combination, and each classification identifier is a unique identifier for a corresponding field value combination. A field value combination is a combination of field values of commonly used fields in a data table to be stored. The designated fields and commonly used fields in the embodiments of the present application are the same concept.
[0032] In the embodiments of the present application, the commonly used fields of the data table to be stored are usually determined first. The determination method can be specified by the user or can be screened out from all the fields of the data table to be stored by the machine through the commonly used field screening rules. There can be one or more commonly used fields. After obtaining the commonly used fields, different field values will be screened out from each commonly used field, and the screened field values will be combined to obtain multiple field value combinations. It should be noted that each field value in each field value combination is obtained from different fields, that is, multiple field values of the same field will not appear in one field combination. In addition, it should also be noted that when there is only one commonly used field, each field value combination contains only one field value.
[0033] In order to distinguish different field combinations, each field combination will be marked to obtain multiple classification identifiers, and each classification identifier is the unique identifier of the corresponding field combination. When storing each field in the data table to be stored, the embodiment of the present application will store the field value corresponding to the same classification identifier together, so that when data is searched, the field value corresponding to the same classification identifier will be extracted together. Taking the student information table as an example, when storing the height field, the height data of all students in the first class of the first grade will be stored in a data block, and the height data of all students in the second class of the first grade will be stored in another data block. When the user needs to query the height of the first class of the first grade, the data in the corresponding data block is sent. Of course, if the height of all classes of students in the first grade needs to be queried, the data in the data blocks corresponding to all classes in the first grade can be obtained. Assuming that there are two classes in the first grade, it is necessary to obtain the data blocks corresponding to the first class of the first grade and the data blocks corresponding to the second class of the first grade.
[0034] Exemplarily, commonly used fields in the data table to be stored are filtered out according to preset filtering rules, duplicate field values in the commonly used fields are deleted, field values in the commonly used fields are combined to obtain multiple field value combinations, the field value combinations are numbered one by one, and a classification identifier for identifying the field value combination is obtained.
[0035] Exemplarily, multiple commonly used fields specified by a user are received, and for each commonly used field, different field values in the commonly used field are filtered out to obtain a field value set corresponding to the commonly used field, and a field value is extracted from each field value set to obtain a field value combination, and the field value combination is marked to obtain a classification identifier corresponding to the field value.
[0036] 102. Determine a data block for storing a field value corresponding to a target field in a data table to be stored.
[0037] Among them, different data blocks store field values corresponding to different classification identifiers in the target field. Each data block corresponds to a classification identifier. The embodiment of the present application is applied to column storage, that is, when storing data in the data table to be stored, the field value in the first field is stored first. When all field values in the first field are stored, the field value in the second field is stored, and so on. When storing field values in the target field, the embodiment of the present application needs to set up multiple data blocks first, so that the field values corresponding to different classification identifiers can be stored in different data blocks.
[0038] For example, the capacity of the data block is configured first, and a data block is allocated for each classification identifier. When a data block is full, a new data block is allocated to store the remaining field values. After all field values in the target field are stored, all data blocks are sorted in the order of the classification identifiers to achieve storage of the target field.
[0039] Exemplarily, based on the capacity of the data table to be stored and the number of fields in the data table to be stored, the field capacity of each field is estimated, and based on the field capacity, a corresponding storage space in the data warehouse is allocated to the target field, and the storage space is divided into multiple data blocks based on the number of classification identifiers. A corresponding relationship between each data block and a corresponding classification identifier is established so that the field value is written into the corresponding data block based on the classification identifier.
[0040] 103. Write the field value in the target field into the corresponding data block according to different classification identifiers.
[0041] In the embodiment of the present application, each field value in the target field corresponds to a classification identifier. When determining the classification identifier corresponding to the field value, it is possible to first determine which piece of data the field value belongs to, extract the field values corresponding to all the specified fields in the data, obtain a group of field values, traverse each field value combination, find the field value combination that is the same as the group of field values, and the classification identifier corresponding to the field value combination is the classification identifier corresponding to the field value. Since each data block corresponds to a classification identifier in the embodiment of the present application, the field value in the target field can be written into the corresponding data block using the classification identifier.
[0042] In addition, in the embodiment of the present application, when storing the target field, each field value is extracted in sequence according to the order of the field values in the target field. For example, the first field value in the target field is first extracted, and the first field value is written into the data block corresponding to the first field value, and then the second field value in the target field is extracted, and the second field value is written into the data block corresponding to the second field value. When the field value matches the corresponding data block, all data blocks will be traversed to find the data block corresponding to the field value. In order to improve the efficiency of matching data blocks, the embodiment of the present application can first sort the field values in the target field, and the field values corresponding to the same classification identifier are continuously arranged in the field, so that the multiple field values extracted continuously correspond to the same data block. If it is determined that the field value currently extracted corresponds to the same classification identifier as the field value extracted last, it is not necessary to traverse the data block, and the field value is directly written into the data block where the last field value is stored. It should be noted that when sorting the field values in the target field, the entire data table to be stored can be sorted according to the classification identifier, or the target field can be sorted separately.
[0043] 104. After receiving a read instruction carrying the field name of the target field and the field value combination, search for a corresponding data block according to the classification identifier corresponding to the field value combination and the field name of the target field.
[0044] The read instruction is used to query the data in the data table to be stored. The read instruction carries the field name and field value, and uses the carried field name and field value to match the data block in the data warehouse, and sends the data in the matched data block to the user. The field value is the field value in the field value combination.
[0045] In this step, the read instruction may carry the field name and field value of the target field, or may carry the field name and classification identifier of the target field. If the read instruction carries the field name and classification identifier of the target field, the data blocks corresponding to the classification identifier in each field may be found according to the classification identifier carried in the read instruction. Then, the data blocks corresponding to the target field may be screened out from multiple data blocks using the field name of the target field. The field name of the target field in the embodiment of the present application may be the field name of one target field, or may be the field name of multiple target fields. The specific implementation method of carrying the field value combination in the read instruction is similar to that of carrying the classification identifier, and will not be repeated here. After finding the corresponding data block according to the classification identifier corresponding to the field value combination and the field name of the target field, the data in the partial data block will be sent to the client so as to display the data required by the user on the client.
[0046] When storing a data table to be stored, the embodiment of the present application classifies the field values of each field in the data table to be stored based on multiple classification identifiers, and continuously stores the field values corresponding to the same classification identifier in a data block. When reading and storing the data in the data table to be stored, according to the field name and field value combination of the target field carried in the read instruction, and the correspondence between the field value combination and the classification identifier, the data in the data block corresponding to the classification identifier in the target field is sent. The embodiment of the present application stores the classified field values again, which is equivalent to the stored data being the pre-calculated result. When receiving the read instruction, all the data in one or more data blocks are directly loaded, eliminating the process of filtering out the required data from all the data in the data table to be stored, and there is no need to perform secondary pre-calculation after the data is stored, and the pre-calculated results are stored in the data warehouse. The occupation of data warehouse storage resources is reduced. In addition, in the process of writing the field value in the target field into the corresponding data block, the field value in the target field is first sorted so that the field values corresponding to the same data block are extracted continuously. When matching the data block, the process of traversing the data block is eliminated. As long as the current field value corresponds to the same classification identifier as the last extracted field value, the current field value is directly written into the data block containing the last field value. Therefore, the embodiment of the present application also improves the efficiency of writing field values into the data block, that is, speeds up the data storage speed.
[0047] In order to explain in more detail a method for reading and writing a data table proposed by the present invention, the present application proposes an embodiment of another method for reading and writing a data table. The specific implementation steps of the embodiment of the present invention are as follows: Figure 2 As shown, including:
[0048] 201. Receive a specified field set by a user.
[0049] The designated fields in the embodiment of the present application are commonly used fields. When storing a data table to be stored, the user can specify one or more most commonly used fields in the table according to the actual business scenario, so as to generate a classification identifier using these fields.
[0050] 202. Filter out different field values in each specified field to obtain multiple field value sets.
[0051] Among them, the field value set is a set containing multiple field values, and the field values in each field value set are the field values in the same specified field.
[0052] In this step, for each specified field, different field values in the specified field are screened out, and a field value set corresponding to the specified field is generated based on these different field values. For example, the field values in a specified field are A, A, B, C, C. After screening, the field values in the obtained field value set are A, B, C.
[0053] 203. Extract field values from multiple field value sets to obtain a field value combination.
[0054] exist Figure 1 In the embodiment shown, the number of field values in each field value combination is the same as the number of specified fields, that is, one field value is extracted from each specified field to obtain a field value combination. Therefore, each field value in the data table to be stored corresponds to only one classification identifier, and each field value is stored only once in the data warehouse. Figure 1 In addition to the field value combination in the illustrated embodiment, the field values in the partial field value set can also be combined to obtain more field value combinations, wherein, it also includes generating a field value combination only for a specified field, that is, a field combination contains only one field value. Taking the student information table as an example, the specified fields are grade and class, the field values of the grade field are the first grade and the second grade, and the field values of the class field are the first class and the second class. If you want to query the height of all students in the first grade, you only need to load data from the data block corresponding to the first grade. There is no need to load data from the data block corresponding to the first grade and the data block corresponding to the first grade and the second class, respectively, which further improves the data query efficiency. Each field value in the data table to be stored in the embodiment of the present application can correspond to multiple classification identifiers, that is, each field value is stored in multiple data blocks, and the data query efficiency is improved by storing the field value multiple times.
[0055] 204. Mark the field value combination to obtain a classification identifier corresponding to the field value combination.
[0056] In this step, among all the field value combinations obtained, each field value combination is marked so that each field combination has a unique identifier.
[0057] 205. Allocate storage space for the target field according to the capacity of the data table to be stored and the number of fields in the data table to be stored.
[0058] In this step, the capacity of the data table to be stored and the number of fields in the data table to be stored can be divided to obtain the estimated field capacity, and the corresponding storage space can be allocated to the target field in the data warehouse according to the estimated field capacity and the preset redundant capacity. The redundant capacity is the additional capacity to prevent the target field from having a large amount of data and the allocated storage space being insufficient.
[0059] 206. Divide the storage space to obtain multiple data blocks.
[0060] In this step, after allocating the storage space, the storage space needs to be divided into multiple small spaces to obtain multiple data blocks. The number of data blocks can be equal to the number of classification identifiers, or can be greater than the number of classification identifiers.
[0061] When each field value in the data table to be stored corresponds to only one classification identifier, the specific implementation method of dividing the storage space can be as follows:
[0062] Step 1: Calculate the ratio of the number of field values corresponding to each classification identifier to the total number of field values in the target field.
[0063] In this step, the number of field values corresponding to each classification identifier is counted in the target field, and the number of field values corresponding to each classification identifier is divided by the total number of field values in the target field to obtain the proportion of the number of field values corresponding to each classification identifier in the total number. The storage space is divided using the proportion, which is the ratio.
[0064] Step 2: Divide the storage space according to the ratio to obtain the data blocks corresponding to each classification identifier.
[0065] The storage space is divided according to the ratio, so that the size of each data block can be determined according to the number of stored field values. If there are many field values to be stored, the allocated data block capacity will be larger, and if there are few field values to be stored, the allocated data block capacity will be smaller.
[0066] The embodiment of the present application determines the size of the data block according to the number of field values that actually need to be stored, so as to minimize the waste of storage space caused by too many idle data blocks.
[0067] In another achievable manner, the specific steps for dividing the storage space are as follows. This implementation manner can be applicable to each field value corresponding to only one classification identifier, or can be applicable to each field value corresponding to multiple classification identifiers:
[0068] Step 1: Configure data block capacity.
[0069] In this step, a fixed capacity is configured for the data block. The data block capacity can be determined according to the actual business scenario. If the capacity of the data table to be stored is large, the data block capacity can be configured larger, and vice versa.
[0070] Step 2: Divide the storage space according to the data block capacity to obtain multiple data blocks of the same capacity.
[0071] In this step, a corresponding data block can be divided for each classification identifier in the storage space according to the number of classification identifiers and the data block capacity, and the remaining space is temporarily not divided. When a data block is full, a new data block is divided for the full data block to store the remaining field values corresponding to the full data block. The storage space can also be divided into two spaces, a target storage space and a spare storage space. The data blocks divided from the target storage space are target data blocks, and the data blocks divided from the spare storage space are spare data blocks. Different target data blocks store field values corresponding to different classification identifiers. The spare data blocks are used to store the remaining field values corresponding to the same classification identifier when the target data block is full. The specific implementation steps for dividing the storage space into two spaces to obtain multiple data blocks are as follows:
[0072] Step 2a: Count the number of classification labels.
[0073] Step 2b: divide the storage space according to the number of classification identifiers and the data block capacity to obtain target storage space and spare storage space. The target storage space is the storage space for dividing the target data block, and the capacity of the spare storage space is the difference between the storage space and the target storage space.
[0074] In this step, the number of classification identifiers and the data block capacity are multiplied, and the resulting capacity value is the capacity value of the target storage space.
[0075] Step 2c: based on the data block capacity, divide the target storage space to obtain the target data block.
[0076] Step 2d: Based on the data block capacity, divide the spare storage space to obtain spare data blocks.
[0077] It should be noted that before the spare data block is not written with a field value, the correspondence between the spare data block and the classification identifier may not be generated. When the target data block is full, the classification identifier corresponding to the target data block is obtained, and the correspondence between the spare data block and the classification identifier is generated based on the classification identifier.
[0078] The specific implementation steps of storing the field value into the spare data block may be: determining whether the target data block corresponding to the classification identifier is full; if not, writing the field value into the target data block; if so, writing the field value into the spare data block corresponding to the classification identifier, and merging the target data block and the spare data block. After merging the target data block and the spare data block, a new data block with twice the data block capacity is formed, and the remaining field values corresponding to the same classification identifier will be directly written into the new data block.
[0079] The embodiment of the present application merges small-capacity data blocks into large-capacity data blocks. For field values of different categories, a category of field values with large data volume corresponds to large-capacity data blocks, and a category of field values with small data volume corresponds to small-capacity data blocks, thereby achieving reasonable and accurate division of data blocks.
[0080] 207. Determine the classification identifier corresponding to the field value of the target field according to the field value in each designated field that belongs to the same piece of data as the field value of the target field.
[0081] In this step, first determine in which data the target field is located, then obtain the other field values in the data except the target field, filter out the field values in the specified field from the partial field values, and use the filtered field values to determine the classification identifier corresponding to the field value of the target field.
[0082] 208. Write the field value of the target field into the data block corresponding to the classification identifier.
[0083] In this step, when the target data blocks are full, the data blocks corresponding to the classification identifier include the target data blocks corresponding to the classification identifier and the spare data blocks corresponding to the classification identifier.
[0084] 209. After receiving a read instruction carrying the field name of the target field and the field value combination, search for a corresponding data block according to the classification identifier corresponding to the field value combination and the field name of the target field.
[0085] This step is performed in the same way as Figure 1 The specific content of step 104 of the illustrated embodiment will not be repeated any more.
[0086] Furthermore, as a response to the above Figure 1-2 The implementation of the method embodiment shown in the figure, the embodiment of the present invention provides a data table reading and writing device, which is used to reduce the occupation of data warehouse storage space while improving query efficiency. The embodiment of the device corresponds to the aforementioned method embodiment. For ease of reading, this embodiment will no longer repeat the details of the aforementioned method embodiment one by one, but it should be clear that the device in this embodiment can correspond to all the contents of the aforementioned method embodiment. Specifically, Figure 3 As shown, the device comprises:
[0087] An acquiring unit 31 is used to acquire a plurality of classification identifiers, wherein the classification identifiers are used to mark a field value combination, and the field values in the field value combination are field values corresponding to different specified fields in the data table to be stored;
[0088] A determination unit 32 is used to determine a data block for storing a field value corresponding to a target field in the data table to be stored, wherein different data blocks store field values corresponding to different classification identifiers obtained by the acquisition unit 31 in the target field;
[0089] A writing unit 33, configured to write the field value in the target field into a corresponding data block according to different classification identifiers, wherein the data block is determined by the determining unit 32;
[0090] The reading unit 34 is used to search for the corresponding data block according to the classification identifier corresponding to the field value combination and the field name of the target field after receiving a reading instruction carrying the field name of the target field and the field value combination, so as to send the data in the data block.
[0091] Further, such as Figure 4 As shown, the acquisition unit 31 includes:
[0092] A receiving module 311 is used to receive a specified field set by a user;
[0093] A screening module 312 is used to screen out different field values in the designated field specified by each receiving module 311 to obtain multiple field value sets, each field value set corresponding to a designated field;
[0094] An extraction module 313 is used to extract field values from the field value sets obtained by the multiple screening modules 312 to obtain a field value combination;
[0095] The marking module 314 is used to mark the field value combination obtained by the extraction module 313 to obtain a classification identifier corresponding to the field value combination.
[0096] Further, such as Figure 4 As shown, the determining unit 32 includes:
[0097] An allocation module 321, configured to allocate storage space for the target field according to the capacity of the data table to be stored and the number of fields in the data table to be stored;
[0098] The partitioning module 322 is used to partition the storage space obtained by the allocation module 321 to obtain multiple data blocks.
[0099] Furthermore, the division module 322 includes:
[0100] A calculation submodule, used to calculate the ratio of the number of field values corresponding to each classification identifier to the total number of field values in the target field;
[0101] The first division submodule is used to divide the storage space according to the ratio obtained by the calculation submodule to obtain data blocks corresponding to each classification identifier.
[0102] Furthermore, the division module 322 further includes:
[0103] Configuration submodule, used to configure data block capacity;
[0104] The second division submodule is used to divide the storage space according to the data block capacity obtained by the configuration submodule to obtain multiple data blocks with the same capacity.
[0105] Further, the data block is a target data block or a spare data block, different target data blocks store field values corresponding to different classification identifiers, and the spare data block is used to store the remaining field values corresponding to the same classification identifier when the target data block is full, then the second division submodule is specifically used for:
[0106] Counting the number of classification identifiers;
[0107] According to the number of classification identifiers and the data block capacity, the storage space is divided to obtain a target storage space and a spare storage space, wherein the target storage space is the storage space for dividing the target data block, and the capacity of the spare storage space is the difference between the storage space and the target storage space;
[0108] Based on the data block capacity, dividing the target storage space to obtain a target data block;
[0109] Based on the data block capacity, the spare storage space is divided to obtain spare data blocks.
[0110] Further, such as Figure 4 As shown, the writing unit 33 includes:
[0111] A determination module 331, configured to determine a classification identifier corresponding to the field value of the target field according to the field value in each designated field that belongs to the same piece of data as the field value of the target field;
[0112] A judging module 332, used to judge whether the target data block corresponding to the classification identifier determined by the determining module 331 is full;
[0113] A first writing module 333, configured to write the field value into the target data block when the judging module 332 judges as no;
[0114] The second writing module 334 is used to write the field value into the spare data block corresponding to the classification identifier and merge the target data block and the spare data block when the judging module 332 judges to be yes.
[0115] Furthermore, an embodiment of the present invention further provides a processor, the processor is used to run a program, wherein the program executes the above Figure 1-2 The reading and writing methods of the data table described in .
[0116] Furthermore, an embodiment of the present invention further provides a storage medium, wherein the storage medium is used to store a computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute the above Figure 1-2 The reading and writing methods of the data table described in .
[0117] Furthermore, an embodiment of the present invention provides an electronic device 4, such as Figure 5 As shown, the device includes at least one processor 41, and at least one memory 42 and a bus 43 connected to the processor 41; wherein the processor 41 and the memory 42 communicate with each other through the bus 43; the processor 41 is used to call the program instructions in the memory 42 to execute the above-mentioned data table reading and writing method.
[0118] The devices in this article can be servers, PCs, PADs, mobile phones, etc.
[0119] Furthermore, the present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program for initializing the reading and writing method steps of the data table as described above.
[0120] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0121] It is understandable that the related features in the above methods and devices can be referenced to each other. In addition, the "first", "second" and the like in the above embodiments are used to distinguish the embodiments, but do not represent the advantages and disadvantages of the embodiments.
[0122] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0123] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages can be utilized to realize the content of the present invention described herein, and the description of the above specific languages is for disclosing the best mode of the present invention.
[0124] In addition, the memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.
[0125] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0126] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0127] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0128] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0129] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0130] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0131] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0132] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0133] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0134] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.
Claims
1. A method for reading and writing a data table, characterized in that: The method comprises: Acquire multiple classification identifiers, where the classification identifiers are used to mark field value combinations, where field values in the field value combinations are field values corresponding to different specified fields in the data table to be stored; Determine a data block for storing field values corresponding to a target field in the data table to be stored, wherein different data blocks store field values corresponding to different classification identifiers in the target field; Writing the field value in the target field into the corresponding data block according to different classification identifiers; After receiving a read instruction carrying the field name of the target field and a field value combination, the corresponding data block is searched according to the classification identifier corresponding to the field value combination and the field name of the target field, so as to send the data in the data block.
2. The method according to claim 1, characterized in that Determining a data block for storing a field value corresponding to a target field in the data table to be stored includes: Allocating storage space for the target field according to the capacity of the data table to be stored and the number of fields in the data table to be stored; The storage space is divided into multiple data blocks.
3. The method according to claim 2, characterized in that The storage space is divided to obtain a plurality of data blocks, including: Calculate the ratio of the number of field values corresponding to each classification identifier to the total number of field values in the target field; The storage space is divided according to the ratio to obtain data blocks corresponding to each classification identifier.
4. The method according to claim 2, characterized in that: The storage space is divided to obtain a plurality of data blocks, including: Configure data block capacity; The storage space is divided according to the data block capacity to obtain multiple data blocks with the same capacity.
5. The method according to claim 4, characterized in that The data block is a target data block or a spare data block, different target data blocks store field values corresponding to different classification identifiers, and the spare data block is used to store remaining field values corresponding to the same classification identifier when the target data block is full. The storage space is divided according to the data block capacity to obtain multiple data blocks of the same capacity, including: Counting the number of classification identifiers; According to the number of classification identifiers and the data block capacity, the storage space is divided to obtain a target storage space and a spare storage space, wherein the target storage space is the storage space for dividing the target data block, and the capacity of the spare storage space is the difference between the storage space and the target storage space; Based on the data block capacity, dividing the target storage space to obtain a target data block; Based on the data block capacity, the spare storage space is divided to obtain spare data blocks.
6. The method according to claim 5, characterized in that Writing the field value in the target field into the corresponding data block according to different classification identifiers includes: Determine the classification identifier corresponding to the field value of the target field according to the field value in each designated field that belongs to the same piece of data as the field value of the target field; Determine whether the target data block corresponding to the classification identifier is full; If not, writing the field value into the target data block; If so, the field value is written into the spare data block corresponding to the classification identifier, and the target data block and the spare data block are merged.
7. The method according to any one of claims 1 to 6, characterized in that: Get multiple classification identifiers, including: Receive the specified fields set by the user; Filter out different field values in each specified field to obtain multiple field value sets, each field value set corresponding to a specified field; Extracting field values from the multiple field value sets to obtain a field value combination; The field value combination is marked to obtain a classification identifier corresponding to the field value combination.
8. A data table reading and writing device, characterized in that: The device comprises: An acquisition unit, used for acquiring a plurality of classification identifiers, wherein the classification identifiers are used for marking a field value combination, wherein the field values in the field value combination are field values corresponding to different designated fields in the data table to be stored; A determination unit, used to determine a data block for storing a field value corresponding to a target field in the data table to be stored, wherein different data blocks store field values corresponding to different classification identifiers in the target field; A writing unit, used for writing the field value in the target field into the corresponding data block according to different classification identifiers; The reading unit is used to search for the corresponding data block according to the classification identifier corresponding to the field value combination and the field name of the target field after receiving a reading instruction carrying the field name of the target field and the field value combination, so as to send the data in the data block.
9. A storage medium, characterized in that: The storage medium is used to store a computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute the data table reading and writing method according to any one of claims 1 to 7.
10. An electronic device, characterized in that: The device includes at least one processor, and at least one memory and a bus connected to the processor; wherein the processor and the memory communicate with each other through the bus; the processor is used to call program instructions in the memory to execute the data table reading and writing method as described in any one of claims 1-7.
Citation Information
Patent Citations
Information storage method and electronic equipment
CN107861694A
Data writing method and device
CN110019210A
Data storage method, data reading method and device
CN110069488A
Data storage method and device and electronic device
CN111506569A
Data storage method, system, device and equipment and medium
CN112817543A
Cited By
Data processing method and device and storage medium
CN122614842A
