Feature storage method and device, computer and storage medium

By constructing key-value pairs and compressing them based on hash values, the method addresses inefficient storage allocation, achieving balanced data distribution and improved resource utilization in feature storage systems.

CN119066232APending Publication Date: 2024-12-03WEBANK (CHINA)
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411055696.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Current feature storage solutions in the banking industry require excessive space for large data volumes without a reasonable adjustment mechanism, leading to inefficient allocation of storage resources.

Method used

A method involving constructing key-value pairs, performing hash calculations to determine storage partitions, and compressing these pairs to accurately allocate storage space based on feature values.

Benefits of technology

This approach enables efficient and accurate allocation of storage space, balancing data distribution and reducing resource wastage, thereby enhancing the scalability and efficiency of feature storage systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119066232A_ABST
    Figure CN119066232A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a feature storage method and device, a computer and a storage medium, applied to a server of a distributed storage system.The method comprises the steps that feature group configuration information of a data set is obtained, and the feature group configuration information comprises dimension keys, feature names, feature numbers and feature data types; combining the dimension key and the value of the feature name corresponding to the dimension key to obtain a key value pair; performing hash calculation on the dimension key in the key value pair to obtain a target hash value; determining a target storage partition of the key value pair according to the target hash value and a preset partition number; and compressing the key value pair in the target storage partition, and determining a storage space of the key value pair. The key value pair is constructed according to the feature value, then the storage partition is determined according to the hash value of the key value pair, and finally the key value pair is compressed in the storage partition so as to determine the space required by feature value storage, so that accurate allocation of the storage space is facilitated.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of feature processing, and in particular to a feature storage method, device, computer and storage medium. Background Art

[0002] With the rapid development and popularization of mobile Internet, the banking industry is facing unprecedented pressure for change. In order to maintain competitiveness in the current environment, banks must keep up with the pace of the Internet era and strengthen their online marketing and risk control capabilities.

[0003] Traditional banks mainly conduct risk assessment and control through offline channel development by sales staff and the experience of business experts. However, with the significant growth of client user traffic, it is difficult to rely solely on people to control it. At present, the industry usually uses user portraits and behavioral features collected by the system to make real-time system decisions through rules and model engines. Therefore, to achieve more accurate and efficient decision-making, the key is to obtain more comprehensive user features, so the storage of features becomes particularly critical.

[0004] However, the feature storage solutions currently used in the industry often require more space to store these features when dealing with large amounts of data. There is no reasonable adjustment mechanism to allocate feature storage space, which leads to unreasonable allocation of storage space required for features. Summary of the invention

[0005] The embodiments of the present application provide a feature storage method, device, computer and storage medium, which construct key-value pairs according to feature values, then determine storage partitions according to hash values ​​of the key-value pairs, and finally compress the key-value pairs in the storage partitions to determine the space required for feature value storage, which is conducive to achieving accurate allocation of storage space.

[0006] In a first aspect, an embodiment of the present application provides a feature storage method, including:

[0007] Acquire feature group configuration information of a data set, the feature group configuration information including dimension key, feature name, feature number and feature data type; obtain a key-value pair by combining the dimension key and the value of the feature name corresponding to the dimension key; perform hash calculation on the dimension key in the key-value pair to obtain a target hash value; determine a target storage partition for the key-value pair according to the target hash value and a preset number of partitions; compress the key-value pair within the target storage partition to determine the storage space for the key-value pair.

[0008] In a second aspect, an embodiment of the present application provides a feature storage device, including:

[0009] An acquisition module, used to acquire feature group configuration information of a data set, wherein the feature group configuration information includes a dimension key, a feature name, a feature number, and a feature data type;

[0010] A processing module, used to obtain a key-value pair by combining the dimension key and the value of the feature name corresponding to the dimension key; and used to perform hash calculation on the dimension key in the key-value pair to obtain a target hash value; and used to determine a target storage partition for the key-value pair based on the target hash value and a preset number of partitions; and used to compress the key-value pair within the target storage partition to determine a storage space for the key-value pair.

[0011] In a third aspect, an embodiment of the present application provides a feature storage device, the device comprising:

[0012] A memory, a processor, and a feature storage program stored in the memory and executable on the processor, wherein the feature storage program is configured to implement part or all of the steps described in any method in the first aspect.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a feature storage program is stored. When the feature storage program is executed by a processor, some or all of the steps described in any method in the first aspect are implemented.

[0014] In a fifth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, and the computer program is operable to cause a computer to execute some or all of the steps described in any method of the first aspect of the embodiment of the present application. The computer program product can be a software installation package.

[0015] By implementing the embodiment of the present application, the server of the distributed storage system first obtains the feature group configuration information of the data set, and the feature group configuration information includes dimension key, feature name, feature number and feature data type; then the key-value pair is obtained by combining the value of the dimension key and the feature name corresponding to the dimension key; then the value of the dimension key in the key-value pair is hashed to obtain a hash value; then the target storage partition of the key-value pair is determined according to the hash value of the key-value pair and the preset number of partitions; finally, the key-value pair is compressed in the target storage partition to determine the storage space of the key-value pair. By constructing the key-value pair according to the feature value, then determining the storage partition according to the hash value of the key-value pair, and finally compressing the key-value pair in the storage partition to determine the space required for the feature value storage, it is conducive to the accurate allocation of storage space. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0017] Figure 1 It is a schematic diagram of the architecture of a distributed storage system provided in an embodiment of the present application;

[0018] Figure 2 is a flow chart of a feature storage method provided in an embodiment of the present application;

[0019] Figure 3 It is a schematic diagram of a table of feature group configuration information provided in an embodiment of the present application;

[0020] Figure 4 is a schematic diagram of a computing target storage partition provided in an embodiment of the present application;

[0021] Figure 5 It is a structural diagram of a feature storage provided by an embodiment of the present application;

[0022] Figure 6 is a schematic diagram of another structure of feature storage provided in an embodiment of the present application;

[0023] Figure 7 This is a flowchart of constructing an SST file provided in an embodiment of the present application;

[0024] Figure 8 It is a schematic diagram of SST file parsing provided in an embodiment of the present application;

[0025] Fig. 9 It is a schematic diagram of a feature reading method provided in an embodiment of the present application;

[0026] Fig.10 is a structural schematic diagram of a feature storage device provided in an embodiment of the present application;

[0027] Fig.11 It is a structural diagram of a computer provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. According to the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0029] The terms "first", "second", "third", etc. in the specification, claims and drawings of this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0030] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0031] With the rapid development and popularization of mobile Internet, the banking industry is facing unprecedented pressure for change. In order to maintain competitiveness in the current environment, banks must keep up with the pace of the Internet era and strengthen their online marketing and risk control capabilities. Traditional banks mainly conduct risk assessment and control through offline channel development of sales personnel and the experience of business experts. However, with the significant increase in client user traffic, it is difficult to rely solely on people to control. At present, the industry usually uses user portraits and behavioral features collected by the system to make real-time system decisions through rules and model engines. Therefore, the key to achieving more accurate and efficient decision-making is to obtain more comprehensive user features, so the storage of features becomes particularly critical.

[0032] However, the feature storage solutions commonly used in the industry often require more space to store these features when dealing with large amounts of data. There is no reasonable adjustment mechanism to allocate feature storage space, which leads to unreasonable allocation of storage space required for features. The current common feature storage methods include the following three:

[0033] Static feature storage partitioning: The static feature storage partitioning method allocates storage space to each feature in advance. Each feature is allocated a fixed storage space and will not be dynamically adjusted according to its access pattern or changing needs.

[0034] Fixed-size feature vector storage method: The fixed-size feature vector storage method assumes that each feature vector occupies a fixed-size storage unit, regardless of the actual dimension and sparsity of the feature vector.

[0035] Fixed allocation storage strategy method: The fixed allocation storage strategy method uses a fixed storage strategy to store features, such as uniform distribution or allocating storage space based on simple feature attributes.

[0036] It can be seen that the above-mentioned common feature storage methods can realize feature storage, but cannot dynamically allocate storage resources according to the data volume and feature attributes, resulting in unreasonable use of storage resources.

[0037] In response to the above problems, the embodiments of the present application provide a feature storage method, device, computer and storage medium. The server of the distributed storage system first obtains the feature group configuration information of the data set, and the feature group configuration information includes dimension key, feature name, feature number and feature data type; then the key-value pair is obtained by combining the value of the dimension key and the feature name corresponding to the dimension key; then the value of the dimension key in the key-value pair is hashed to obtain a hash value; then the target storage partition of the key-value pair is determined according to the hash value of the key-value pair and the preset number of partitions; finally, the key-value pair is compressed in the target storage partition to determine the storage space of the key-value pair. By constructing the key-value pair according to the feature value, then determining the storage partition according to the hash value of the key-value pair, and finally compressing the key-value pair in the storage partition to determine the space required for the feature value storage, it is conducive to the accurate allocation of storage space.

[0038] The feature storage method, device, computer and storage medium provided in the embodiments of the present application can be applied to Figure 1 In the distributed storage system shown in Figure 1 , Figure 1 1 is a schematic diagram of the architecture of a distributed storage system 100 provided in an embodiment of the present application. The distributed storage system 100 includes a client 101 and a server 102. The client 101 can communicate with the server 102 through a network. The client 101 refers to a device used by a user, such as a smart phone, a computer, etc. The user can interact with the system server 102 through the client 101. In this solution, the client 101 is mainly responsible for sending requests to the server, including operations such as reading specific feature values, writing new feature values, or updating existing feature values, and sending some parameters required for feature storage to the server 102.

[0039] The server 102 refers to a remote computer for processing a large number of computing tasks and storing data, and is responsible for data storage and management in the distributed storage system 100. In this solution, the server 102 receives data from the client and performs operations such as reading, writing, updating or deleting the data.

[0040] Based on this, the present application provides a feature storage method, device, computer and storage medium, and the present application is described in detail below in conjunction with the accompanying drawings.

[0041] See also Figure 2 , Figure 2 is a flow chart of a feature storage method provided in an embodiment of the present application, such as Figure 2 As shown, the method comprises the following steps:

[0042] S201, acquiring feature group configuration information of a data set, wherein the feature group configuration information includes a dimension key, a feature name, a feature number, and a feature data type;

[0043] The distributed storage system used in this solution is Tair (The Alibaba Infrastructure for Resource Sharing). Tair supports multiple operation interfaces, including standard key-value operations (Get, Put, Delete), atomic counter operations, and range query functions. Tair also provides powerful monitoring tools and management console interfaces to help administrators monitor cluster status, diagnose problems, and perform configuration management and performance tuning tasks. In addition to using Tair as a distributed storage system, you can also use Hadoop Distributed File System (HDFS) or Gluster File System (GlusterFS).

[0044] Among them, see Figure 3 , Figure 3 is a schematic diagram of a table of feature group configuration information provided in an embodiment of the present application, such as Figure 3 As shown, a data model Domain is defined. Domain is a collection of features. The feature group configuration information of the data model Domain includes the following parameters: parameter domainId, named as the feature group number, used to identify the unique number of a set of feature configurations; parameter domainKey, named as the dimension key, used to represent the primary key on which a set of feature reading depends; parameter featureId, named as the feature number, used to represent the serial number located and increasing; parameter datatype, named as the feature data type; parameter featureName, named as the feature name, used to represent the name of the field in the feature; wherein, the feature data type can be a fixed-length type or a variable-length type. Common fixed-length datatype types include integer type (Integer), floating-point type (Floating-point), character type (Character) and Boolean type (Boolean), and common variable-length datatype types include string type (String), array (Array), list (List), set (Set), etc.

[0045] After the data model Domain is configured, the feature group configuration information remains unchanged. For example, in a production task, the data model Domain includes three features, and the configuration information of each feature is as follows:

[0046] Feature 1: "domainId": "1", "domainKey": "userId", "featureId": 1, "dataType": "Integer", "featureName": "last_3_pay_times";

[0047] Feature two: "domainId": "1", "domainKey": "userId", "featureId": 2, "dataType": "Double", "featureName": "last_3_pay_amount";

[0048] Feature three: "domainId": "1", "domainKey": "userId", "featureId": 3, "dataType": "Double", "featureName": "last_point_product".

[0049] Among them, the feature group number domainId of all features in the same data model Domain is the same, and the feature number featureId of all features in the same data model Domain is different. Users can create, edit and delete parameters in the data model Domain from the management console of the Tair system. At the same time, users can load the configuration information of the data model Domain from the management console configuration of the Tair system to the executors in various distributed computing frameworks (i.e. Hadoop). The executor is the working node in Tair used to execute tasks and process data.

[0050] S202, obtaining a key-value pair by combining the dimension key and the value of the feature name corresponding to the dimension key;

[0051] Among them, the dimension key, that is, the parameter in the data model Domain is domainKey, which is the key of the key-value pair. The value of the feature name corresponding to the dimension key is aggregated as the value of the key-value pair. The two are combined to form a key-value pair.

[0052] For example, in a production task, the configuration information of a feature in the data model Domain is as follows:

[0053] Feature 4: "domainId": "1", "domainKey": "userId", "featureId": 1, "dataType": "Integer", "featureName": "last_3_pay_times".

[0054] The userId under each dimension key domainKey is used as the key of the key-value pair, and the value corresponding to the key of the key-value pair is a JSON string composed of multiple feature names and feature values. The three key-value pairs obtained are as follows:

[0055] Key-value pair 1: "zhangsan":

[0056] {\"last_3_pay_times\":10,\"last_3_pay_amount\":200.0,\"last_point_product\":\"WYD\"}";

[0057] Key-value pair 2: "lisi":

[0058] {\"last_3_pay_times\":5,\"last_3_pay_amount\":50.0,\"last_point_product\":\"GHD\"}";

[0059] Key-value pair three: "wangwu":

[0060] {\"last_3_pay_times\":22,\"last_3_pay_amount\":320.21,\"last_point_product\":\"GHD\"}".

[0061] S203, performing hash calculation on the dimension key in the key-value pair to obtain a target hash value;

[0062] Among them, each execution node in Tair pulls the feature group configuration information from the Hive table in the behavior dimension, and uses the dimension key domainKey as the key of the key-value pair. The values ​​of a group of feature names are aggregated as the value of the key-value pair, and then extracted into the memory of the corresponding execution node according to the key-value sequence. After that, each execution node performs hash calculation on the key in each stored key-value pair to obtain a hash value.

[0063] Among them, hash calculation can map data of any length to a value of fixed length. Specifically, hash calculation can select different hash functions as needed. Common hash functions include Message Digest Algorithm 5 (MD5), Secure Hash Algorithm 1 (SHA-1) and Murmur Hash function.

[0064] For example, the key in key-value pair 4 is "John". Now we need to perform hash calculation on "John" to obtain its corresponding hash value. The hash calculation process is as follows:

[0065] Step 1: Split the four keys of the key-value pair according to letters, and convert each letter obtained by the split into the American Standard Code for Information Interchange (ASCII). The ASCII code corresponding to the letter "J" is 74, the ASCII code corresponding to the letter "o" is 111, the ASCII code corresponding to the letter "h" is 104, and the ASCII code corresponding to the letter "n" is 110. The sum of the ASCII codes of these four letters is 74+111+104+111=399.

[0066] Step 2: Set the remainder base to 100, and then take the remainder 99 of the sum of the ASCII codes of the four letters as the hash value of "John".

[0067] If the hash function is MurmurHash, a parameter seed can be set, for example, the parameter seed is 13, and then the hash value corresponding to "John" can be directly calculated through the formula Mrumurhash ("John", 13).

[0068] S204, determining a target storage partition for the key-value pair according to the target hash value and a preset number of partitions;

[0069] The hash value of the key of the key-value pair, that is, the target hash value, is represented by hash(key). The preset number of partitions is the number of Tair buckets, which represents the number of shards of data in the distributed storage system. The preset number of partitions can be customized according to the amount of data, and the default number of preset partitions is 1023.

[0070] In a possible implementation, determining a target storage partition of the key-value pair according to the target hash value and a preset number of partitions includes:

[0071] Determine a first storage partition for the key-value pair according to the hash value and a preset number of partitions; determine a target storage node for the key-value pair according to the first storage partition and a plurality of storage nodes; and determine a second storage partition for the key-value pair in the target storage node according to the first storage partition and the number of repartitions, the second storage partition being the target storage partition for the key-value pair.

[0072] The step of determining the first storage partition of the key-value pair according to the hash value and the preset number of partitions includes:

[0073] Obtaining a hash value and a preset number of partitions of the key-value pair; and obtaining a first remainder by performing a modulo operation on the hash value of the key-value pair and the preset number of partitions, wherein the first remainder is a first storage partition of the key-value pair.

[0074] Specifically, the calculation formula for determining the first storage partition of the key-value pair according to the target hash value and the preset number of partitions is as follows:

[0075] bucket=hash(key)%bucketSum%2^32;

[0076] Among them, bucket is the first storage partition, bucket is a Tair bucket number, hash(key) is the target hash value, and bucketSum is the preset number of partitions.

[0077] Exemplarily, the key in key-value pair five is "John", the target hash value obtained after hash calculation of "John" is 99, bucketSum is the default value 1023, and according to the calculation formula of the above target storage partition, bucket = 99% 1023% 2^32 = 99, so the first storage partition of key-value pair five obtained by the calculation formula of the first storage partition is 99.

[0078] It can be seen that in this example, different key-value pairs can be mapped to different storage partitions by performing a modulo operation on the hash value of the key-value pair and the preset number of partitions, and then determining the storage partition of the key-value pair based on the remainder. This helps to keep the data volume in each storage partition relatively balanced and avoid overloading of certain nodes or partitions.

[0079] In a possible implementation, each storage node in the plurality of storage nodes corresponds to a storage partition, and determining a target storage node for the key-value pair according to the first storage partition and the plurality of storage nodes includes:

[0080] Sort the multiple storage nodes in order according to the storage partitions to obtain a storage node sequence; connect the storage node sequence end to end to obtain a storage node circular linked list; determine whether the first storage partition corresponds to a storage node in the storage node circular linked list; detect that the first storage partition corresponds to a storage node in the storage node circular linked list, and determine that the storage node corresponding to the first storage partition is the target storage node of the key-value pair; detect that the first storage partition does not correspond to a storage node in the storage node circular linked list, obtain a first storage node in the storage node circular linked list that is adjacent to the first storage partition and located in a clockwise direction of the first storage partition; determine that the first storage node is the target storage node of the key-value pair.

[0081] Among them, if there is a storage node among multiple storage nodes whose storage partition is equal to the first storage partition, the storage node is determined to be the target storage node of the key-value pair. For example, Tair currently deploys four nodes, namely node 1, node 2, node 3 and node 4, among which the storage partition corresponding to node 1 is 10, the storage partition corresponding to node 2 is 20, the storage partition corresponding to node 3 is 30, and the storage partition corresponding to node 4 is 50. At this time, the first storage partition of the key-value pair is 20, and the storage partition of node 2 is equal to the first storage partition. Node 2 is determined to be the target storage node of the key-value pair.

[0082] Tair deploys multiple storage nodes, which are physical or virtual servers used to store and process data in the Tair system. These storage nodes together form a distributed storage cluster. In order to achieve data load balancing and concurrent access capabilities, Tair divides the entire data set into multiple buckets and assigns each bucket to a different storage node for management. Each storage node corresponds to a bucket number.

[0083] For example, Tair currently deploys four nodes, namely node 1, node 2, node 3 and node 4, where the storage partition corresponding to node 1 is 10, the storage partition corresponding to node 2 is 20, the storage partition corresponding to node 3 is 30, and the storage partition corresponding to node 4 is 50. The key of key-value pair 6 is "zhangsan". After the hash value of the key of key-value pair 6 is calculated for the first storage partition, it is determined that the first storage partition of "zhangsan" is 5. At this time, the storage node group greater than 5 is [node 1, node 2, node 3, node 4], where the storage nodes adjacent to the first storage partition and located in the first storage partition in chronological order are [node 1, node 2, node 3, node 4]. The first storage node in the clockwise direction is node 1, therefore, "zhangsan" is stored on node 1, and node 1 is the target storage node for key-value pair six; similarly, the key of key-value pair seven is "wangwu", and after performing the above-mentioned calculation of the first storage partition on the hash value of the key of key-value pair seven, it is determined that the first storage partition of "wangwu" is 25, and the storage node group greater than 25 is [node 3, node 4] at this time, among which the first storage node adjacent to the first storage partition and located in the clockwise direction of the first storage partition is node 3, therefore, "zhangsan" is stored on node 3, and node 3 is the target storage node for key-value pair seven.

[0084] It can be seen that in this example, by comparing the first storage partition of the key-value pair with the storage partitions of multiple storage nodes and finally determining the target storage node of the key-value pair, the key-value pair can be located to the appropriate target storage node to achieve load balancing, thereby meeting large-scale data processing and high concurrent access requirements.

[0085] In another possible implementation, a storage node group is detected in which storage partitions of the multiple storage nodes are all smaller than the first storage partition; and a storage node with the smallest storage partition in the storage node group is determined as a target storage node for the key-value pair.

[0086] Exemplarily, Tair currently deploys four nodes, namely node 1, node 2, node 3 and node 4, where the bucket number corresponding to node 1 is 10, the bucket number corresponding to node 2 is 20, the bucket number corresponding to node 3 is 30, and the bucket number corresponding to node 4 is 50. The key of the key-value pair eight is "zhaoqian". After the hash value of the key of the key-value pair eight is calculated for the above-mentioned first storage partition, the storage partition of "zhaoqian" is determined, that is, the bucket number is 55. At this time, there is no storage node greater than 55. Among the four nodes, the node with the smallest bucket number is node 1. Therefore, "zhaoqian" is stored on node 1, and node 1 is the target storage node for the key-value pair eight.

[0087] It can be seen that in this example, when the first storage partition of the key-value pair is larger than the storage partitions of multiple storage nodes, the storage node with the smallest storage partition is determined as the target storage node of the corresponding key-value pair. The key-value pair can be located at the appropriate target storage node to achieve load balancing, thereby meeting large-scale data processing and high concurrent access requirements.

[0088] In a possible implementation, determining the second storage partition of the key-value pair in the target storage node according to the first storage partition and the number of repartitions includes:

[0089] Obtain a ratio of a data transmission speed of a distributed storage system to a data processing speed of the distributed storage system, wherein the ratio is the number of repartitions; obtain a second remainder by performing a modulo operation on the first storage partition and the number of repartitions, wherein the second remainder is a second storage partition of the key-value pair.

[0090] The calculation formula for the number of repartitions is as follows:

[0091] partitionNum = bps / fbps;

[0092] Among them, partitionNum is the number of repartitions, bps represents the number of bytes transmitted per second by the Tair cluster, that is, the data transmission speed of the distributed storage system, and fbps represents the number of bytes parsed and injected per second for a single file, that is, the data processing speed of the distributed storage system.

[0093] The calculation formula for the second storage partition is as follows:

[0094] partition=bucket%partitionNum;

[0095] Among them, partition is the second storage partition of the key-value pair, bucket is the first storage partition of the key-value pair, and partitionNum is the number of repartitions.

[0096] Specifically, see Figure 4 , Figure 4 is a schematic diagram of a calculation target storage partition provided by an embodiment of the present application, such as Figure 4As shown in the figure, Tair deploys 4 nodes: Node1(10), Node2(20), Node3(30) and Node4(50). The bucket numbers in brackets are the node numbers. The storage partition of "zhangsan" calculated according to the formula is 5. At this time, the storage node group with a bucket number greater than 5 is [Node1, Node2, Node3, Node4], and the node with the smallest bucket number is Node1. "zhangsan" is stored on Node1. The storage partition of "lisi" calculated according to the formula is 5. At this time, the storage node group with a bucket number greater than 5 is [Node1, Node2, Node3, Node4], and the node with the smallest bucket number is Node1. "lisi" is stored on Node1. The storage partition of "wangwu" calculated according to the formula is 15. At this time, the storage node group with a bucket number greater than 15 is [Node2, Node3, Node4], and the node with the smallest bucket number is Node2. "wangwu" is stored on Node2.

[0097] After that, each key-value pair is repartitioned according to the number of partitions and bucket numbers set in the Tair cluster. According to the calculation formula for the second storage partition, assuming that the number of repartitions is 1000, the storage partition of "zhangsan" is 5 after calculation, and the storage partition of "lisi" is also 5 after calculation. Then these two key-value pairs will be placed on the same physical node of the Tair server. Since the calculation results of the two are the same through bucket%1000, the two key-value pairs will be placed in the same partition for subsequent processing.

[0098] It can be seen that in this example, by ensuring that the key-value pairs with the same bucket number in the current task are repartitioned into the same partition according to the modulo operation, and subsequent logical processing is performed by the same execution node, it is beneficial to perform parallel calculations and operations on these related data, reducing network bandwidth consumption and transmission delays, thereby improving the overall efficiency of the system.

[0099] In another possible implementation, the calculation method of the number of repartitions may be a combination of one or more repartition calculation methods such as linear scaling calculation or fixed increment calculation. Specifically, the linear scaling calculation formula is as follows:

[0100] partitionNum=bucketSum*γ;

[0101] Among them, partitionNum is the number of repartitions, bucketSumt is the preset number of partitions, γ is the scaling factor, and γ is not 0;

[0102] The fixed increment calculation formula is as follows:

[0103] partitionNum=bucketSumt+α;

[0104] Among them, partitionNum is the number of repartitions, bucketSumt is the preset number of partitions, and α is the increment value;

[0105] It can be seen that in this example, using one repartitioning calculation method or combining multiple repartitioning calculation methods to determine the number of repartitions according to actual conditions is conducive to optimizing resource utilization and improving the operating efficiency of the distributed storage system.

[0106] S205: compress the key-value pair in the target storage partition to determine a storage space for the key-value pair.

[0107] In a possible implementation, the feature group configuration information further includes a feature number and a feature data type, and the compressing the key-value pair in the storage partition to determine the storage space of the key-value pair includes:

[0108] Use the first storage byte length to store the feature number of the key-value pair; determine whether the feature data type of the key-value pair is a fixed-length type based on the feature number; when it is detected that the feature data type is a fixed-length type, determine that the byte length corresponding to the feature data type is a second storage byte length; and obtain the storage space of the key-value pair by merging the first byte length and the second storage byte length.

[0109] The characteristic data is compressed and stored using binary encoding technology, and the value of the first storage byte length is a preset byte length, which can be modified by a client of the distributed storage system.

[0110] The feature data in different partitions are processed separately by different executors. In each partition, the executor will traverse the key-value pair data belonging to the current partition one by one, perform dynamic spatial calculation on the feature value set according to the feature group configuration information of the data model Domain, store the feature data using binary encoding technology, and perform aggregation processing according to the storage partition of each key-value pair. Specifically, the dynamic spatial calculation processing flow is as follows:

[0111] The executor traverses the key-value pair data in the current partition one by one, obtains the corresponding feature value set and the feature group configuration information of the data model Domain, and compresses the feature values ​​one by one.

[0112] Among them, see Figure 5 , Figure 5 is a structural diagram of a feature storage provided by an embodiment of the present application, such as Figure 5As shown, first obtain the feature ID featureId, and use 4 bytes to store the feature ID. Then obtain the feature data type datatype from the domain configuration according to the feature ID featureId. The feature data type datatype is a fixed-length type, and the feature value can be directly stored in a fixed-length space. For example, in the integer type Integer, byte is fixed to occupy 1 byte, short is fixed to occupy 2 bytes, int is fixed to occupy 4 bytes, and long is fixed to occupy 8 bytes.

[0113] For example, taking "wangwu" as an example, the feature type of last_3_pay_times is a fixed-length type INTEGER. At this time, 4 bytes are used to store its number value 1, and then 4 bytes are used to store its feature value 22. The feature type of last_3_pay_amount is a fixed-length type DOUBLE. 4 bytes are used to store its number value 2, and 8 bytes are used to store its feature value 320.21.

[0114] It can be seen that in this example, the storage space size of the key-value pair is determined by obtaining the data type according to the feature number and then dynamically determining the corresponding byte length according to the data type, which is conducive to more flexible and efficient use of memory resources.

[0115] In a possible implementation, the method further includes:

[0116] It is detected that the characteristic data type is a variable-length type, and a first storage byte length is used to store the value of the length of the key-value pair; the second storage byte length is obtained by combining the first storage byte length and the third storage byte length, wherein the third storage byte length is the length of the key-value pair.

[0117] The characteristic data is compressed and stored using binary encoding technology, and the value of the first storage byte length is a preset byte length, which can be modified by the client of the distributed storage system.

[0118] Among them, see Figure 6 , Figure 6 is a schematic diagram of another structure of feature storage provided by an embodiment of the present application, such as Figure 6 As shown, first obtain the feature number featureId, and use 4 bytes to store the feature number. Then obtain the feature data type datatype from the domain configuration according to the feature number featureId. The feature data type datatype is a variable-length type, such as a string string. The length of the feature value is not fixed, so first use 4 bytes to store the length of the feature value, and then use the bytes corresponding to the length of the feature value to store the feature value. The above process is compressed for all features under the key-value pair in turn.

[0119] For example, taking "wangwu" as an example, the last_point_product feature type is a variable-length type STRING, using 4 bytes to store its number value 3, using 4 bytes to store the feature value length 3, and using the bytes of the feature value length, that is, 3 bytes to store its feature value "GHD".

[0120] It can be seen that in this example, by determining the storage space size required for the key-value pair according to the variable-length attribute of the characteristic data type, the storage space size is dynamically determined, which is conducive to more flexible and efficient use of memory resources.

[0121] In a possible implementation manner, after determining the storage space of the key-value pair, the method further includes:

[0122] Determine whether there is a corresponding state synchronization transfer file under the storage space of the key-value pair; when it is detected that there is a corresponding state synchronization transfer file under the storage space of the key-value pair, inject the byte array of the key-value pair into the state synchronization transfer file in sequence; determine whether the file size of the state synchronization transfer file reaches the target threshold; when it is detected that the file size of the state synchronization transfer file reaches the target threshold, back up the state synchronization transfer file and upload it to the distributed storage system.

[0123] Specifically, see Figure 7 , Figure 7 This is a flowchart of constructing an SST file provided in an embodiment of the present application, such as Figure 7 As shown, after the value of each key-value pair is binary serialized and compressed into a byte array, the target storage partition corresponding to the key of the calculated key-value pair is aggregated, and the compressed byte array is injected into the local ordered string table (Sorted String Table, SST) file in order. After the local SST file size reaches the target threshold, the file is uploaded to the server of the distributed file system through the client of the distributed storage system. After the local SST file is backed up, the local SST file is deleted. At this time, it is determined whether all byte arrays have been stored in the SST file. If the traversal has been completed, the remaining SST files that have not reached the target threshold are uploaded to the distributed storage system together. After the local SST file is backed up, the local SST file is deleted. The target threshold can be adjusted according to actual needs. For example, the target threshold can be 8MB.

[0124] Among them, if the SST file does not exist locally, initialize an SST file. Specifically, initializing the SST file includes the following steps: create a new blank SST file in the specified directory or path to store key-value pair data; set some metadata information, the metadata information includes version number, creation time, size limit, index structure, etc.; in order to speed up the search and retrieval of key-value pairs, establish a corresponding index structure, specifically, the corresponding index structure can be established in the form of B+ tree, hash table, etc.; then set the configuration parameters related to the SST file, including compression algorithm selection, partition strategy, write strategy, etc.; finally, persist the initialized SST file to the disk to ensure that the data can be stored for a long time and can be accessed and used by subsequent operations.

[0125] For example, taking the above-mentioned "wangwu" as an example, the target threshold is 8MB, the value of the key-value pair is serialized and compressed into a byte array with a length of 27, and the calculated target storage partition is 25. According to the target storage partition, the value of the key-value pair of "wangwu" is serialized and compressed into a byte array and injected into the corresponding SST local file. After other data is written to the SST file and the file size reaches 8MB, the SST file is uploaded to the distributed file system, and then the local SST file is deleted after the SST file is backed up.

[0126] Among them, the client is deployed in the Tair storage engine to monitor the newly uploaded SST files in the distributed file system. Once there is a new SST file uploaded, Tair downloads the corresponding SST file from the distributed storage system and injects it into the data server dataServer of the storage engine. Among them, the data server is mainly used for data storage, that is, it is responsible for persisting key-value pair data on physical media (such as disks) and providing efficient read and write operation interfaces. The data server uses SST files as the persistence format of data and optimizes storage space and read and write performance through technical means such as merging and compression.

[0127] The specific steps are as follows:

[0128] Step 1: Concurrently download SST files from the distributed storage system. The data in each SST file belongs to the same target storage partition and the same physical node in the data server.

[0129] Step 2: Parse the SST file and inject the data in the file into the corresponding physical nodes in batches. Since the data in the SST file is ordered, the starting range of the key value in the SST file can be obtained. For example, the starting range of the key value in the SST1 file is 10-20, and the SST file size is fixed, so the file can be directly imported into the corresponding appropriate level.

[0130] For example, see Figure 8 , Figure 8 is a schematic diagram of SST file parsing provided in an embodiment of the present application, such as Figure 8 As shown, according to the file size of the downloaded file SST2, it is determined that both the L0 layer and the L1 layer have insufficient space to store the data of the corresponding file SST2, then the file will be directly imported into the L2 layer, avoiding the L0 layer data from being imported into the file SST2 and then merged to generate a new file, thereby further affecting the L1 layer and deeper layers, reducing the additional resource overhead generated by the merging process, wherein the above-mentioned L0 layer, L1 layer and L2 layer refer to the physical layer, data link layer and network layer respectively.

[0131] It can be seen that in this example, according to the target storage partition corresponding to the key of the calculated key-value pair, the compressed byte array is injected into the local SST file in order, which can achieve the aggregation of data in the same partition and maintain order during the writing process; at the same time, the target threshold is set to control the size of the local SST file. When the local SST file reaches the target threshold, the upload operation is triggered, which can avoid a single SST file being too large and causing the performance of the distributed storage system to deteriorate.

[0132] In one possible implementation, see Fig. 9 , Fig. 9 This is a schematic diagram of a feature reading method provided in an embodiment of the present application. When reading a feature, it is necessary to obtain the corresponding compressed feature value according to the dimension key, and then restore the feature value through a dynamic deserialization method. The specific feature reading steps are as follows:

[0133] Step 1: Get the target storage partition through the dimension key and the calculation formula of the first storage partition of the key-value pair determined according to the target hash value and the preset number of partitions, and obtain the byte array storing the corresponding feature value from the physical node of the corresponding data server according to the consistent hash calculation result. For example, assuming that the target storage partition is 25, it is necessary to retrieve the SST file for data storage starting from the L0 layer according to the key value range corresponding to each layer and each SST file, and then take out the corresponding compressed feature value from the SST file.

[0134] Step 2: According to the feature group configuration information of the data model domain, parse the byte array that stores the corresponding feature value bit by bit. First, read the first four bytes to obtain the corresponding feature number, and then obtain the feature data type from the configuration according to the feature number. If the feature data type is a fixed-length type, then obtain the feature value according to the fixed length of the feature value. For example, in the integer type Integer, byte is fixed to 1 byte, short is fixed to 2 bytes, int is fixed to 4 bytes, and long is fixed to 8 bytes. If the feature data type is a variable-length type, then read the last four bytes to obtain the feature value length, and then obtain the corresponding feature value according to the corresponding feature value length.

[0135] For example, taking "wangwu" as an example, the first 4 bytes are parsed to obtain the feature number value 1. According to the feature group configuration information of the data model domain, the last_3_pay_times feature data type is a fixed-length type INTEGER, and then the following 4 bytes are read to obtain the feature value 22. The parsing process of the last_3_pay_amount feature value is the same as the last_3_pay_times feature. For the last_point_product feature, the first 4 bytes are parsed to obtain the feature number value 3. According to the feature group configuration information of the data model domain, the last_point_product feature data type is a variable-length type STRING, so the following 4 bytes are read to obtain the feature value length 3, and finally the following 3 bytes are read to obtain its feature value "GHD".

[0136] It can be seen that in this example, through the above-mentioned feature parsing method, the entire parsing process only involves binary operations, and only necessary data is stored in the byte array, which can fully restore the feature value while ensuring performance, which is conducive to improving the stability of the distributed storage system.

[0137] See also Fig.10 , Fig.10 is a structural diagram of a feature storage device provided in an embodiment of the present application, such as Figure 5 As shown, the feature storage device 1000 includes:

[0138] The acquisition module 1001 is used to obtain a question-answer pair according to the input customer question and the pre-trained first question-answer model; and to obtain a question-answer pair classification result according to the question-answer pair and the pre-trained question-answer pair classification model, wherein the question-answer pair classification result includes a question that needs to be combined with a knowledge base search and a question that does not need to be combined with a knowledge base search;

[0139] The processing module 1002 is used to determine target answer information according to the question-answer pair and the classification result of the question-answer pair, and reply to the customer question based on the target answer information.

[0140] In one possible implementation, in terms of determining the target storage partition of the key-value pair according to the target hash value and the preset number of partitions, the processing module 1002 is specifically used to: determine a first storage partition of the key-value pair according to the hash value and the preset number of partitions; determine a target storage node of the key-value pair according to the first storage partition and multiple storage nodes; and determine a second storage partition of the key-value pair in the target storage node according to the first storage partition and the number of repartitions, the second storage partition being the target storage partition of the key-value pair.

[0141] In a possible implementation, in terms of determining the first storage partition of the key-value pair according to the hash value and the preset number of partitions, the processing module 1002 is specifically used to: obtain the hash value and the preset number of partitions of the key-value pair; obtain a first remainder by performing a modulo operation on the hash value of the key-value pair and the preset number of partitions, and the first remainder is the first storage partition of the key-value pair.

[0142] In a possible implementation, each of the multiple storage nodes corresponds to a storage partition, and in terms of determining the target storage node of the key-value pair based on the first storage partition and the multiple storage nodes, the processing module 1002 is specifically used to: sort the multiple storage nodes in order according to the storage partitions to obtain a storage node sequence; connect the storage node sequence end to end to obtain a storage node circular linked list; determine whether the first storage partition corresponds to a storage node in the storage node circular linked list; detect that the first storage partition corresponds to a storage node in the storage node circular linked list, and determine that the storage node corresponding to the first storage partition is the target storage node of the key-value pair; detect that the first storage partition does not correspond to a storage node in the storage node circular linked list, obtain the first storage node in the storage node circular linked list that is adjacent to the first storage partition and located in the clockwise direction of the first storage partition; determine that the first storage node is the target storage node of the key-value pair.

[0143] In a possible implementation, each storage node among the multiple storage nodes corresponds to a storage partition, and in terms of determining the target storage node of the key-value pair based on the first storage partition and the multiple storage nodes, the processing module 1002 is specifically used to: detect a storage node group in which the storage partitions among the multiple storage nodes are all smaller than the first storage partition; and determine the storage node with the smallest storage partition in the storage node group as the target storage node of the key-value pair.

[0144] In one possible implementation, in terms of determining the second storage partition of the key-value pair in the target storage node based on the first storage partition and the number of repartitions, the processing module 1002 is specifically used to: obtain the ratio of the data transmission speed of the distributed storage system to the data processing speed of the distributed storage system, where the ratio is the number of repartitions; obtain a second remainder by performing a modulo operation on the first storage partition and the number of repartitions, and the second remainder is the second storage partition of the key-value pair.

[0145] In a possible implementation, the feature group configuration information also includes a feature number and a feature data type. In terms of compressing the key-value pair within the storage partition and determining the storage space of the key-value pair, the processing module 1002 is specifically used to: use a first storage byte length to store the feature number of the key-value pair; determine whether the feature data type of the key-value pair is a fixed-length type based on the feature number; when it is detected that the feature data type is a fixed-length type, determine that the byte length corresponding to the feature data type is a second byte length; and obtain the storage space of the key-value pair by merging the first byte length and the second byte length.

[0146] In a possible implementation, the processing module 1002 is further used to: detect that the characteristic data type is a variable-length type, determine that the length of the key-value pair is a third byte length; and obtain the storage space of the key-value pair by merging the first byte length and the third byte length.

[0147] In a possible implementation, the processing module 1002 is also used to: obtain the target storage partition through the dimension key and the calculation formula of the first storage partition that determines the key-value pair according to the target hash value and the preset partition number, and obtain the byte array storing the corresponding feature value from the physical node of the corresponding data server according to the consistent hash calculation result; parse the byte array storing the corresponding feature value bit by bit according to the feature group configuration information of the data model domain, first read the first four bytes, obtain the corresponding feature number, and then obtain the feature data type from the configuration according to the feature number. If the feature data type is a fixed-length type, then obtain the feature value according to the fixed length of the feature value. If the feature data type is a variable-length type, then read the next four bytes to obtain the feature value length, and then obtain the corresponding feature value according to the corresponding feature value length.

[0148] It is worth noting that the specific functional implementation of the feature storage device 1000 is shown in the above Figure 1The description of the feature storage method shown in the figure, for example, the acquisition module 1001 is used to implement the relevant content of executing S201, and the processing module 1002 is used to implement the relevant content of executing S202-S205. Each unit or module in the feature storage device 1000 can be separately or completely combined into one or more other units or modules to form, or one (some) of the units or modules can be further divided into multiple smaller units or modules in function to form, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present invention. The above-mentioned units or modules are divided according to logical functions. In actual applications, the function of one unit (or module) is realized by multiple units (or modules), or the functions of multiple units (or modules) are realized by one unit (or module).

[0149] According to the description of the above method embodiment and related device embodiment, please refer to Fig.11 , Fig.11 It is a structural diagram of a computer 1100 provided in an embodiment of the present application. Fig.11 The computer 1100 shown includes a processor 1101, a memory 1102, a communication interface 1103 and a bus 1104. The processor 1101, the memory 1102 and the communication interface 1103 are connected to each other through the bus 1104.

[0150] Optionally, the memory 1102 is a ROM, a static storage device, a dynamic storage device or a RAM.

[0151] The memory 1102 can store executable program codes. When the executable program codes stored in the memory 1102 are executed by the processor 1101, the processor 1101 and the communication interface 1103 are used to execute the executable program codes. Figure 2 The various steps of the feature storage method of the illustrated embodiment.

[0152] The processor 1101 adopts a general-purpose CPU, a microprocessor, an application-specific integrated circuit ASIC, a GPU or one or more integrated circuits to execute relevant programs to execute the feature storage method of the method embodiment of the present application.

[0153] The processor 1101 can also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the feature storage method of the present application can be completed by the hardware integrated logic circuit or software instructions in the processor 1101. Optionally, the processor 1101 is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component. The processor can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor is a microprocessor or the processor is any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to execute, or a combination of hardware and software modules in the decoding processor to execute. The optional software module is located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the memory 1102, and the processor 1101 reads the information in the memory 1102, and combines its hardware to complete the functions required to be performed by a module included in a feature storage device 1000 of an embodiment of the present application, or executes the feature storage method of the method embodiment of the present application.

[0154] The communication interface 1103 uses a transceiver or other related device such as, but not limited to, a transceiver.

[0155] The bus 1104 may include a path for transmitting information between various components of the computer 1100 (eg, the memory 1102 , the processor 1101 , and the communication interface 1103 ).

[0156] It should be noted that although Fig.11 The computer 1100 shown only shows a memory, a processor, and a communication interface, but in the specific implementation process, those skilled in the art should understand that the computer 1100 also includes other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the computer 1100 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the computer 1100 may also include only the devices necessary for implementing the embodiments of the present application, and does not necessarily include Fig.11 All devices shown in .

[0157] An embodiment of the present application provides a computer-readable storage medium, in which a computer program for electronic data exchange is stored. The computer program includes execution instructions, and the execution instructions are used to execute part or all of the steps of any one of the feature storage methods recorded in the above-mentioned feature storage method embodiments. The above-mentioned computer includes an electronic client device.

[0158] An embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, and the computer program can be operated to enable a computer to perform part or all of the steps of any feature storage method recorded in the above method embodiment. The computer program product can be a software installation package.

[0159] It should be noted that, for the sake of simplicity, the embodiments of any of the aforementioned feature storage methods are described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the present application.

[0160] The above is a detailed introduction to the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of a feature storage method, device, computer and storage medium of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, based on the idea of ​​a feature storage method, device, computer and storage medium of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0161] The present application is described with reference to the flowcharts and / or block diagrams of the methods, hardware products, and computer program products of the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0162] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1The memory may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0163] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "one" does not exclude multiple situations. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0164] A person skilled in the art can understand that all or part of the steps in the various methods of the method embodiments of any of the above-mentioned feature storage methods can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, and the memory can include: a flash drive, a read-only memory (English: Read-Only Memory, abbreviated as: ROM), a random access memory (English: Random Access Memory, abbreviated as: RAM), a disk or an optical disk, etc.

[0165] It can be understood that any product that is controlled or configured to execute the processing method of the flowchart described in an embodiment of a feature storage method of the present application, such as the device and computer program product of the above flowchart, falls within the scope of the related products described in the present application.

[0166] Obviously, those skilled in the art can make various changes and modifications to the feature storage method, device, computer and storage medium provided by the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A feature storage method, characterized in that: The method comprises: Acquire feature group configuration information of the data set, wherein the feature group configuration information includes dimension key, feature name, feature number and feature data type; Obtaining a key-value pair by combining the dimension key and the value of the feature name corresponding to the dimension key; Performing hash calculation on the dimension key in the key-value pair to obtain a target hash value; Determine a target storage partition for the key-value pair according to the target hash value and a preset number of partitions; The key-value pair is compressed in the target storage partition according to the feature number and the feature data type to determine a storage space for the key-value pair.

2. The method according to claim 1, characterized in that Determining a target storage partition of the key-value pair according to the target hash value and a preset number of partitions includes: A first remainder is obtained by performing a modulo operation on the target hash value and the preset partition number, where the first remainder is a first storage partition of the key-value pair; Determine a target storage node for the key-value pair according to the first storage partition and a plurality of storage nodes; A second storage partition of the key-value pair is determined in the target storage node according to the first storage partition and the number of repartitions, and the second storage partition is a target storage partition of the key-value pair.

3. The method according to claim 2, characterized in that Each storage node in the plurality of storage nodes corresponds to a storage partition, and determining a target storage node of the key-value pair according to the first storage partition and the plurality of storage nodes includes: Sort the multiple storage nodes in order according to the storage partitions to obtain a storage node sequence; Connecting the storage node sequence head to tail to obtain a storage node circular linked list; Determining whether the first storage partition corresponds to a storage node in the storage node circular linked list; Detecting that the first storage partition corresponds to a storage node in the storage node circular linked list, determining that the storage node corresponding to the first storage partition is a target storage node for the key-value pair; It is detected that the first storage partition does not correspond to a storage node in the storage node circular linked list, and a first storage node in the storage node circular linked list that is adjacent to the first storage partition and located in a clockwise direction of the first storage partition is obtained; and the first storage node is determined to be a target storage node for the key-value pair.

4. The method according to claim 2, characterized in that Determining the second storage partition of the key-value pair in the target storage node according to the first storage partition and the number of repartitions includes: Obtaining a ratio of a data transmission speed of a distributed storage system to a data processing speed of the distributed storage system, wherein the ratio is the number of repartitions; A second remainder is obtained by performing a modulo operation on the first storage partition and the repartition number, where the second remainder is the second storage partition of the key-value pair.

5. The method according to claim 1, characterized in that The compressing the key-value pair in the storage partition to determine the storage space of the key-value pair includes: Using a first storage byte length to store the feature number of the key-value pair; Determine the characteristic data type of the key-value pair according to the characteristic number of the key-value pair; Determine the second storage byte length of the key-value pair according to a characteristic data type of the key-value pair, wherein the characteristic data type of the key-value pair includes a variable-length type and a fixed-length type; The storage space of the key-value pair is obtained by combining the first storage byte length and the second storage byte length.

6. The method according to claim 5, characterized in that The determining the second storage byte length of the key-value pair according to the characteristic data type of the key-value pair includes: When it is detected that the characteristic data type is a fixed-length type, determining the second storage byte length to be a byte length corresponding to the characteristic data type; When it is detected that the characteristic data type is a variable-length type, the first storage byte length is used to store the value of the length of the key-value pair; the second storage byte length is obtained by merging the first storage byte length and the third storage byte length, wherein the third storage byte length is the length of the key-value pair.

7. The method according to any one of claims 1 to 6, characterized in that: After determining the storage space of the key-value pair, the method further includes: Determine whether there is a corresponding state synchronization transmission file in the storage space of the key-value pair; It is detected that there is a corresponding state synchronization transmission file under the storage space of the key-value pair, and the byte array of the key-value pair is injected into the state synchronization transmission file in sequence; Determine whether the file size of the state synchronous transmission file reaches a target threshold; It is detected that the file size of the state synchronous transmission file reaches a target threshold, and the state synchronous transmission file is backed up and uploaded to a distributed storage system.

8. A feature storage device, characterized in that: The device comprises: An acquisition module, used to acquire feature group configuration information of a data set, wherein the feature group configuration information includes a dimension key, a feature name, a feature number, and a feature data type; A processing module, used to obtain a key-value pair by combining the dimension key and the value of the feature name corresponding to the dimension key; and used to perform hash calculation on the dimension key in the key-value pair to obtain a target hash value; and used to determine a target storage partition for the key-value pair based on the target hash value and a preset number of partitions; and used to compress the key-value pair within the target storage partition based on the feature number and the feature data type to determine the storage space for the key-value pair.

9. A computer, characterized in that: include: A memory, a processor, and an executable program code stored in the memory and executable on the processor, wherein the processor executes the steps of the feature storage method according to any one of claims 1 to 7 when executing the executable program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores executable program code, wherein the executable program code includes execution instructions, and the execution instructions are used to execute the steps of the feature storage method according to any one of claims 1 to 7.