Data processing method and device

Abstract

The embodiment of the invention provides a data processing method and device which are used for improving the storage efficiency of a hash table. The method comprises: when an element is stored in a storage position associated with a first storage position in a first hash table, determining a candidate storage position of a first element in the first hash table, the first storage position being a storage position indicated by a hash value of the first element, and the first element being an element to be stored in the first hash table; when the distance between the candidate storage position and a second storage position is larger than a threshold value, a tree structure is determined, and the second storage position is the last storage position associated with the first storage position; and inserting the first element into the tree structure.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of electronic technology, and in particular, to a data processing method and device. Background Art

[0002] A hash table is a widely used data table with high read and write efficiency. Storage devices can use hash tables to store data. For example, the storage device can calculate a hash value based on the keyword of the element to be stored (the element to be stored is, for example, a key-value (KV)) and a given hash function, and then store the element to be stored in the storage location indicated by the hash value. When performing data query, the storage device can calculate a hash value based on the given keyword and hash function, and then query the corresponding element at the storage location indicated by the hash value. In actual use, the storage space of the hash table is limited, and the storage locations indicated by the hash values ​​corresponding to different elements to be stored may be the same, that is, hash conflicts may occur. Summary of the invention

[0003] The embodiments of the present application provide a data processing method and device for improving the storage efficiency of a hash table.

[0004] In a first aspect, an embodiment of the present application provides a data processing method, the method comprising: when an element is already stored in a storage position associated with a first storage position in a first hash table, determining a candidate storage position of the first element in the first hash table, the first storage position being a storage position indicated by a hash value of the first element, and the first element being an element to be stored in the first hash table; when a distance between the candidate storage position and a second storage position is greater than a threshold, determining a tree structure, the second storage position being the last storage position associated with the first storage position; and inserting the first element into the tree structure.

[0005] In the embodiment of the present application, when storing the first element in the hash table, if the storage position associated with the first storage position has stored an element, a candidate storage position for storing the first element can be found from other storage positions in the hash table, thereby solving the problem of hash conflicts. And if the distance between the candidate storage position of the first element and the last storage position associated with the storage position indicated by the hash value of the first element is greater than a threshold value, it indicates that when storing the first element in the candidate storage position, it may be necessary to perform multiple access operations to access the candidate storage position, so a tree structure is determined, and the element to be stored in the first hash table is inserted into the tree structure, so that it is not necessary to remove the elements stored in the first hash table, which helps to improve the storage efficiency of the hash table.

[0006] In a possible implementation, the storage location indicated by the hash value of the element stored in each storage location associated with the first storage location is the first storage location. The actual storage locations of multiple elements whose hash values ​​indicate the same storage location are associated, so that when storing elements, the candidate storage location for the element to be stored can be quickly found based on the associated storage location. Compared with the method of detecting each storage location in turn during linear detection, the number of storage locations detected is reduced, and the storage efficiency of the elements is effectively improved.

[0007] In a possible implementation, the tree structure is determined at a third storage location, and the third storage location is any storage location associated with the first storage location. Determining the tree structure at any location associated with the first storage location can allow some elements to be stored in the hash table and some elements to be stored in the tree structure, which can reduce the probability of remapping. At the same time, since the storage locations of the hash table are continuous, storing some elements in the hash table without expanding the capacity of the hash table helps to improve the read and write efficiency of the hash table.

[0008] In a possible implementation, the method further includes: taking out the elements stored in all storage locations from the third storage location to the last storage location, and determining the tree structure at the third storage location based on the taken out elements. Taking out the elements stored in the tree structure from the original storage location can make the storage location available for storing other elements, which helps to improve the storage efficiency of the hash table.

[0009] In a possible implementation, the third storage location is the Mth storage location associated with the first storage location, and the method further includes: disassociating all storage locations from the M+1th storage location to the last storage location from the first storage location. Disassociating the storage location of the retrieved element from the first storage location allows the storage location to be used to store other elements, which helps to improve the storage efficiency of the hash table.

[0010] In a possible implementation, the association relationship between all storage locations from the M+1th storage location to the last storage location and the first storage location is removed, including: modifying the indication information corresponding to all storage locations from the third storage location to the last storage location, the indication information being used to indicate the next storage location. After the indication information corresponding to each storage location is modified, there is no information indicating the next storage location, that is, it can be considered that the subsequent storage locations have no association relationship with the first storage location.

[0011] In a possible implementation, the method further includes: storing a first identifier for indicating the tree structure in a third storage location, the third storage location being a storage location for determining the tree structure. After the tree structure is determined, an identifier for indicating the tree is stored in the storage location for determining the tree structure, so that when storing an element whose next hash value indicates the first storage location and determining a candidate storage location for the next element, the next element can be inserted into the tree based on the identifier without having to search for a candidate storage location in the hash table.

[0012] In a possible implementation, the method further includes: querying the first element in sequence according to the indication information corresponding to each storage location in the storage location associated with the first storage location; if the information stored in the location to be queried includes a first identifier for indicating the tree structure, querying the first element in the tree structure. When the location to be queried is the last storage location associated with the first storage location, determining whether the location to be queried includes the first identifier does not require checking whether the first identifier exists in each storage location, which helps to improve the efficiency of element search.

[0013] In a possible implementation, the first hash table is a secondary hash table. Data is stored in the secondary hash table, so that multiple threads can write or read. For example, if the storage locations indicated by the hash values ​​of two elements are not the same storage location, the threads used to insert or query the two elements can be parallel, which helps to improve the read and write efficiency of the hash table.

[0014] In a second aspect, a communication device is provided, which may be a device such as a server that can store and query data, or a chip or chip system used in a communication device. The communication device includes a communication interface and a processor, and optionally, a memory. The memory is used to store a computer program, and the processor is coupled to the memory and the communication interface. When the processor reads the computer program or instruction, the communication device executes the method provided in the first aspect.

[0015] According to a third aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium is used to store a computer program. When the computer program is run on a computer, the computer executes the method provided in the first aspect.

[0016] According to a fourth aspect, a computer program product is provided, comprising a computer program, wherein when the computer program is run on a computer, the computer is enabled to execute the method according to the first aspect.

[0017] In a fifth aspect, a chip system is provided, comprising a processor and an interface, wherein the processor is used to call and execute instructions from the interface so that the chip system implements the method described in the first aspect above.

[0018] For the beneficial effects of the second to fifth aspects mentioned above, refer to the beneficial effects of the first aspect and will not be repeated. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of an application scenario applicable to an embodiment of the present application;

[0020] Figure 2 A flowchart of a data processing method provided in an embodiment of the present application;

[0021] Figure 3 A schematic diagram of the structure of a hash table provided in an embodiment of the present application;

[0022] Figure 4 A flowchart for determining a candidate storage location for an element to be inserted provided in an embodiment of the present application;

[0023] Figure 5 An example of determining a candidate storage location for an element to be inserted provided in an embodiment of the present application;

[0024] Figure 6 A schematic diagram of determining a tree structure provided in an embodiment of the present application;

[0025] Figure 7 A schematic diagram of another tree structure determination provided in an embodiment of the present application;

[0026] Figure 8 A flowchart of another data processing method provided in an embodiment of the present application;

[0027] Fig. 9 A schematic diagram of the structure of a communication device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the embodiments of the present application will be further described in detail with reference to the accompanying drawings. Below, some terms or concepts in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0029] (1) Hash table: A hash table is a data structure that implements an associative array. A hash table can be used to store data, that is, to write the elements to be stored into the hash table through a write (put) operation. A hash table can also be used to query data, that is, to query the elements to be queried from the hash table through a read (get) operation. A hash table may include the following structural elements:

[0030] Item (entry), the element or data stored in the hash table is called an item (also called an entry);

[0031] Bucket: Each item in the hash table is hashed into a bucket. The hash value of the keyword is calculated using a hash function. The storage location of the keyword in the hash table can be found through the hash value. For example, for a cuckoo hash table, a bucket is an array set. The array can be (s, pointer), (key, pointer) or other organizational forms. Among them, pointer is used to indicate the storage location of the element corresponding to the keyword, and s represents the hash value calculated based on the keyword and the hash function. For example, s = h (key), h refers to the hash function. When storing an element in the cuckoo hash table, s1 is first calculated based on the keyword and h of the element, and then s1 is modulo-operated with the capacity of the cuckoo hash table. The result is the candidate storage location of the element in the cuckoo hash table. When a storage location in a bucket is occupied, the storage location is marked as occupied. When a storage location in a bucket is not occupied, the storage location is marked as free.

[0032] (2) Auxiliary space, including two parameters, delta0 and delta1, which are used to indicate the relative position relationship of the elements on the virtual chain.

[0033] delta0 is used to indicate the storage position of the first conflicting element on the virtual chain relative to the storage position indicated by the hash value of the element. For example, element a has been stored in position A, and the storage position indicated by the hash value of element a is position A, or the storage position indicated by the hash value of element A is a storage position before position A (for example, position B), and the storage position indicated by the hash value of element b is position A. Therefore, when inserting element b into the hash table, if a hash conflict occurs, element b is considered to be a conflicting element, and element b needs to be stored in other storage positions of the hash table. If element b is the first conflicting element, the storage position of element b relative to element a is indicated by delta0 of position A.

[0034] delta1 is used to indicate the storage position of other conflicting elements on the virtual chain, except for the first conflicting element, relative to the previous conflicting element. Taking the above example, if there is an element c, the storage position indicated by the hash value of element c is position A. When inserting element c into the hash table, a hash conflict occurs. In this case, element c is considered to be a conflicting element, and element c needs to be stored in other storage positions in the hash table. If element c is inserted into the hash table after element b is inserted, element c is considered to be the second conflicting element, and the storage position of element c relative to element b is indicated by delta1 of the storage position of element b.

[0035] In the embodiments of the present application, "multiple" refers to two or more than two. In view of this, in the embodiments of the present application, "multiple" can also be understood as "at least two". "At least one" can be understood as one or more, for example, one, two or more. For example, including at least one means including one, two or more, and there is no restriction on which ones are included. For example, including at least one of A, B and C, then A, B, C, A and B, A and C, B and C, or A and B and C may be included. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / ", unless otherwise specified, generally indicates that the previously associated objects are in an "or" relationship.

[0036] Unless otherwise specified, the ordinal numbers such as "first" and "second" mentioned in the embodiments of the present application are used to distinguish multiple objects and are not used to limit the order, timing, priority or importance of the multiple objects.

[0037] Please refer to Figure 1 , is a schematic diagram of an application scenario applicable to the embodiment of the present application, the scenario is a recommendation system scenario, the recommendation system includes a training server and at least one parameter server ( Figure 1 In the example, the recommendation system includes three parameter servers. Each parameter server includes at least two levels of hash tables, each level of hash tables may include multiple hash tables, and the hash tables can be used to store data. In the following embodiments, each parameter server includes two levels of hash tables as an example.

[0038] During the data storage phase, the training server may use a load balancing algorithm to determine the parameter server with the smallest load among the at least one parameter server (e.g. Figure 1The parameter server 1 shown in the figure) and the training server can generate a key-value pair (key-value, KV) based on the elements to be stored, and send the generated key-value pair to the parameter server 1. When receiving the key-value pair, the parameter server 1 can determine the storage location of the key-value pair in the hash table based on the keyword in the key-value pair, and store the value in the key-value pair to the storage location, wherein the hash table can be a primary hash table, or it can also be a secondary hash table. When the hash table is a secondary hash table, the parameter server 1 can first calculate the hash value 1 according to the hash function 1 and the keyword in the key-value pair, find the target storage location in the primary hash table according to the hash value 1, and the target storage location refers to the target hash bucket, and then find the secondary hash table. Then calculate the hash value 2 according to the hash function 2 and the keyword, find the storage location of the key-value pair in the secondary hash table according to the hash value 2, and store the value in the key-value pair to the storage location.

[0039] In the data search phase, the training server can use the aforementioned load balancing algorithm to determine the parameter server (for example, Figure 1 The parameter server 2 shown in the figure) sends the keyword corresponding to the element to be queried to the parameter server 2. When the parameter server 2 receives the keyword, it queries the value corresponding to the keyword in the hash table based on the keyword. If the parameter server 2 queries the value corresponding to the keyword, the parameter server 2 can send the value to the training server. If the parameter server 2 does not query the value corresponding to the keyword, the parameter server 2 ends the query operation and returns, sending a response message to the training server that the value corresponding to the keyword is not found.

[0040] When the parameter server stores elements to be stored, hash conflicts may occur. When a hash conflict occurs, one solution is to expand the hash table, that is, to increase the capacity of the hash table. After expanding the capacity of the hash table, all elements stored in the original hash table are remapped (rehash) to the expanded hash table, and then the elements to be stored are inserted into the expanded hash table. During remapping, because all elements in the original hash table need to be re-stored in the new hash table, it is necessary to recalculate the hash value of each element of all the elements, and then store each element in the new hash table at the storage location indicated by the hash value of each element. This process takes a long time and has low storage efficiency.

[0041] Another way to resolve hash conflicts is to store the conflicting elements in other storage locations in the hash table. For example, the hash values ​​corresponding to element 1 and element 2 both indicate the same storage location in the hash table, such as location 1, and element 1 has been stored in location 1, so when storing element 2, element 2 is stored in other storage locations in the hash table. However, for this solution, when looking for a storage location for storing element 2, since only fixed-spaced storage locations can be accessed each time, when the fixed-spaced storage location of location 1 has stored an element, it is necessary to revisit the next fixed-spaced storage location, resulting in a large number of memory accesses and low access efficiency. Therefore, under this solution, it is generally required that the distance between the storage location found and location 1 is less than or equal to a certain threshold. If the distance between the storage location found and location 1 is greater than the threshold, it is necessary to determine a storage location (for example, location 3) in the storage location whose distance from location 1 is less than the threshold, delete the element stored in location 3 (for example, element 3), store element 2 in location 3, and the distance between location 3 and location 1 is less than or equal to the threshold. In addition, it is necessary to search for a storage location for storing element 3 in the hash table again, and so on, until all elements to be stored and temporarily deleted elements are stored again in the hash table. This method may involve multiple elements being temporarily deleted and re-stored when storing an element, and the storage efficiency is low.

[0042] In view of this, an embodiment of the present application provides a data processing method. In this method, when the distance between the candidate storage position of the element to be stored in the first hash table is greater than a threshold, a tree structure is determined, and the element to be stored in the first hash table is inserted into the tree structure. In this way, there is no need to expand the capacity of the hash table or move out the data stored in the first hash table, which helps to improve the storage efficiency of the hash table.

[0043] Please refer to Figure 2 , is a flow chart of a data processing method provided in an embodiment of the present application. The method can be implemented by a storage device, and the storage device is, for example, Figure 1 The parameter server shown.

[0044] S201: When a storage location associated with a first storage location in a first hash table has stored an element, the storage device determines a candidate storage location of the first element in the first hash table.

[0045] The first storage location is the storage location indicated by the hash value of the first element, and the first element is the element to be stored in the first hash table. The storage device can use a given hash function (for example, h1) to calculate the hash value (for example, the first hash value) of the first element, and the storage location indicated by the first hash value is the first storage location. For example, the first element is a key-value pair, and the storage device can use a given hash function to calculate the first hash value (value1) of the first element based on the keyword (for example, key1) in the key-value pair.

[0046] The storage location indicated by the hash value of the element stored in each storage location associated with the first storage location is the first storage location. The storage location associated with the first storage location may include the first storage location, and may also include other storage locations in the first hash table. In the following embodiments, the storage location associated with the first storage location may be indicated by a first virtual chain, for example, the first element on the virtual chain is the first element of the first storage location indicated by the hash value.

[0047] Optionally, the storage location of the first element may be the first storage location or may not be the first storage location. For example, if the first storage location is occupied by an element whose hash value indicates another storage location before the first element is stored, the storage location of the first element is not the first storage location. If the first storage location is not occupied by an element whose hash value indicates another storage location before the first element whose hash value indicates the first storage location is stored, the storage location of the first element is the first storage location.

[0048] Optionally, the first hash table is a secondary hash table, and the storage device may first find the first hash table before executing S201. Figure 3 , the storage device can calculate a hash value (for example, a second hash value) according to a given hash function (for example, h2) and key1, and determine the storage location indicated by the second hash value in the second hash table. The second hash table is a first-level hash table, which is used to store the location information of the second-level hash table. The first-level hash table can be, for example, a linear table of HashEntry, and the second-level hash table can be, for example, a skiptree, which is a combination of a linear table and a tree structure. Then, the storage device can find the first hash table according to the location information stored in the storage location indicated by the second hash value in the second hash table.

[0049] After calculating the first hash value, the storage device can determine the first storage location indicated by the first hash value. And determine whether the storage location associated with the first storage location has stored an element. If the first storage location has stored an element, it indicates that the storage location associated with the first storage location has stored an element, and the storage device can determine the candidate storage location of the first element in the first hash table. For example, please refer to Figure 4 , which is a flowchart of determining a candidate storage location of a first element.

[0050] S1: The storage device calculates a first hash value based on key1, and determines a first storage location indicated by the first hash value.

[0051] S2: The storage device determines whether the first storage location is empty.

[0052] The storage device may obtain the tag of the first storage location (e.g., location 1), and determine whether the location 1 is empty according to the tag. For example, if the location 1 is marked as free, it indicates that the location 1 is empty, and the storage device may execute S3. If the location 1 is marked as occupied, it indicates that the location 1 is occupied, other elements have been stored in the location 1, and the location 1 is not empty, and the storage device may execute S4.

[0053] S3: The storage device determines that the first storage location is a candidate storage location for the first element.

[0054] S4: The storage device determines a candidate storage location of the first element based on the indication information corresponding to the storage location associated with the first storage location.

[0055] Taking the storage position indicated by the value of the auxiliary space as an example, the storage device can determine whether the value of delta0 corresponding to the position 1 in the auxiliary space is 0. If the value of delta0 corresponding to the position 1 is 0, the storage device can query the storage position marked as free (for example, position 2) from the first storage position downward, and determine the queried free storage position as the candidate storage position of the first element. If the delta0 corresponding to the position 1 is not 0, the storage device can obtain the value of delta0 corresponding to the position 1 and determine the storage position corresponding to the value (for example, position 3).

[0056] Optionally, before executing S4, the storage device may also determine whether the key of other elements stored in position 1 is the same as the key of the first element, that is, determine whether the key of the other element is key1. If the key of the other element is key1, the storage device updates the value of the other element stored in position 1 to the value of the first element (i.e., value1). It should be understood that each time the storage device queries a storage position, if the storage position already stores an element, the storage device may first determine whether the key of the element stored in the storage position is key1. If it is key1, the value stored in the storage position is updated to value1. For example, if the storage device queries position 3 based on the value of delta0 corresponding to position 1, the storage device may determine whether the key of the element stored in position 3 is key1. If the key of the element stored in position 3 is key1, the storage device updates the value in position 3 to the aforementioned value1.

[0057] If the key of the element stored in the storage position is not key1, the storage device can obtain the value of the parameter of the auxiliary space corresponding to the storage position (for example, the value of delta0 or delta1), and query the candidate storage position according to the obtained value. Taking the storage device querying position 3 according to the value of delta0 corresponding to position 1 as an example, the storage device can obtain the value of delta1 of position 3 and query the candidate storage position according to the value. For example, if the value of delta1 of position 3 is 0, it indicates that the element stored in position 3 is the last element on the first virtual chain. The storage device can query the storage position marked as free (for example, position 4) from position 3 downward, and determine position 4 as the candidate storage position. If the value of delta1 corresponding to position 3 is not 0, it indicates that the element stored in position 3 is not the last element on the first virtual chain. The storage device can determine the next storage position (for example, position 5) based on the value of delta1 corresponding to position 3, and obtain the value of delta1 corresponding to position 5, that is, loop the process performed by the storage device after determining position 3, until the storage device finds the storage position of the last element on the first virtual chain (for example, position a). For example, if delta1 corresponding to position a is 0, it indicates that the element stored in position a is the last element on the first virtual chain. After finding position a, the storage device can query downward from position a for a storage position marked as free (for example, position b), and determine position b as the candidate storage position.

[0058] Please refer to Figure 5 , Figure 5The capacity of the first hash table shown is 6, and 4 elements have been stored in the first hash table, namely (0, 12), (12, 23), (8, 34) and (9, 45).

[0059] For example, the first element is (7, 3), and the storage location indicated by the hash value of the first element calculated by the hash function is the storage location with the hash subscript 1. Figure 5 As shown, the storage location with hash subscript 1 has stored the element (12, 23), that is, the storage location with hash subscript 1 is not empty, and the value of delta0 corresponding to the storage location with hash subscript 1 is 0, indicating that there is no virtual chain corresponding to the storage location with hash subscript 1. The storage device starts to query the free storage location (that is, the storage location with hash subscript 4) downward from the storage location with hash subscript 1. Therefore, the storage device can determine that the storage location with hash subscript 4 is the candidate storage location for the first element.

[0060] For another example, the first element is (18, 20), and the storage location indicated by the hash value of the first element calculated by the hash table is the storage location with a hash index of 0. Figure 5 As shown, the storage location with hash subscript 0 has stored the element (0, 12), that is, the storage location with hash subscript 0 is not empty, and the delta0 corresponding to the storage location with hash subscript 0 is 1, indicating that there is a virtual chain corresponding to the storage location with hash subscript 0. The storage device can determine the storage location of the first element on the virtual chain based on the value of delta0. The distance between the storage location of the first element and the storage location with hash subscript 0 is 1, that is, the storage location of the first element is the storage location with hash subscript 1. Then, the storage device can obtain the value of delta1 corresponding to the storage location with hash subscript 1, as shown in Figure 5 As shown, the value of delta1 corresponding to the storage location with hash subscript 1 is 0, indicating that the element stored in the storage location with hash subscript 1 is the last element on the virtual chain. The storage device can query the free storage location (that is, the storage location with hash subscript 4) starting from the storage location with hash subscript 1, so the storage device can determine that the storage location with hash subscript 4 is the candidate storage location of the first element.

[0061] When the storage device determines the candidate storage position of the first element in the first hash table, if it is determined that there is a first virtual chain, the storage device can also determine the distance between the candidate storage position and the storage position (e.g., the second storage position) of the last element (e.g., the second element) on the first virtual chain. If the distance is greater than a threshold, S202 is executed; if the distance is less than or equal to the threshold, S203 is executed.

[0062] For example, the preset threshold is 1, if the hash subscript of the candidate storage position of the first element is 4, the hash subscript of the second storage position is 1, and the distance between the candidate storage position and the second storage position is 3, the storage device can determine that the distance between the candidate storage position and the second storage position is greater than the threshold, and the storage device can execute S202. If the hash subscript of the candidate storage position is 2, the hash subscript of the second storage position is 1, and the distance between the candidate storage position and the second storage position is 1, the storage device can determine that the distance between the candidate storage position and the second storage position is equal to the threshold, and the storage device can execute S203.

[0063] S202: The storage device determines a tree structure, and inserts the first element into the tree structure.

[0064] Determining a tree structure can also be called building a tree, and the tree can be a balanced tree, a binary tree, etc. This application does not limit the structure of the tree. Optionally, the storage device can build a tree in any storage location associated with the first storage location (for example, the third storage location). When building the tree, the storage device can take out the elements in all storage locations from the third storage location to the last storage location associated with the first storage location, and use the taken out elements to build a tree. That is, the storage device can build a tree based on some or all of the elements on the first virtual chain.

[0065] After the storage device takes out the element from the corresponding storage position, except for the third storage position where the tree is constructed, the storage positions of other taken-out elements are vacated, and the storage device can disassociate the other storage positions of the taken-out elements except the third storage position from the first storage position. For example, the third storage position is the Mth storage position among the storage positions associated with the first storage position, and the storage device can disassociate all storage positions from the M+1th storage position to the Nth storage position from the first storage position. Wherein, N refers to the total number of storage positions associated with the first storage position, that is, the Nth storage position is the last storage position associated with the first storage position.

[0066] Optionally, since the M+1th storage position is indicated by the indication information of the Mth storage position, and the Nth storage position is indicated by the indication information of the N-1th storage position, the storage device can remove the association relationship by modifying the indication information corresponding to the Mth storage position to the N-1th storage position. The indication information may be the aforementioned auxiliary space, and the storage device may modify the value of delta0 or delta1 corresponding to the Mth storage position to the N-1th storage position to 0; or, the indication information may also be a pointer, and the storage device may delete the pointer from the Mth storage position to the N-1th storage position. As for the first storage position to the M-1th storage position, since they are stored as elements and not stored in the tree, there is no need to modify the corresponding delta0 or delta1 values. It should be understood that the aforementioned indication information is only an example, and the indication information may also be other information that can indicate a position, and the embodiment of the present application does not limit the indication information.

[0067] For example, see Figure 6 and Figure 7 , two examples of building trees for storage devices. In both examples, Figure 5 The hash table and auxiliary space are shown as an example.

[0068] Please refer to Figure 6 , which is a schematic diagram of a storage device building a tree based on all elements on the first virtual chain.

[0069] by Figure 5 Taking the example shown as an example, the first element is (18, 20). When the storage device constructs a tree based on all elements on the first virtual chain, the storage location for constructing the tree is the storage location with a hash index of 0. At this time, the storage device can construct a tree based on the element (0, 12) stored in the storage location with a hash index of 0 and the element (12, 23) stored in the storage location with a hash index of 1. The constructed tree is as follows: Figure 6 As shown, Figure 6 In the figure, ptr0 is the storage address of the value of the element with key 0, ptr1 is the storage address of the value of the element with key 12, and ptr4 is the storage address of the value of the element with key 18. Since the first element is not stored in the tree when the tree is constructed, the first element is represented by a dotted line.

[0070] After the storage device builds a tree based on all elements on the first virtual chain, the storage device may also modify the value of the auxiliary space (ie, delta0 or delta1) corresponding to the storage location for storing each element. Figure 6As shown, after the storage location with hash subscript 0 builds the tree, there is no storage element in the storage location with hash subscript 1, so the storage device changes the value of delta0 corresponding to the storage location with hash subscript 0 to 0.

[0071] like Figure 7 As shown, it is a schematic diagram of a storage device constructing a tree based on some elements on the first virtual chain. Figure 7 Take the example of a storage device constructing a tree at the storage location of the last element of the first virtual chain.

[0072] by Figure 5 Taking the example shown as an example, the first element is (18, 20), and the storage location of the last element on the first virtual chain corresponding to the storage location with hash subscript 0 is the storage location with hash subscript 1. When the storage device constructs a tree at the storage location of the last element of the first virtual chain, the tree only includes the last element. For example, the storage device can be based on the element (12, 23) stored in the storage location with hash subscript 1. The constructed tree is as follows: Figure 7 As shown, Figure 7 In the figure, ptr1 is the storage address of the value of the element with key 12, and ptr4 is the storage address of the value of the element with key 18. Since the first element is not stored in the tree when the tree is constructed, the first element is represented by a dotted line.

[0073] Optionally, when the storage device constructs a tree based on some elements on the first virtual chain, the storage device may also modify the value of the auxiliary space (ie, delta1) corresponding to the storage location for storing the some elements. For example, please refer to Figure 7 , the element stored in the storage location with hash subscript 0 is not saved to the tree, but is saved in the storage location with hash subscript 1, and this storage location still needs to be indicated by the value of delta0 corresponding to the storage location with hash subscript 0. Therefore, the storage device does not modify the value of delta0 corresponding to the storage location with hash subscript 0.

[0074] Optionally, when constructing the tree, the storage device constructs the tree based on the storage rule of the tree. For example, the storage rule of the tree is to store elements with keywords smaller than the keyword of the element of the current node on the left side of the tree, and to store elements with keywords larger than the keyword of the element of the current node on the right side of the tree. Optionally, after constructing the tree, the storage device may also store a first identifier for indicating the tree in a third storage location. The first identifier is, for example, Figure 6 and Figure 7 ptr in.

[0075] S203: The storage device inserts the first element into the candidate storage location.

[0076] Optionally, if the candidate storage location is not the storage location indicated by the first hash value of the first element, that is, the candidate storage location is not the first storage location, then after the storage device stores the first element in the candidate storage location, it can also modify the value of delta0 or delta1 corresponding to the second storage location. For example, if the second storage location is the first storage location, that is, there are no other storage locations associated with the first storage location except the first storage location, then the storage device can modify the value of delta0 of the second storage location, and the storage location indicated by the modified value of delta0 is the candidate storage location. If the second storage location is not the first storage location, that is, there are other storage locations associated with the first storage location except the first storage location, then the storage device can modify the value of delta1 of the second storage location, and the storage location indicated by the modified value of delta1 is the candidate storage location.

[0077] In the above technical solution, when the distance between the candidate storage position of the element to be stored in the first hash table queried and the storage position of the last stored element is greater than a threshold, the storage device can build a tree and insert the element to be stored in the first hash table into the tree. In this way, memory can be applied for progressively without obtaining a new hash table, which effectively reduces the probability of rehashing and helps to improve the storage efficiency of using hash tables.

[0078] The previous section introduces the process of storing data on a storage device. The following section introduces the process of querying data on a storage device. Figure 8 , is a flowchart of another data processing method provided in an embodiment of the present application, which can be Figure 1 The storage device shown is implemented, or it can also be implemented by other storage devices. In the embodiment of the present application, the method is implemented by a storage device as an example, and the storage device may include a multi-level hash table. In the following embodiment, the element to be queried is stored in a secondary hash table as an example.

[0079] S801: The storage device calculates a third hash value based on the keyword of the element to be queried and the first hash function, determines a storage location indicated by the third hash value, and determines a target hash table based on location information stored in the storage location, wherein the target hash table is a hash table storing the element to be queried.

[0080] S802: The storage device calculates a fourth hash value based on the second hash function and the keyword, and determines a storage location indicated by the fourth hash value in the target hash table.

[0081] In the following embodiments, it is taken as an example that the storage location indicated by the fourth hash value in the target hash table is location i.

[0082] S803: The storage device determines whether the position i is empty.

[0083] If the position i is empty, it indicates that there is no stored element, that is, it indicates that the value corresponding to the keyword of the element to be queried is not found, and S804 can be executed at this time. If the position i is not empty, the storage device can execute S805.

[0084] S804: The storage device returns "empty".

[0085] After executing S804, the storage device ends the query operation.

[0086] S805: The storage device determines whether the keyword stored in the position i is the same as the keyword of the element to be queried. If they are the same, S806 is executed; if they are different, S807 is executed.

[0087] S806: The storage device obtains the value and returns the value.

[0088] After executing S806, the storage device ends the query operation.

[0089] S807: The storage device determines whether the value of delta0 corresponding to the position i is 0. If it is 0, execute S808. If not 0, execute S810.

[0090] S808: The storage device determines whether there is a tree at the position i. If so, S809 is executed; if not, S804 is executed.

[0091] Optionally, when the storage device determines that a tree exists at the position i, it can also obtain an element from the tree and calculate the hash value of the element based on the keyword of the element. If the storage location indicated by the hash value of the element is the same storage location as the storage location indicated by the hash value of the element to be queried, execute S809; if the storage location indicated by the hash value of the element is not the same storage location as the storage location indicated by the hash value of the element to be queried, execute S804.

[0092] S809: The storage device queries value based on the tree. If the value is found, S806 is executed; if not, S804 is executed.

[0093] S810: The storage device determines the next position j according to the value of delta0 corresponding to the position i.

[0094] S811: Determine whether the keyword stored in position j is the same as the keyword of the element to be queried. If they are the same, execute S806. If they are different, execute S812.

[0095] Wherein, j=i+delta0[i], delta0[i] is the value of delta0 corresponding to position i. For example, if position i is a storage position with a hash index of 1, delta0[i]=4, then position j is a storage position with a hash index of 5.

[0096] S812: The storage device determines whether the value of delta1 corresponding to the position j is 0. If it is 0, S813 is executed. If not 0, S814 is executed.

[0097] S813: The storage device determines whether there is a tree at the position j. If so, S809 is executed. If not, S804 is executed.

[0098] S814: The storage device determines the next position j according to the value of delta1 corresponding to the position j, and executes S811.

[0099] Wherein, j=j+delta1[j]. When the storage device executes S811-S814, the value of delta1 obtained is the value of delta1 corresponding to the current storage location. For example, when S811-S814 is executed for the first time, the storage device obtains the value of delta1 corresponding to the storage location with a hash index of 5. If the value of delta1 corresponding to the storage location with a hash index of 5 is 1, the storage device obtains the value of delta1 corresponding to the storage location with a hash index of 6 when S811-S814 is executed for the second time.

[0100] In the above technical solution, when the storage device queries for an element to be queried, since the distance between the storage locations of two adjacent numbers in the hash table is less than a threshold, it helps to improve the query efficiency of the storage device.

[0101] The following is a schematic diagram of the structure of a communication device provided in an embodiment of the present application. Fig. 9 As shown. The communication device 900 may be the parameter server mentioned above. The communication device 900 may include: one or more processors 901; one or more memories 902; a communication interface 903, and one or more computer programs 904, and the above-mentioned devices may be connected via one or more communication buses 905. The one or more computer programs 904 are stored in the above-mentioned memory 902 and are configured to be executed by the one or more processors 901, and the one or more computer programs 904 include instructions. For example, the above-mentioned instructions can be used to execute the relevant steps of the parameter server in the corresponding embodiment above. The communication interface 903 is used to realize the communication between the parameter server and other devices (such as a training server), for example, the communication interface may be a transceiver.

[0102] An embodiment of the present application also provides a computer program product, including a computer program. When the computer program is run on a computer, the computer is enabled to execute the method in the above embodiment.

[0103] An embodiment of the present application also provides a chip system, including a processor and an interface, wherein the processor is used to call and execute instructions from the interface so that the chip system implements the method in the above embodiment.

[0104] In the embodiments provided by the present application, the method provided by the embodiments of the present application is introduced from the perspective of the communication device as the execution subject. In order to implement the functions in the methods provided by the embodiments of the present application, the terminal device may include a hardware structure and / or a software module to implement the above functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. Whether a function of the above functions is executed in the form of a hardware structure, a software module, or a hardware structure plus a software module depends on the specific application and design constraints of the technical solution.

[0105] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0106] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0107] For the purpose of explanation, the foregoing description is described by reference to specific embodiments. However, the above exemplary discussion is not intended to be exhaustive, nor is it intended to limit the present application to the disclosed precise form. According to the above teachings, many modifications and variations are possible. The embodiments are selected and described in order to fully illustrate the principles of the present application and its practical application, so that other persons skilled in the art can make full use of the present application and various embodiments with various modifications suitable for the specific purpose contemplated.

Claims

1. A data processing method, characterized in that: The method comprises: When an element is already stored in a storage position associated with the first storage position in the first hash table, determining a candidate storage position of the first element in the first hash table, the first storage position being a storage position indicated by a hash value of the first element, and the first element being an element to be stored in the first hash table; determining a tree structure when the distance between the candidate storage location and a second storage location is greater than a threshold, the second storage location being the last storage location associated with the first storage location; Insert the first element into the tree structure.

2. The method according to claim 1, characterized in that The storage location indicated by the hash value of the element stored in each storage location associated with the first storage location is the first storage location.

3. The method according to claim 1 or 2, characterized in that The tree structure is determined in a third storage location, and the third storage location is any storage location associated with the first storage location.

4. The method according to claim 3, characterized in that The method further comprises: The elements stored in all storage locations from the third storage location to the last storage location are retrieved, and the tree structure is determined at the third storage location based on the retrieved elements.

5. The method according to claim 4, characterized in that The third storage location is an Mth storage location associated with the first storage location, and the method further includes: The association relationship between all storage locations from the M+1th storage location to the last storage location and the first storage location is released.

6. The method according to claim 5, characterized in that The association relationship between all storage locations from the M+1th storage location to the last storage location and the first storage location is removed, including: Modify the indication information corresponding to all storage positions in the third storage position to the last storage position, wherein the indication information is used to indicate the next storage position.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: A first identifier for indicating the tree structure is stored in a third storage location, and the third storage location is a storage location for determining the tree structure.

8. The method according to claim 6, characterized in that The method further comprises: querying the first element in sequence according to the indication information corresponding to each storage location in the storage locations associated with the first storage location; If the information stored in the to-be-queried location includes a first identifier for indicating the tree structure, the first element is searched in the tree structure.

9. The method according to any one of claims 1 to 8, characterized in that: The first hash table is a secondary hash table.

10. A data processing device, characterized in that: It includes a storage unit and a processing unit, wherein: The storage unit is used to store a first hash table, and the first hash table is used to store data; The processing unit is used to determine a candidate storage position of the first element in the first hash table when an element is already stored in a storage position associated with the first storage position in the first hash table, the first storage position being a storage position indicated by a hash value of the first element, and the first element being an element to be stored in the first hash table; determine a tree structure when a distance between the candidate storage position and a second storage position is greater than a threshold, the second storage position being the last storage position associated with the first storage position; and insert the first element into the tree structure.

11. The device according to claim 10, characterized in that The storage location indicated by the hash value of the element stored in each storage location associated with the first storage location is the first storage location.

12. The device according to claim 10 or 11, characterized in that The tree structure is determined in a third storage location, and the third storage location is any storage location associated with the first storage location.

13. The device according to claim 12, characterized in that The processing unit is further used for: The elements stored in all storage locations from the third storage location to the last storage location are retrieved, and the tree structure is determined at the third storage location based on the retrieved elements.

14. The device according to claim 13, characterized in that The third storage location is an Mth storage location associated with the first storage location, and the processing unit is further used to: The association relationship between all storage locations from the M+1th storage location to the last storage location and the first storage location is released.

15. The device according to claim 14, characterized in that The processing unit is specifically used for: Modify the indication information corresponding to all storage positions in the third storage position to the last storage position, wherein the indication information is used to indicate the next storage position.

16. The device according to any one of claims 10 to 15, characterized in that: The processing unit is further used for: A first identifier for indicating the tree structure is stored in a third storage location, and the third storage location is a storage location for determining the tree structure.

17. The device according to claim 15, characterized in that The processing unit is further used for: querying the first element in sequence according to the indication information corresponding to each storage location in the storage locations associated with the first storage location; If the information stored in the to-be-queried location includes a first identifier for indicating the tree structure, the first element is searched in the tree structure.

18. The device according to any one of claims 10 to 17, characterized in that: The first hash table is a secondary hash table.

19. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are called by the computer, the method according to any one of claims 1 to 9 is executed.

20. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 9.

Images