A data processing method, device and storage medium

Abstract

The present disclosure provides a data processing method, device and storage medium. The method comprises: obtaining full update data corresponding to to-be-updated data; sequentially allocating shared memory blocks to each target file based on metadata files in the full update data; loading each target file into a corresponding shared memory block based on a correspondence between the block number of each allocated shared memory block stored in the shared memory area and the hash value of the corresponding target file and the metadata files in the full update data; and adjusting a pointer pointing to a current storage area of the to-be-updated data to point to each shared memory block in the shared memory area to implement updating of the to-be-updated data. The present disclosure solves the problems of high response delay and reduced reliability of the system during data updating in the related art.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a data processing method and device and storage medium. BACKGROUND

[0002] A KV (Key Value) system is a computing system for storing data by using keys, and generally stores a large amount of data to meet the needs of various data services.

[0003] In the related art, when updating data of the KV system in real time, a key-by-key updating manner is usually used, which is not only inefficient, but also causes great computing pressure to the system when the data update concurrency is large, thereby resulting in problems such as high response delay and reduced reliability of the system, which is not conducive to providing good data services to service requesters. SUMMARY

[0004] The present disclosure provides a data processing method, device and storage medium to solve at least one technical problem in the prior art.

[0005] In one aspect, the present disclosure provides a data processing method, characterized in that comprising:

[0006] When a data update request instruction is received, full update data corresponding to to-be-updated data is acquired; the full update data comprises a plurality of target files and metadata files corresponding to the plurality of target files, and different target files correspond to different hash values;

[0007] Based on the metadata files, shared memory blocks are sequentially allocated for each target file in a pre-created shared memory area; the shared memory area stores a correspondence between a block serial number of each allocated shared memory block and a hash value of the corresponding target file;

[0008] Based on the correspondence and the metadata files, each target file is loaded into the corresponding shared memory block;

[0009] A pointer to a current storage area of the to-be-updated data is adjusted to point to each shared memory block in the shared memory area, so as to realize updating of the to-be-updated data.

[0010] In another aspect, a data processing device is also provided, and the device comprises:

[0011] An acquisition module is configured to acquire full update data corresponding to to-be-updated data when a data update request instruction is received; the full update data comprises a plurality of target files and metadata files corresponding to the plurality of target files, and different target files correspond to different hash values;

[0012] an allocation module configured to sequentially allocate, based on the metadata file, shared memory blocks to each target file in a pre-created shared memory area; the shared memory area stores a correspondence between a block sequence number of each allocated shared memory block and a hash value of a corresponding target file;

[0013] a loading module configured to load each target file into a corresponding shared memory block based on the correspondence and the metadata file;

[0014] an adjusting module configured to adjust a pointer pointing to a current storage area of the to-be-updated data to point to each shared memory block in the shared memory area, so as to implement updating of the to-be-updated data.

[0015] In another aspect, a computer device is provided, which includes a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement any of the above-described data processing methods.

[0016] In another aspect, a computer readable storage medium is provided, which stores at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by a processor to implement any of the above-described data processing methods.

[0017] In another aspect, a computer program product or a computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to perform any of the above-described data processing methods.

[0018] The data processing method, device and storage medium provided by the present disclosure have the following technical effects:

[0019] The embodiment of the present disclosure acquires full update data corresponding to to-be-updated data when receiving a data update request instruction; the full update data includes a plurality of target files and metadata files corresponding to the plurality of target files, different target files correspond to different hash values; based on the metadata files, shared memory blocks are sequentially allocated for each target file in a pre-created shared memory area; the shared memory area stores a correspondence between the block serial number of each allocated shared memory block and the hash value of the corresponding target file; based on the correspondence and the metadata files, each target file is loaded into the corresponding shared memory block; a pointer pointing to the current storage area of the to-be-updated data is adjusted to point to each shared memory block in the shared memory area, so as to realize the update of the to-be-updated data. The present disclosure loads offline update data files into shared memory, notifies the KV system to switch the shared memory pointer to realize full update of the data, and the required update time is the time for loading the data file into the memory, which greatly shortens the update time. By loading the target file into the shared memory, block switching and one-time writing are realized. In addition, by dividing the data into a plurality of target files and updating each file, the requirement for system memory and the calculation pressure during the update are reduced, the update efficiency is high, and the query speed is also improved, solving the problems of slow update speed and large system pressure and system response delay during concurrent update in the related art. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, and the advantages thereof, a brief introduction will be given to the drawings needed in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0021] Figure 1 is a flowchart of a data processing method provided by an embodiment of the present disclosure;

[0022] Figure 2 is a structural diagram of a plurality of target files provided by an embodiment of the present disclosure;

[0023] Figure 3 is a flowchart of a step of allocating shared memory blocks for each target file provided by an embodiment of the present disclosure;

[0024] Figure 4 is a flowchart of a step of loading each target file into the corresponding shared memory block provided by an embodiment of the present disclosure;

[0025] Figure 5 is a flowchart of a step of realizing the update of the to-be-updated data provided by an embodiment of the present disclosure;

[0026] Figure 6 is a flow diagram of another data processing method provided by an embodiment of the present disclosure;

[0027] Figure 7 is a process diagram of a data processing method provided by an embodiment of the present disclosure;

[0028] Figure 8 is a flow diagram of a data query step using updated data provided by an embodiment of the present disclosure;

[0029] Figure 9 is a framework diagram of a data processing device provided by an embodiment of the present disclosure;

[0030] Figure 10 is a framework diagram of a data processing device provided by an embodiment of the present disclosure;

[0031] Figure 11 is a hardware structure diagram of an apparatus for implementing the method provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the present disclosure scheme, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, not all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present disclosure.

[0033] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the embodiments of the present disclosure will be described in further detail below in conjunction with the drawings.

[0034] Cloud computing refers to a delivery and use model of IT infrastructure where IT resources are accessed over the network as a service, and are available on demand, easily scalable, and manageable. Broadly, cloud computing refers to the delivery and use of services over the network as a service, and is available on demand, easily scalable, and manageable. Such services can be IT and software, Internet related, or other services. Cloud computing is the product of the development of grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, load balancing, and other traditional computer and network technologies.

[0035] With the development of the Internet, real-time data flow, and the diversification of connected devices, and the promotion of search services, social networks, mobile commerce, and open collaboration, cloud computing has rapidly developed. Unlike previous parallel distributed computing, the generation of cloud computing will revolutionize the entire Internet model and enterprise management model.

[0036] Cloud storage is a new concept that extends and develops from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that, through cluster applications, grid technologies, and distributed storage file systems, etc., collects and cooperatively works a large number of various types of storage devices (storage devices are also referred to as storage nodes) in the network through application software or application interfaces, and provides data storage and business access functions.

[0037] Currently, the storage method of the storage system is as follows: a logical volume is created, and when the logical volume is created, a physical storage space is allocated to each logical volume, which can be a disk of a storage device or a disk group of several storage devices. The client stores data on a certain logical volume, that is, stores data on the file system, the file system divides the data into many parts, each part is an object, and the object contains not only data but also additional information such as data identification (ID, ID entity). The file system writes each object to the physical storage space of the logical volume, and records the storage location information of each object, so that when the client requests to access the data, the file system can access the data according to the storage location information of each object.

[0038] The storage system allocates physical storage space to a logical volume, specifically: according to the capacity estimation of the object stored in the logical volume (the estimation often has a large margin relative to the actual capacity of the object to be stored) and the group of redundant arrays of independent disks (RAID, Redundant Array of Independent Disk), the physical storage space is divided into a section in advance, and a logical volume can be understood as a section, thereby allocating the physical storage space to the logical volume.

[0039] The names related to the present disclosure are explained as follows:

[0040] Data unit: refers to a memory unit composed of a key and at least one value. The memory size occupied by each data unit is equal to the sum of the memory sizes occupied by the key and at least one value.

[0041] Metadata: refers to system data used to describe the characteristics of a file.

[0042] Hash number (Hashnum): refers to the number of files after the original data is processed by a hash function.

[0043] Shared memory: refers to a physical space that allows multiple kernels to access.

[0044] The following introduces a specific embodiment of a data processing method of the present disclosure, and the execution subject of the method can be a server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms.

[0045] Figure 1 is a flowchart of a data processing method provided by an embodiment of the present disclosure, Figure 7 is a process diagram of a data processing method provided by an embodiment of the present disclosure. The present disclosure provides method operation steps as described in the embodiments or flowcharts, but more or fewer operation steps can be included based on conventional or non-creative labor. The order of steps listed in the embodiments is only one of the many execution orders, and does not represent the only execution order. For example, Figure 1 and 7 As shown in the method can include:

[0046] S101: When receiving a data update request instruction, obtaining full update data corresponding to the data to be updated; the full update data includes a plurality of target files and metadata files corresponding to the plurality of target files, and different target files correspond to different hash values.

[0047] The to-be-updated data refers to data in the KV system that needs to be updated, which is stored in the KV system in the form of key-value. Since the KV system stores a large amount of data, the large amount of data can be divided into multiple pieces of to-be-updated data according to service types, business types, regions, service areas, and the like. When updating a piece of data is triggered, it is usually desired to update all data in the piece of to-be-updated data to obtain full-amount updated data.

[0048] The full-amount updated data refers to data obtained by performing data preprocessing on first original data of the full-amount updated data corresponding to the to-be-updated data. Optionally, the data preprocessing includes, but is not limited to, hash operations, data sharding, and the like. The full-amount updated data can include multiple target files and metadata files corresponding to the multiple target files, where different target files correspond to different hash values.

[0049] For example only, after obtaining the first original data of the full-amount updated data corresponding to the to-be-updated data, a hash operation can be performed on the key of the first original data to divide the first original data into multiple target files and metadata files corresponding to the multiple target files. Each target file arranges multiple data units, and each data unit can be a memory unit composed of one key and at least one value. The metadata file can record the position of each target file, the format of the data unit, and the total number of target files. Of course, the content recorded in the metadata file includes but is not limited to this, for example, the number of data units, the total size of the target file, and the like.

[0050] For example, Figure 2 is a structural diagram of the multiple target files provided by the embodiment of the present disclosure. As shown in Figure 2 , the original data is subjected to a hash operation, and is divided into 100 parts (i.e., hashnum=100). The hash function can be key%100, thereby obtaining 100 target files. Optionally, the data units in each target file can be sorted according to the size of the key value, for example, arranged in ascending order of the key value. Each target file arranges multiple data units in a compact arrangement structure. As shown in Figure 2 , each data unit is composed of one key and two values, which is a memory unit. Each target file corresponds to a different hash value, for example, the first target file corresponds to hash=0, the second target file corresponds to hash=1, and the 100th target file corresponds to hash=99.

[0051] For example only, the metadata file corresponding to the multiple target files can be as shown in Figure 2The metadata file contents corresponding to the plurality of target files in the metadata file can at least include the following:

[0052] file: / data / 0.data fields:uint64,uint32,uint64 hashnum:100

[0053] file: / data / 1.data fields:uint64,uint32,uint64 hashnum:100

[0054] file: / data / 99.data fields:uint64,uint32,uint64 hashnum:100

[0055] The metadata file can sequentially record the location of each target file (such as file: / data / 0.data, etc.), the format of the data unit (such as fields:uint64, uint32, uint64, etc.), and the total number of target files (such as hashnum:100). Among them, fields:uint64, uint32, uint64, that is, the data type of 1 key is uint64, and the data types of 2 values are uint32 and uint64, respectively. The size and number of each data unit can be calculated through the format of the data unit. Taking the format of the data unit as fields:uint64, uint32, uint64 as an example, the key occupies sizeof(uint64)=8 bytes, the value1 occupies sizeof(uint32)=4 bytes, and the value2 occupies sizeof(uint64)=8 bytes. Therefore, one data unit occupies 8+4+8=20 bytes, and the number of data units=file total size / size of one unit.

[0056] S103: Based on the metadata file, sequentially allocate shared memory blocks for each target file in a pre-created shared memory area; the shared memory area stores a correspondence between the block number of each allocated shared memory block and the hash value of the corresponding target file.

[0057] Optionally, a shared memory area can be applied for allocation in advance, and the shared memory area can be divided into a plurality of shared memory blocks. The size and number of shared memory blocks can be pre-set, or can be determined according to the metadata file.

[0058] In an optional embodiment, as shown in Figure 3 the sequentially allocating shared memory blocks for each target file in the pre-created shared memory area based on the metadata file can include:

[0059] S301: Determine the size and the total number of the shared memory blocks to be divided based on the attribute information of each target file in the metadata file.

[0060] Optionally, the attribute information of the target file can include the format of the data unit and the total number of the target files, and can also include the number of data units, the total size of the target files, etc. According to the format of the data unit and the number of data units, or the total size of the target files recorded in the metadata file, the size of each shared memory block to be divided is determined. According to the total number N of the target files in the metadata file, the total number M of the shared memory blocks to be divided is determined. Each shared memory block corresponds to store a target file, and therefore M is greater than or equal to N. Of course, in order to save memory overhead, the values of M and N can be equal, and the size of each shared memory block can be no less than the size of the corresponding target file.

[0061] S303: Divide the pre-created shared memory area into a plurality of shared memory blocks according to the determined size and the total number of the shared memory blocks.

[0062] Optionally, after the size and the total number of each shared memory block are determined, the pre-created shared memory area can be divided into a plurality of shared memory blocks. The size of each shared memory block can be the same.

[0063] In addition, optionally, in addition to dividing a plurality of shared memory blocks in the shared memory area, a storage block can also be divided, which is used to store the correspondence between the target files and the corresponding shared memory blocks.

[0064] S305: Obtain the hash value of each target file, and sort the plurality of target files according to the hash value size.

[0065] Optionally, the plurality of target files can be sorted in descending order of the hash value. Alternatively, the plurality of target files can also be sorted in ascending order of the hash value. Of course, in other embodiments, the plurality of target files can also be sorted according to other preset hash value sorting rules.

[0066] S307: Based on the sorting result and the order of each shared memory block in the shared memory area, each target file is assigned a corresponding shared memory block in turn.

[0067] Optionally, when the shared memory blocks are divided in the shared memory area, each shared memory block can have a sequence, which can be a time sequence or a memory address position sequence, etc. Meanwhile, each shared memory block can be numbered according to the sequence, and each shared memory block corresponds to a block number. According to the sorting result of the plurality of target files and the sequence of the shared memory blocks, each target file is sequentially assigned a shared memory block corresponding to a sequence.

[0068] In some optional embodiments, each target file can be sequentially assigned at least one corresponding shared memory block based on the sorting result and the sequence of the shared memory blocks in the shared memory area. For example, each target file is assigned two shared memory blocks, or the number of shared memory blocks assigned to each target file is not completely the same, for example, according to a preset assignment rule. Illustratively, the preset assignment rule can be a periodic assignment rule, for example, the first target file is assigned 1 shared memory block, the second target file is assigned 2 shared memory blocks, the third target file is assigned 1 shared memory block, the fourth target file is assigned 2 shared memory blocks, and so on. Alternatively, the preset assignment rule can be an incremental or decremental assignment rule, for example, the number of shared memory blocks assigned to the last t target files is s more than the number of shared memory blocks assigned to the first t target files, where t and s are positive integers. Specifically, if t = 2 and s = 1, that is, the first target file and the second target file are each assigned 1 shared memory block, the third target file and the fourth target file are each assigned 2 shared memory blocks, the fifth target file and the sixth target file are each assigned 3 shared memory blocks, and so on. By sequentially assigning each target file at least one corresponding shared memory block, various types and sizes of data storage and updating can be adapted, and the flexibility of the data processing method can be improved.

[0069] S105: loading each target file into a corresponding shared memory block based on the correspondence and the metadata file.

[0070] The target file includes at least one data unit having a key-value pair structure, and each shared memory block can include a block number storage sub-area, a key value storage sub-area, and a numerical value storage sub-area.

[0071] In an optional embodiment, as shown in Figure 4 Loading each target file into a corresponding shared memory block based on the correspondence and the metadata file can include:

[0072] S401: determining the storage location of the target file to be loaded based on the attribute information of each target file in the metadata file;

[0073] S403: determining a block sequence number of a corresponding shared memory block based on the hash value of each target file and the correspondence;

[0074] S405: obtaining a block pointer of the shared memory block corresponding to the determined block sequence number;

[0075] S407: sequentially loading each data unit in each target file into a key value storage sub-area and a value storage sub-area in the corresponding shared memory block according to the storage location of the target file to be loaded, the block pointer, and the key size order of each data unit in each target file.

[0076] Optionally, the data corresponding to each data unit of the target file can be pulled through the storage location of the target file, and then each data unit of the target file is pointed to the corresponding shared memory block through the block pointer of the shared memory block, and the key and value in each data unit of each target file are sequentially loaded into the key value storage sub-area and the value storage sub-area in the corresponding shared memory block according to the key size order of each data unit in each target file, so that the data processing efficiency can be improved.

[0077] S107: adjusting the pointer pointing to the current storage area of the data to be updated to point to each shared memory block in the shared memory area, so as to realize the update of the data to be updated.

[0078] In an optional embodiment, as shown in Figure 5 and 7 adjusting the pointer pointing to the current storage area of the data to be updated to point to each shared memory block in the shared memory area, so as to realize the update of the data to be updated can include:

[0079] S501: obtaining at least one first pointer value of the pointer pointing to the current storage area of each data unit in the data to be updated;

[0080] S503: obtaining at least one second pointer value of the pointer pointing to each shared memory block in the shared memory area;

[0081] S505: adjusting the first pointer value of each data unit in the data to be updated to the corresponding second pointer value based on the hash value of each data unit in the data to be updated and the correspondence, so as to realize the update of the data to be updated.

[0082] The current storage area includes at least one of shared storage, local storage, disk storage, and cache storage. Optionally, the current storage area can also be cloud storage or other feasible storage.

[0083] The present disclosure adopts loading the offline update data file into shared memory, notifying the KV system to switch the shared memory pointer to realize full update of data, and the required time for updating is the time for loading the data file into memory, which greatly shortens the update time. By loading the target file into shared memory, block-by-block switching and one-time writing are realized. In addition, by dividing the data into multiple target files and updating them one by one, the requirement for system memory and the computing pressure during updating are reduced, the update efficiency is high, and it is also beneficial to improve the query speed, solving the problems of slow update speed during data updating and large system pressure and system response delay caused by concurrent updating in related technologies.

[0084] In an optional embodiment, after the pointer pointing to the current storage area of the data to be updated is adjusted to point to each shared memory block in the shared memory area, the method further comprises:

[0085] S109: When it is detected that all the first pointer data corresponding to all the data units in the data to be updated are adjusted to the second pointer values, the current storage area storing the data to be updated is released.

[0086] In the above embodiment, when it is detected that all the first pointer data corresponding to all the data units in the data to be updated are adjusted to the second pointer values, that is, all the pointer switching in the current storage area of the data to be updated is completed. Then, by releasing the current storage area storing the data to be updated, that is, releasing the original storage space of the data to be updated, memory leakage is avoided.

[0087] In an optional embodiment, as shown in Figure 6 The method can further comprise:

[0088] S601: Obtain the hash value of each target file in the full update data and a preset adjustment value, the preset adjustment value being a natural number;

[0089] S603: Perform the same size sorting on the hash value of each target file and the preset adjustment value;

[0090] S605: Take the sum of the hash value and the preset adjustment value of the target file with the same sorting sequence number to obtain an adjusted sequence number;

[0091] S607: Take the adjusted sequence number as the block sequence number of the shared memory block.

[0092] Correspondingly, based on the correspondence and the metadata file, loading each target file into the corresponding shared memory block comprises:

[0093] S609: Based on the block sequence number of the shared memory block, the preset adjustment value, and the metadata file, load each target file into the corresponding shared memory block.

[0094] The number of preset adjustment values can be the number of target files, and the values of the plurality of preset adjustment values can be different from each other or partially the same. The same size sorting refers to that the sorting manner of the hash value of each target file and the sorting manner of the preset adjustment value are the same, for example, both are sorted in descending order or both are sorted in ascending order, and the like.

[0095] In the above embodiment, the block sequence number of the shared memory block is determined by the preset adjustment value and the hash value of the target file, the hash value of the target file can be directly determined based on the block sequence number of the shared memory block and the preset adjustment value, the step of searching according to the correspondence between the block sequence number of each allocated shared memory block and the hash value of the corresponding target file is omitted, the time consumption of updating is reduced, and the data updating efficiency is further improved.

[0096] Figure 8 is a flowchart of a data query step using updated data provided by the embodiments of the present disclosure. Please refer to Figure 8 As shown in the figure, the method further comprises:

[0097] S801: When receiving a data query request instruction, extracting the key character of the to-be-queried data carried in the data query request.

[0098] Optionally, the terminal sends a data query request instruction to the server, and the key character of the to-be-queried data is carried in the data query request. The server extracts the key character of the to-be-queried data carried in the data query request in response to the data query request instruction. Then, the server can perform index query based on the key to feed back the query result to the terminal.

[0099] The terminal herein can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, and the like, but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication, and the present disclosure does not limit this.

[0100] S803: Determining a target shared memory block from the shared memory area according to the hash value corresponding to the key character and the correspondence.

[0101] The correspondence is a mapping relationship between the block sequence number of each allocated shared memory block and the hash value of the corresponding target file. After obtaining the hash value corresponding to the key character, the block sequence number of the shared memory block corresponding to the hash value corresponding to the key character can be determined according to the correspondence stored in the shared memory area, so as to find out the corresponding target shared memory block from the shared memory area according to the determined block sequence number.

[0102] Optionally, the determining the target shared memory block from the shared memory area according to the hash value corresponding to the key character and the correspondence relationship comprises:

[0103] S8031: determining the hash value corresponding to the key character according to the key of the key character by using a hash algorithm;

[0104] S8033: determining the block sequence number of the target shared memory block based on the hash value corresponding to the key character and the correspondence relationship;

[0105] S8035: obtaining the block pointer corresponding to the block sequence number and locating to the target shared memory block;

[0106] S8037: determining the target shared memory block from the shared memory area.

[0107] S805: taking the key corresponding to the key character as a query object to search in the target shared memory block and returning a search result of the data to be queried.

[0108] Optionally, after determining the target shared memory block, the server can perform a binary search in the target shared memory block according to the key corresponding to the key character of the data to be queried carried in the data query request. If a matching key is found, the value corresponding to the matching key is taken as the search result of the data to be queried, which is returned to the terminal and displayed on the interface of the terminal. If no matching key is found, an empty result is returned.

[0109] Regarding the binary search algorithm, the bsearch function of the C library is taken as an example.

[0110] void*bsearch(const void*key,const void*base,size_t nmem,size_t size,int(*comp)(cosnt void*,const void*))

[0111] key--a pointer to the element to be searched;

[0112] base--a pointer to the first object of the array to be searched, i.e., the starting address of the memory block;

[0113] nitems--the number of elements in the array pointed to by base, i.e., the number of data units;

[0114] size--the size of each element in the array, in bytes, i.e., the size of the data unit, which is 20 bytes in this example;

[0115] compar--a function used to compare two elements, that is, to compare the key field of the data unit.

[0116] In an optional embodiment, the data processing method provided by the embodiment of the present disclosure supports that the value corresponding to the key is an array, such as key1, value11, value12; key1, value13, value14, that is, the value corresponding to key1 is an array, the first element is value11, value12; and the second element is value13, value14. In a specific implementation, the target file is kept in a monotonically increasing order according to the key; when querying, after a binary search finds a data unit, it is determined whether the key of the previous data unit is also equal to the target key until the key is not equal to the target key, and then the first data unit with the key equal to the target key is found; then it is determined backward until the key is not equal to the target key, and then the last data unit with the key equal to the target key is found; in this way, all data units with the key equal to the target key are found, and an array is returned.

[0117] The above embodiment, when receiving a data query request instruction, extracts the key character of the data to be queried in the data query request, and determines the target shared memory block from the shared memory area based on the hash value corresponding to the key character, and then finds the search result corresponding to the key character in the target shared memory block. Thus, synchronous full data update and data query are realized, the real-time performance and query speed of data query are improved, and the amount of calculation during data query is reduced.

[0118] The following is an embodiment of the device of the present disclosure, which can be used to execute the method embodiments of the present disclosure. For details not disclosed in the device embodiments of the present disclosure, please refer to the method embodiments of the present disclosure.

[0119] Please refer to Figure 9 which shows a structure block diagram of a data processing device provided by an embodiment of the present disclosure. The device has the functions in the above method examples, which can be realized by hardware or corresponding software executed by hardware. The data processing device can include:

[0120] The acquisition module 910 is configured to, when receiving a data update request instruction, acquire full update data corresponding to to-be-updated data; the full update data includes a plurality of target files and metadata files corresponding to the plurality of target files, and different target files correspond to different hash values.

[0121] The allocation module 920 is configured to sequentially allocate, based on the metadata file, shared memory blocks for each target file in a pre-created shared memory area; and the shared memory area stores a correspondence between a block sequence number of each allocated shared memory block and a hash value of a corresponding target file.

[0122] The loading module 930 is configured to load, based on the correspondence and the metadata file, each target file into a corresponding shared memory block.

[0123] The adjustment module 940 is configured to adjust a pointer pointing to a current storage area of the to-be-updated data to a pointer pointing to each shared memory block in the shared memory area, so as to implement updating of the to-be-updated data.

[0124] In an optional embodiment, the allocation module 920 can include:

[0125] determining, based on attribute information of each target file in the metadata file, a size and a total number of to-be-divided shared memory blocks;

[0126] dividing, according to the determined size and total number of shared memory blocks, the pre-created shared memory area into a plurality of shared memory blocks;

[0127] obtaining a hash value of each target file, and performing hash value size sorting on the plurality of target files;

[0128] sequentially allocating, based on a sorting result and an order of each shared memory block in the shared memory area, a corresponding shared memory block for each target file.

[0129] In an optional embodiment, the target file includes at least one data unit having a key-value pair structure, and each shared memory block includes a block sequence number storage sub-area, a key value storage sub-area, and a value storage sub-area. The loading module 1030 can include:

[0130] determining, based on attribute information of each target file in the metadata file, a storage location of a to-be-loaded target file;

[0131] determining, based on a hash value of each target file and the correspondence, a block sequence number of a corresponding shared memory block;

[0132] obtaining a block pointer of the shared memory block corresponding to the determined block sequence number;

[0133] sequentially loading, according to the storage location of the to-be-loaded target file, the block pointer, and a key size order of each data unit in each target file, each data unit in each target file into the key value storage sub-area and the value storage sub-area of the corresponding shared memory block.

[0134] In an optional embodiment, the adjusting module 940 can comprise:

[0135] obtaining at least one first pointer value of a pointer of each data in the to-be-updated data pointing to the current storage area;

[0136] obtaining a plurality of second pointer values of pointers of each shared memory block in the shared memory area;

[0137] adjusting the first pointer value of each data in the to-be-updated data to a corresponding second pointer value based on the hash value of each data in the to-be-updated data and the corresponding relationship, so as to realize the update of the to-be-updated data.

[0138] In an optional embodiment, the apparatus can further comprise:

[0139] releasing the current storage area storing the to-be-updated data when it is detected that all the first pointer data corresponding to all the data in the to-be-updated data are adjusted to the second pointer values;

[0140] wherein the current storage area comprises at least one of shared storage, local storage, disk storage, and cache storage.

[0141] In an optional embodiment, the apparatus can further comprise:

[0142] obtaining a hash value of each target file in the full update data and a preset adjustment value, the preset adjustment value being a natural number;

[0143] performing the same size sorting on the hash value of each target file and the preset adjustment value;

[0144] taking the sum of the hash value and the preset adjustment value of the target file with the same sorting sequence number to obtain an adjusted sequence number;

[0145] taking the adjusted sequence number as a block sequence number of a shared memory block;

[0146] Correspondingly, the loading module can comprise:

[0147] loading each target file into a corresponding shared memory block based on the block sequence number of the shared memory block, the preset adjustment value, and the metadata file.

[0148] In an optional embodiment, as shown in Figure 10 the apparatus can further comprise:

[0149] an instruction receiving module 950, configured to extract key characters of to-be-queried data carried in a data query request when receiving a data query request instruction;

[0150] The target block determination module 960 is configured to determine a target shared memory block from the shared memory area according to the hash value corresponding to the key character and the correspondence relationship.

[0151] The query module 970 is configured to search the target shared memory block by taking the key corresponding to the key character as a query object, and return a search result of the data to be queried.

[0152] In an optional embodiment, the target block determination module can include:

[0153] The hash value corresponding to the key character is determined by using a hash algorithm according to the key of the key character.

[0154] The block sequence number of the target shared memory block is determined based on the hash value corresponding to the key character and the correspondence relationship.

[0155] The block pointer corresponding to the block sequence number is acquired, and the target shared memory block is located.

[0156] The target shared memory block is determined from the shared memory area.

[0157] The apparatuses provided in the above embodiments can execute the corresponding methods in the embodiments of the present disclosure, and have the corresponding function modules and beneficial effects of executing the methods. Technical details not described in the above embodiments can be referred to the methods provided in any of the embodiments of the present disclosure.

[0158] The embodiments of the present disclosure provide a computer device, which can include a processor and a memory. The memory stores at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the data processing method according to any one of the above method embodiments.

[0159] The embodiments of the present disclosure also provide a computer readable storage medium. The storage medium stores at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the data processing method according to any one of the above method embodiments.

[0160] The embodiments of the present disclosure also provide a computer program product or a computer program. The computer program product or the computer program includes computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium. The processor executes the computer instructions to enable the computer device to execute the data processing method according to any one of the above embodiments.

[0161] Further, Figure 11A hardware structure schematic diagram of a device for implementing the method provided by the embodiments of the present disclosure is shown, which can be a computer terminal, a mobile terminal or other device, and the device can also participate in constituting or containing the apparatus provided by the embodiments of the present disclosure. As shown in Figure 11 The computer terminal 11 can include one or more processors 112 (the processor 112 can include but is not limited to a microprocessor MCU or a programmable logic device FPGA processing device), a memory 114 for storing data, and a transmission device 116 for communication functions. In addition, it can also include a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. Those skilled in the art can understand that Figure 11 The structure shown is only a schematic, which does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 11 can include more or fewer components than those shown in Figure 11 or have a different configuration than that shown in Figure 11 .

[0162] It should be noted that the one or more processors 112 and / or other data processing circuits described above can be referred to as "data processing circuits" herein. The data processing circuit can be embodied in whole or in part as software, hardware, firmware or any other combination. In addition, the data processing circuit can be a single independent processing module, or all or part of any one of the other elements combined into the computer terminal 11 (or mobile device). As referred to in the embodiments of the present disclosure, the data processing circuit controls the processor (for example, the selection of the variable resistance terminal path connected to the interface).

[0163] The memory 114 can be used to store software programs and modules of application software, and program instructions / data storage devices corresponding to the method described in the embodiments of the present disclosure. The processor 112 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implements the above-mentioned neural network processing method. The memory 114 can include a high-speed random access memory, and can also include a non-volatile memory such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 114 can further include a memory remotely arranged with respect to the processor 112, which can be connected to the computer terminal 11 through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0164] The transmission device 116 is configured to receive or send data via a network. The network can include, for example, a wireless network provided by a communication provider of the computer terminal 11. In one example, the transmission device 116 includes a network interface controller (NIC) that can connect to other network devices through a base station to communicate with the Internet. In one example, the transmission device 116 can be a radio frequency (RF) module that is configured to communicate with the Internet wirelessly.

[0165] The display can be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 11 (or mobile device).

[0166] It should be noted that the above-mentioned sequence of the embodiments of the present disclosure is only for description, and does not represent the advantages and disadvantages of the embodiments. The above-mentioned specific embodiments of the present disclosure are described. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that in the embodiments and still achieve the desired result. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired result. In some embodiments, multi-task processing and parallel processing are possible or can be advantageous.

[0167] Each of the embodiments of the present disclosure is described in a progressive manner, and the same or similar parts of each embodiment can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, the device and server embodiments are described simply because they are basically similar to the method embodiments, and the relevant parts can be referred to the part of the method embodiments.

[0168] A person of ordinary skill in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by program instructing relevant hardware, and the program can be stored in a computer readable storage medium, such as read-only memory, magnetic disk or optical disk.

[0169] The above-mentioned is only the preferred embodiment of the present disclosure, and does not limit the present disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A data processing method, characterized in that, include: When a data update request instruction is received, the full update data corresponding to the data to be updated is obtained; the full update data includes multiple target files and metadata files corresponding to the multiple target files, and different target files correspond to different hash values; Based on the metadata file, shared memory blocks are sequentially allocated for each target file in a pre-created shared memory area; The shared memory area stores the correspondence between the block number of each allocated shared memory block and the hash value of the corresponding target file; Based on the correspondence and the metadata file, each target file is loaded into the corresponding shared memory block; The pointer that was pointing to the current storage area of ​​the data to be updated is adjusted to point to each shared memory block in the shared memory area, so as to realize the update of the data to be updated.

2. The method according to claim 1, characterized in that, The step of sequentially allocating shared memory blocks for each target file in a pre-created shared memory area based on the metadata file includes: Based on the attribute information of each target file in the metadata file, the size and total number of shared memory blocks to be divided are determined; Based on the determined size and total number of shared memory blocks, the pre-created shared memory region is divided into multiple shared memory blocks; Obtain the hash value of each target file, and sort the multiple target files by hash value size; Based on the sorting results and the order of the shared memory blocks in the shared memory area, corresponding shared memory blocks are allocated to each target file in turn.

3. The method according to claim 1, characterized in that, The target file includes at least one data unit with a key-value pair structure, and each shared memory block includes a block sequence number storage sub-area, a key-value storage sub-area, and a value storage sub-area; The step of loading each target file into the corresponding shared memory block based on the correspondence and the metadata file includes: Based on the attribute information of each target file in the metadata file, the storage location of the target file to be loaded is determined; Based on the hash value of each target file and the corresponding relationship, the block number of the corresponding shared memory block is determined; Get the block pointer of the shared memory block corresponding to the determined block number; Based on the storage location of the target file to be loaded, the block pointer, and the key size order of each data unit in each target file, each data unit in each target file is sequentially loaded into the key-value storage sub-area and the value storage sub-area in the corresponding shared memory block.

4. The method according to claim 1, characterized in that, The step of adjusting the pointer that points to the current storage area of ​​the data to be updated to point to each shared memory block in the shared memory area, so as to realize the update of the data to be updated, includes: Obtain at least one first pointer value from the pointers to the current storage area pointed to by each piece of data in the data to be updated; Obtain multiple second pointer values ​​that point to pointers to each shared memory block in the shared memory region; Based on the hash value of each data in the data to be updated and the corresponding relationship, the first pointer value of each data in the data to be updated is adjusted to the corresponding second pointer value to realize the update of the data to be updated.

5. The method according to claim 4, characterized in that, After adjusting the pointer that points to the current storage area of ​​the data to be updated to point to each shared memory block in the shared memory area, the method further includes: When it is detected that the first pointer data corresponding to all data in the data to be updated has been adjusted to the second pointer value, the current storage area storing the data to be updated is released. The current storage area includes at least one of shared storage, local storage, disk storage, and cache storage.

6. The method according to claim 1, characterized in that, The method further includes: Obtain the hash value and preset adjustment value of each target file in the full update data, wherein the preset adjustment value is a natural number; The hash values ​​and preset adjustment values ​​of each target file are sorted by the same size; The adjusted sequence number is obtained by summing the hash value of the target files with the same sorting number with the preset adjustment value. The adjusted sequence number will be used as the block sequence number of the shared memory block; Accordingly, loading each target file into the corresponding shared memory block based on the correspondence and the metadata file includes: Based on the block number of the shared memory block, the preset adjustment value, and the metadata file, each target file is loaded into the corresponding shared memory block.

7. The method according to claim 1, characterized in that, The method further includes: When a data query request instruction is received, the key characters of the data to be queried carried in the data query request are extracted; Based on the hash values ​​corresponding to the key characters and the corresponding relationships, the target shared memory block is determined from the shared memory area; The key corresponding to the key character is used as the query object, and the search is performed in the target shared memory block to return the search result of the data to be queried.

8. The method according to claim 7, characterized in that, The step of determining the target shared memory block from the shared memory region based on the hash value corresponding to the key character and the correspondence includes: Based on the key of the key character, the hash value corresponding to the key character is determined using a hash algorithm; Based on the hash value corresponding to the key character and the corresponding relationship, the block number of the target shared memory block is determined; Obtain the block pointer corresponding to the block number and locate the target shared memory block; The target shared memory block is determined from the shared memory region.

9. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire the full update data corresponding to the data to be updated when a data update request instruction is received; the full update data includes multiple target files and metadata files corresponding to the multiple target files, and different target files correspond to different hash values; The allocation module is used to sequentially allocate shared memory blocks for each target file in a pre-created shared memory area based on the metadata file; the shared memory area stores the correspondence between the block number of each allocated shared memory block and the hash value of the corresponding target file; The loading module is used to load each target file into the corresponding shared memory block based on the correspondence and the metadata file; The adjustment module is used to adjust the pointer that points to the current storage area of ​​the data to be updated to point to each shared memory block in the shared memory area, so as to realize the update of the data to be updated.

10. A computer storage medium, characterized in that, The computer storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the data processing method as described in any one of claims 1-8.

Images