Data processing method, device, server and storage medium

Abstract

The present disclosure relates to a data processing method, device, server and storage medium, and belongs to the field of computer technology. The method includes: receiving a first data file and a second data file sent by a first server, the first data file includes multiple data, the second data file includes a data identifier and location indication information corresponding to each of the data, and the location indication information of the data is used to indicate the storage location of the data; loading the first data file into the memory; and associating and storing the data identifier and the location indication information corresponding to each of the data in the second data file into the memory. In the data recommendation process, the storage location of the data is first queried based on the data identifier, so as to obtain the data at the storage location in the first data file. In this way, data acquisition can be achieved without loading the data and the data identifier corresponding to the data into the memory one by one, thereby greatly improving the data loading efficiency.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a data processing method, device, server and storage medium. Background Art

[0002] With the development of computer technology, the application of data recommendation is becoming more and more extensive. In the related art, a large amount of data is stored in the database. The server obtains each piece of data from the database and loads each piece of data and the data identifier corresponding to each piece of data into the memory one by one. In the data recommendation process, the server selects the target data from the memory based on the data identifier corresponding to the target data. However, the amount of data in the database is large, and it takes a long time to load these data and the corresponding data identifiers one by one, resulting in low data loading efficiency of the server. Summary of the invention

[0003] The present disclosure provides a data processing method, device, server and storage medium, which improves the data loading efficiency. The technical solution of the present disclosure is as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, a data processing method is provided, the method comprising:

[0005] Receive a first data file and a second data file sent by a first server, wherein the first data file includes a plurality of data items, and the second data file includes a data identifier and location indication information corresponding to each of the data items, wherein the location indication information of the data is used to indicate a storage location of the data;

[0006] Loading the first data file into the memory;

[0007] The data identifier and the position indication information corresponding to each piece of the data in the second data file are associated and stored in the memory.

[0008] In some embodiments, the method further comprises:

[0009] Determine the target data identifier corresponding to the target data to be queried;

[0010] querying the location indication information associated with the target data identifier from the memory;

[0011] The target data at the storage location indicated by the location indication information is obtained from the first data file.

[0012] In some embodiments, the location indication information of any of the data includes a file identifier of a first data file where the data is located, an offset of the data in the first data file, and a length corresponding to the data;

[0013] The acquiring, from the first data file, data at the storage location indicated by the location indication information comprises:

[0014] The target data of the length is obtained from the first data file corresponding to the file identifier, starting from the address indicated by the offset.

[0015] In some embodiments, obtaining the target data at the storage location indicated by the location indication information from the first data file includes:

[0016] In a case where the queried location indication information includes a status identifier, the target data at the storage location indicated by the location indication information is acquired from the first data file, and the status identifier is used to indicate that the data is stored in the first data file.

[0017] In some embodiments, each piece of the location indication information is associated with a data identifier corresponding to the data and stored in a target memory block of the memory; the method further includes:

[0018] Get incremental data;

[0019] The incremental data and the data identifier corresponding to the incremental data are associated and stored in the target memory block.

[0020] In some embodiments, the method further comprises:

[0021] Determine the target data identifier corresponding to the target data to be queried;

[0022] Querying the location indication information associated with the target data identifier from the target memory block;

[0023] In the case where the position indication information corresponding to the target data identifier is not found, the target data corresponding to the target data identifier is obtained from the target memory block.

[0024] In some embodiments, determining the target data identifier corresponding to the target data to be queried includes:

[0025] Receiving a data acquisition request sent by a user terminal, and acquiring the target data identifier carried in the data acquisition request; or receiving a recommendation request sent by a user terminal, and acquiring user information carried in the recommendation request, and determining the target data identifier associated with the user information;

[0026] The method further includes: sending the target data to a user terminal.

[0027] In some embodiments, loading the first data file into the memory includes:

[0028] In response to the target service being started, the first data file is loaded into the memory, and the target service is used for performing data recommendation.

[0029] In some embodiments, the associating and storing the data identifier and the location indication information corresponding to each piece of the data in the second data file in the memory includes:

[0030] Map the second data file into the memory.

[0031] According to a second aspect of an embodiment of the present disclosure, a data processing method is provided, the method comprising:

[0032] Writing multiple data in the database into the first data file;

[0033] generating location indication information for each piece of the data, wherein the location indication information is used to indicate a storage location of the data;

[0034] Writing the data identifier corresponding to each piece of the data and the location indication information of the data into a second data file in an associated manner;

[0035] The first data file and the second data file are sent to a second server.

[0036] In some embodiments, generating location indication information of each piece of data includes:

[0037] Determine a file identifier corresponding to a first data file into which each piece of the data is written, an offset of the data in the first data file, and a length of the data;

[0038] The file identifier, the offset and the length are determined as location indication information of the data.

[0039] In some embodiments, after generating the location indication information of each piece of the data, the method further includes:

[0040] storing the data identifier corresponding to the data in association with the location indication information of the data in a memory;

[0041] The step of associating the data identifier corresponding to each piece of the data and the location indication information of the data into the second data file comprises:

[0042] The data identifier and the location indication information corresponding to each piece of the data stored in the memory are written into the second data file.

[0043] In some embodiments, writing the plurality of data in the database into the first data file comprises:

[0044] Acquire the plurality of pieces of data from the database, and associate each piece of data with a data identifier corresponding to the data and store them in a memory;

[0045] Each piece of data stored in the memory is written into the first data file.

[0046] In some embodiments, storing each piece of the data in association with a data identifier corresponding to the data in a memory includes:

[0047] Performing hash calculation on the data identifier corresponding to each piece of data to obtain a data index;

[0048] The data index and the data are associated and stored in the memory.

[0049] In some embodiments, writing each piece of data stored in the memory into the first data file includes:

[0050] Map each piece of data stored in the memory to the first data file.

[0051] According to a third aspect of an embodiment of the present disclosure, a data processing device is provided, the device comprising:

[0052] A file receiving unit is configured to receive a first data file and a second data file sent by a first server, wherein the first data file includes a plurality of data items, and the second data file includes a data identifier and location indication information corresponding to each of the data items, wherein the location indication information of the data is used to indicate a storage location of the data;

[0053] A first loading unit, configured to load the first data file into a memory;

[0054] The first loading unit is further configured to execute the storage of the data identifier and the position indication information corresponding to each piece of the data in the second data file in association with each other in the memory.

[0055] In some embodiments, the apparatus further comprises:

[0056] An identifier determining unit, configured to determine a target data identifier corresponding to the target data to be queried;

[0057] An information query unit, configured to query the location indication information associated with the target data identifier from the memory;

[0058] The first acquisition unit is configured to acquire, from the first data file, target data at a storage location indicated by the location indication information.

[0059] In some embodiments, the location indication information of any of the data includes a file identifier of a first data file where the data is located, an offset of the data in the first data file, and a length corresponding to the data;

[0060] The first acquisition unit is configured to acquire the target data of the length from the first data file corresponding to the file identifier, starting from the address indicated by the offset.

[0061] In some embodiments, the first acquisition unit is configured to execute, when the queried location indication information includes a status identifier, acquiring target data at the storage location indicated by the location indication information from the first data file, wherein the status identifier is used to indicate that the data is stored in the first data file.

[0062] In some embodiments, each piece of the position indication information is associated with a data identifier corresponding to the data and stored in a target memory block of the memory; the device further comprises:

[0063] A second acquisition unit is configured to acquire incremental data;

[0064] The second loading unit is configured to execute the storage of the incremental data and the data identifier corresponding to the incremental data in the target memory block in association with each other.

[0065] In some embodiments, the apparatus further comprises:

[0066] An identifier determining unit, configured to determine a target data identifier corresponding to the target data to be queried;

[0067] An information query unit is configured to query the location indication information associated with the target data identifier from the target memory block;

[0068] The data acquisition unit is configured to acquire the target data corresponding to the target data identifier from the target memory block when the location indication information corresponding to the target data identifier is not found.

[0069] In some embodiments, the identifier determination unit is configured to execute receiving a data acquisition request sent by a user terminal, and obtain the target data identifier carried in the data acquisition request; or, the identifier determination unit is further configured to execute receiving a recommendation request sent by a user terminal, obtain user information carried in the recommendation request, and determine the target data identifier associated with the user information;

[0070] The device further includes: a data sending unit configured to send the target data to a user terminal.

[0071] In some embodiments, the first loading unit is configured to load the first data file into the memory in response to starting a target service, and the target service is used for data recommendation.

[0072] In some embodiments, the first loading unit is configured to execute mapping of the second data file into the memory.

[0073] According to a fourth aspect of an embodiment of the present disclosure, a data processing device is provided, the device comprising:

[0074] A data writing unit, configured to write a plurality of data in the database into a first data file;

[0075] an information generating unit, configured to generate location indication information of each piece of the data, wherein the location indication information is used to indicate a storage location of the data;

[0076] The data writing unit is further configured to write the data identifier corresponding to each piece of the data and the position indication information of the data into the second data file in an associated manner;

[0077] The file sending unit is configured to send the first data file and the second data file to the second server.

[0078] In some embodiments, the information generation unit is configured to determine the file identifier corresponding to the first data file to which each piece of the data is written, the offset of the data in the first data file, and the length of the data; and determine the file identifier, the offset, and the length as location indication information of the data.

[0079] In some embodiments, the apparatus further comprises:

[0080] An information storage unit, configured to associate a data identifier corresponding to the data with the location indication information of the data and store them in a memory;

[0081] The data writing unit is configured to write the data identifier and the location indication information corresponding to each piece of the data stored in the memory into the second data file.

[0082] In some embodiments, the data writing unit is configured to obtain the multiple data from the database, associate each data with a data identifier corresponding to the data and store it in a memory; and write each data stored in the memory into the first data file.

[0083] In some embodiments, the data writing unit is configured to perform a hash calculation on a data identifier corresponding to each piece of the data to obtain a data index; and store the data index and the data in association in the memory.

[0084] In some embodiments, the data writing unit is configured to map each piece of data stored in the memory to the first data file.

[0085] According to a fifth aspect of an embodiment of the present disclosure, a server is provided, the server comprising:

[0086] processor;

[0087] a memory for storing instructions executable by the processor;

[0088] The processor is configured to execute the instructions to implement the data processing method as described in the above aspects.

[0089] According to the sixth aspect provided by the embodiment of the present disclosure, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by the processor of the server, the server can execute the data processing method as described in the above aspects.

[0090] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, comprising instructions, which, when executed by a processor, implement the data processing method described in the above aspects.

[0091] In the embodiment of the present disclosure, the received first data file is directly loaded into the memory, and the location indication information of each data and the data identifier corresponding to the data are associated and stored in the memory. Therefore, in the data recommendation process, the storage location of the data can be queried based on the data identifier, and the data can be obtained at the storage location in the first data file. In this way, data acquisition can be achieved without loading the data and the data identifier corresponding to the data into the memory one by one, thereby greatly improving the data loading efficiency.

[0092] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0094] Figure 1 It is a schematic diagram of an implementation environment according to an exemplary embodiment.

[0095] Figure 2 The invention is a flowchart of a data processing method according to an exemplary embodiment.

[0096] Figure 3 The figure is a flowchart of another data processing method according to an exemplary embodiment.

[0097] Figure 4 is a flowchart of yet another data processing method according to an exemplary embodiment.

[0098] Figure 5 It is a schematic diagram showing a data processing process according to an exemplary embodiment.

[0099] Figure 6 is a flowchart of yet another data processing method according to an exemplary embodiment.

[0100] Figure 7 It is a block diagram of a data processing device according to an exemplary embodiment.

[0101] Figure 8 is a block diagram of another data processing device according to an exemplary embodiment.

[0102] Fig. 9 It is a block diagram of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0103] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0104] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned figure descriptions are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0105] The terms "at least one", "plurality", "each" and "any" used in this disclosure, at least one includes one, two or more, multiple includes two or more, and each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0106] Figure 1is a schematic diagram of an implementation environment provided by an embodiment of the present disclosure. Figure 1 , the implementation environment includes a first server 101 and a second server 102. The first server 101 and the second server 102 are connected via a wireless or wired network.

[0107] Among them, the first server 101 is used to write multiple data obtained from the database into a data file and send the data file to the second server 102. The second server 102 is used to receive the data file and load the data file into the memory to provide data recommendation services for the user terminal based on the multiple data in the data file.

[0108] In some embodiments, the first server 101 is a background server of an application or a cloud server that provides services such as cloud computing and cloud storage, and the second server 102 is a background server of an application or a cloud server that provides services such as cloud computing and cloud storage. It should be noted that the number of the first server 101 and the second server 102 can be one or more, and the embodiments of the present disclosure do not limit this.

[0109] The data processing method provided by the embodiments of the present disclosure can be applied to data recommendation scenarios. For example, taking multimedia data as an example, the first server obtains multiple multimedia data from a database, writes the multiple multimedia data into a data file, sends the data file to the second server, the second server receives the data file and loads it into a memory, and recommends the multimedia data to the user terminal based on the data file.

[0110] Figure 2 is a flow chart of a data processing method according to an exemplary embodiment, the method is executed by a first server, such as Figure 2 As shown, the following steps are included.

[0111] 201. A first server writes multiple pieces of data in a database into a first data file.

[0112] A large amount of data is stored in the database. The data is data related to the object. Optionally, the data includes multimedia data, for example, the data includes the name, image or video of the object. The object can be any type of object, such as daily necessities or office supplies.

[0113] 202. The first server generates location indication information for each piece of data, where the location indication information is used to indicate a storage location of the data.

[0114] 203. The first server writes the data identifier corresponding to each piece of data and the data location indication information into the second data file in an associated manner.

[0115] 204. The first server sends the first data file and the second data file to the second server.

[0116] The second server is used to receive the first data file and the second data file, and load the first data file and the second data file into the memory, so as to recommend data to the user terminal based on the first data file and the second data file.

[0117] In the disclosed embodiment, multiple pieces of data are written into a first data file, and the data identifier and location indication information corresponding to each piece of data are written into a second data file, so that after receiving the first data file and the second data file, the second server can first determine the storage location of the data based on the location indication information during the data recommendation process, and then obtain the data at the storage location in the first data file. In this way, data acquisition can be achieved without loading the data and the data identifiers corresponding to the data into the memory one by one, thereby greatly improving the data loading efficiency.

[0118] Figure 3 is a flowchart of another data processing method according to an exemplary embodiment, the method is executed by the second server, such as Figure 3 As shown, the following steps are included.

[0119] 301. A second server receives a first data file and a second data file sent by a first server, wherein the first data file includes multiple data items, and the second data file includes a data identifier and location indication information corresponding to each data item, wherein the location indication information of the data is used to indicate a storage location of the data.

[0120] The multiple data included in the first data file are obtained by the first server from the database. The first data file and the second data file are both binary files.

[0121] 302. The second server loads the first data file into memory.

[0122] 303. The second server associates and stores the data identifier and the location indication information corresponding to each piece of data in the second data file in the memory.

[0123] In the embodiment of the present disclosure, the received first data file is directly loaded into the memory, and the location indication information of each data and the data identifier corresponding to the data are associated and stored in the memory. Therefore, in the data recommendation process, the storage location of the data can be queried based on the data identifier, and the data can be obtained at the storage location in the first data file. In this way, data acquisition can be achieved without loading the data and the data identifier corresponding to the data into the memory one by one, thereby greatly improving the data loading efficiency.

[0124] Figure 4 is a flow chart of another data processing method according to an exemplary embodiment. This common embodiment is described by taking the data interaction between the first server and the second server as an example. Figure 4 As shown, the following steps are included.

[0125] 401. The first server writes multiple data in a database into a first data file.

[0126] There are multiple data stored in the database. In some embodiments, when a user needs to promote an item, he or she enters the item data in the user terminal, and the user terminal sends the data to the server, which receives the data and stores it in the database. The server may be the first server, or the server may be another server other than the first server for storing data in the database, which is not limited in the embodiments of the present disclosure.

[0127] In some embodiments, the implementation method of the first server writing multiple data in the database into the first data file includes: the first server obtains multiple data from the database, associates each data with the data identifier corresponding to the data and stores it in the memory; and writes each data stored in the memory into the first data file.

[0128] The data identifier corresponding to the data is determined based on the data. Optionally, for each piece of data, the first server extracts a data segment from the data and uses the data segment as the data identifier of the data. The data segment can be a data segment of any length in the data.

[0129] After obtaining multiple pieces of data from the database, the first server can store the multiple pieces of data on the disk, and when it is necessary to send data to the second server, the multiple pieces of data on the disk are loaded into the memory. Optionally, the first server filters the multiple pieces of data obtained from the database and loads the filtered data into the memory. Since a large amount of data is stored in the database, and the second server may not need to use all the data in the database when providing data recommendation services, the first server can first obtain the full amount of data from the database. After obtaining the full amount of data in the database, the obtained data can be filtered to obtain multiple pieces of data for data recommendation services, thereby improving the accuracy of the data.

[0130] In the embodiment of the present disclosure, the acquired data is stored in memory in association with a data identifier corresponding to the data, so that the processor can acquire data from the memory and write the data into a first data file, thereby providing data support for data processing of the processor.

[0131] Optionally, the data and the data identifier corresponding to the data are stored in memory in association with each other in the form of a hash table. Accordingly, the implementation method of the first server storing each piece of data in association with the data identifier corresponding to the data in memory includes: the first server performs a hash calculation on the data identifier corresponding to each piece of data to obtain a data index; and stores the data index in association with the data in memory.

[0132] The hash table stores data in the form of key-value pairs, where the data index is the key and the data is the value. It should be noted that the first server does not directly store the data in the memory, but parses and converts the data to obtain the value, which is the data structure supported by the hash table, so that it can be stored in the memory.

[0133] Optionally, the hash table is a HashMap. HashMap can be implemented in a closed hash (open addressing method) hash manner, so as to facilitate the subsequent use of the persistent storage capability of HashMap to implement data export and import. For example, the hash table is implemented in a sparse hash (a hash implementation) or libcuckoo value (a hash implementation) structure.

[0134] The data stored in the memory (value.data) is stored in the heap memory, that is, the address on the heap, and the heap memory is used to store objects and arrays created (new).

[0135] In the embodiments of the present disclosure, a data index is obtained by performing a hash operation on a data identifier corresponding to the data, so that the data index can uniquely identify the data. The data index and the data are associated and stored in memory, so that the data can be acquired according to the data index when acquiring the data, thereby improving the convenience of data acquisition.

[0136] In some embodiments, the first server writes each piece of data stored in the memory into the first data file in an implementation manner including: the first server maps each piece of data stored in the memory to the first data file. Alternatively, the first server may also serialize each piece of data stored in the memory and store the obtained byte stream in the first data file.

[0137] Optionally, the first server obtains each piece of data stored in the memory through the traversal interface.

[0138] In the embodiment of the present disclosure, data is exported by mapping the data stored in the memory to the first data file, and data transmission is feasible by converting the data structure of the data stored in the memory into a data structure that can be transmitted.

[0139] 402. The first server generates location indication information for each piece of data, where the location indication information is used to indicate a storage location of the data.

[0140] Since the first server writes each piece of data into the first data file, in this case, the first server can record the storage location of each piece of data in the first data file to obtain the location indication information of the data. In some embodiments, the first server generates the location indication information of each piece of data in an implementation manner including: the first server determines the file identifier corresponding to the first data file to which each piece of data is written, the offset of the data in the first data file, and the length of the data; and determines the file identifier, offset, and length as the location indication information of the data.

[0141] The file index is used to distinguish different first data files, for example, the first data file A has different file indexes from the second data file B. In the embodiment of the present disclosure, the first server obtains a large amount of data from the database, while the first data file can store a limited amount of data, so the first server writes the multiple data into at least one first data file.

[0142] If the number of first data files is one, it means that multiple data are written into the same first data file, and the file identifier corresponding to each data is the same. If the number of first data files is multiple, it means that multiple data are written into different first data files. For example, the multiple data include data 1, data 2, data 3, data 4, data 5 and data 6, where data 1, data 2 and data 3 are written into first data file A, and data 4, data 5 and data 6 are written into first data file B.

[0143] The disclosed embodiment does not limit the amount of data that can be stored in the first data file. For example, the amount of data that can be stored in the first data file is 4GB (Gibibyte). If in step 401, multiple data are written into 128 first data files, the storage space occupied by the multiple data in the memory is 128×4GB=512GB.

[0144] The offset of data in the first data file refers to the starting storage position of the data in the first data file, and the length of data refers to the size of the storage space occupied by the data in the first data file. For example, if the first data file is a binary file and the data in the first data file exists in the form of a byte stream, the offset of data 1 can be expressed in the form of the number of bytes moved forward or backward from the specified position of the first data file, and the length of the data can be expressed in bytes. The embodiment of the present disclosure does not limit the setting of the offset, that is, it does not limit the setting of the specified position of the first data file. For example, the offset of data can be expressed as a number of bytes moved backward from the beginning of the first data file.

[0145] In the embodiment of the present disclosure, the location indication information of the data is determined by using the file identifier corresponding to the first data file to which the data is written, the offset of the data in the first data file, and the length of the data. Thus, even if the data cannot be queried through the data identifier in the first data file, the data can be queried through the location indication information, thereby providing data support for data query.

[0146] In some embodiments, after generating the location indication information of each piece of data, the data processing method provided by the embodiment of the present disclosure further includes the following steps: the first server associates the data identifier corresponding to each piece of data with the location indication information of the data and stores it in the memory. The data identifier corresponding to the data and the location indication information of the data can be associated and stored in the memory in the form of a hash table.

[0147] Since the data and the data identifier corresponding to the data are stored in association in the memory of the first server, when the data and the data identifier corresponding to the data are stored in association in the memory in the form of a hash table, the first server can use the hash table to construct a hash table for storing the location information of the data.

[0148] Optionally, the first server copies the first hash table that stores the data and the data identifier corresponding to the data in an associated manner to obtain a second hash table; for any piece of data, the data in the second hash table is replaced with the location indication information of the data. In this way, an updated second hash table can be obtained, in which the data identifier and the location indication information corresponding to the data are stored in an associated manner. The process of implementing the associated storage of the data identifier corresponding to the data and the location indication information of the data in this way can save the time spent on storing the location indication information and improve the efficiency of the associated storage.

[0149] Optionally, the first server directly replaces the data in the first hash table with the location indication information of the data, thereby obtaining an updated first hash table, and the updated first hash table stores the data identifier and location indication information corresponding to the data in an associated manner. In this way, by directly replacing the data in the hash table without copying the first hash table, memory storage resources can be saved.

[0150] It should be noted that, each time the first server generates a piece of location indication information of data, the first server may store the location indication information in the memory.

[0151] In some embodiments, the data location indication information further includes a status flag (is static), and the status flag is used to indicate that the data is stored in the first data file.

[0152] For example, the server stores the offset, length, file identifier, and status identifier contained in the location indication information in the memory through the union keyword in C language, where union is used for union declaration and union variable definition. The server can also set the memory byte size occupied by each information contained in the location indication information through the alignas keyword, for example, alignas(8), which means that the occupied byte size is 8 bytes.

[0153] 403. The first server writes the data identifier corresponding to each piece of data and the data location indication information into the second data file in an associated manner.

[0154] In some embodiments, the implementation method of the server writing the data identifier corresponding to each piece of the data and the location indication information of the data into the second data file in an associated manner includes: the server writes the data identifier and the location indication information corresponding to each piece of the data stored in the memory in an associated manner into the second data file.

[0155] Accordingly, the first server maps the data identifier and data location indication information corresponding to each piece of data stored in the memory to the second data file. Alternatively, the server may also serialize the data identifier and data location indication information corresponding to each piece of data stored in the memory and store the obtained byte stream in the second data file.

[0156] In the disclosed embodiment, the data identifier and the location indication information corresponding to the data are stored in association in the memory, so that the data identifier and the location indication information can be written in association into the second data file, thereby ensuring the relevance of the information in the second data file.

[0157] In some other embodiments, the first server directly writes each location indication information into the second data file, that is, the second data file does not contain the data identifier corresponding to the data. Optionally, the implementation method of the first server writing each location indication information into the second data file is the same as the implementation method of the first server writing the data identifier corresponding to each piece of data and the location indication information of the data into the second data file in an associated manner, which will not be repeated here.

[0158] 404. The first server sends the first data file and the second data file to the second server.

[0159] 405. The second server receives the first data file and the second data file sent by the first server.

[0160] 406. The second server loads the first data file into memory.

[0161] In some embodiments, after receiving the first data file, the second server first stores the first data file in a disk, and only loads the first data file into a memory when the target service is started. Accordingly, the implementation method of the second server loading the first data file into the memory includes: the second server loads the first data file into the memory in response to the start of the target service, and the target service is used for data recommendation.

[0162] The embodiment of the present disclosure does not limit the method for starting the target service. For example, the target service is started when the second server is powered on or restarted, or a technician can manually set the start of the target service.

[0163] In the embodiment of the present disclosure, when the first data file is received, the second server may not currently use the data in the first data file, and there is no need to immediately load the first data file into the memory to save memory storage resources; and the target service for data recommendation is started, indicating that the second server starts to provide data recommendation service to the user terminal. Since the target service needs to use the data in the first data file, the first data file can be loaded into the memory at this time to achieve data restoration.

[0164] Optionally, the second server loads the first data file into an array (Attach_Data), which stores the memory address of the first data file. If there are multiple first data files, the second server loads the first data files into the array one by one. For example, the first data file is loaded into the array through mmap (a file mapping function) or fread (a data reading function). Among them, fread (bufer, file, size ()) means reading data from the first data file file into the array buffer, and size is the number of bytes of each data item to be read, in bytes.

[0165] 407. The second server associates and stores the data identifier and the location indication information corresponding to each piece of data in the second data file in the memory.

[0166] In some embodiments, the implementation method in which the second server associates and stores the data identifier and location indication information corresponding to each piece of data in the second data file in the memory includes: the second server maps the second data file to the memory.

[0167] Since the second data file stores the data identifier and location indication information corresponding to each piece of data, the second server can directly map the second data file to the memory in a file mapping manner. Optionally, the location indication information and the data identifier corresponding to the data are stored in the memory in an associated manner in the form of a hash table.

[0168] In the embodiment of the present disclosure, the second data file is obtained by mapping the data identifier and location indication information associated with the storage in the memory by the first server. By performing file mapping on the second data file, the original data identifier and location indication information can be obtained, thereby realizing data restoration, and greatly improving the data restoration efficiency by file mapping.

[0169] In other embodiments, when the second data file only contains the location indication information but does not contain the data identifier corresponding to the data, if the location indication information in the second data file is directly loaded into the memory, since the data in the memory is stored in a random storage manner, the storage address of the location indication information loaded into the memory is discontinuous, which is not convenient for the subsequent acquisition of the location indication information. Then, the second server associates the data identifier and the location indication information corresponding to each piece of data in the second data file and stores them in the memory in association, including: deserializing the second data file to obtain each piece of location indication information; for each piece of location indication information, obtaining a data segment from the location indication information and determining the data segment as a data identifier; and storing the location indication information and the data identifier in association in the memory.

[0170] The implementation manner in which the second server associates the location indication information with the data identifier and stores it in the memory is similar to the implementation manner in which the first server associates the data with the data identifier corresponding to the data and stores it in the memory in step 401, and will not be repeated here.

[0171] It should be noted that the embodiment of the present disclosure does not limit the execution order of step 405 and step 406.

[0172] Another point to be explained is that the first server can periodically obtain data from the database, that is, periodically perform the operations of step 401-step 403, so that the second server can perform the operations of step 405-step 406 based on the first data file and the second data file newly sent by the first server. Among them, since the user terminal may upload data at any time, the data in the database will be updated, so the first server periodically obtains data from the database so that the second server can receive the updated data in time to improve the accuracy of the data recommendation service.

[0173] The data processing method provided by the disclosed embodiment provides the ability to quickly load hundreds of millions of data, allowing the second server to complete the loading of data in a shorter time, thereby greatly improving the efficiency of data restoration and shortening the service startup time. The storage structure designed by the data processing method provided by the disclosed embodiment enables the first server to export the data stored in the memory offline as a first data file, so that when the second server provides online data recommendation services, it can load the first data file into the memory for data restoration without loading data one by one, thereby reducing the time for data loading and achieving the effect of reducing the data loading time from hours to minutes.

[0174] For example, see Figure 5 , the first server acts as a production module, obtains multiple data from the database, obtains a data set, reads the data one by one and inserts them into the hash table in the memory, obtains the data stored in the form of key-value pairs (data identifier-data), then traverses the hash table, exports the data, obtains the first data file (Attach File), and records the storage location of each data in the first data file in the hash table to obtain location indication information, traverses the hash table to export each associated stored data identifier and location indication information to obtain the second data file, and the first server sends the first data file and the second data file to the second server. Among them, the first data file and the second data file are both binary files to facilitate data transmission. Continue to see Figure 5 The second server acts as a loading module, loads the received first data file into the array in the memory, and restores the second data file into a hash table storing data identification and location indication information in the memory.

[0175] It should be noted that the data length of each piece of data in the embodiment of the present disclosure may be different, and the value stored in the hash table in the memory of the first server is variable in length. In this case, by writing multiple pieces of data into the first data file, and writing the data identifier and location indication information corresponding to the data into the second data file, the first data file and the second data file are sent to the second server, and then the second server can directly load the first data file and the second data file into the memory, for example, map them into the memory in the form of file mapping, then even if the data length of the data is variable in length, the data can be obtained with the help of the location indication information, so that the data processing method can adapt to more data processing scenarios and has higher adaptability.

[0176] In the embodiment of the present disclosure, the received first data file is directly loaded into the memory, and the location indication information of each data and the data identifier corresponding to the data are associated and stored in the memory. Therefore, in the data recommendation process, the storage location of the data can be queried based on the data identifier, and the data can be obtained at the storage location in the first data file. In this way, data acquisition can be achieved without loading the data and the data identifier corresponding to the data into the memory one by one, thereby greatly improving the data loading efficiency.

[0177] In the embodiment of the present disclosure, after the second server loads the first data file and the second data file into the memory, it can provide data recommendation services for the user terminal through the two data files. Accordingly, Figure 6 is a flowchart of another data processing method according to an exemplary embodiment. The embodiment of the present disclosure takes the second server querying the target data based on the target data identifier as an example for explanation. Figure 6 As shown, the following steps are included.

[0178] 601. The second server determines a target data identifier corresponding to target data to be queried.

[0179] In the process of the second server providing data recommendation service for the user terminal, the second server queries the target data (actual value) from multiple data stored in the memory. Since the second server can query data based on the data identifier, the second server needs to first determine the target data identifier corresponding to the target data; optionally, the second server determines the target data identifier corresponding to the target data to be queried. The implementation method includes: the second server receives a data acquisition request sent by the user terminal, and obtains the target data identifier carried in the data acquisition request.

[0180] When the user terminal needs to display the target data, it sends a data acquisition request to the server, so that the second server can obtain the target data identifier corresponding to the target data from the data acquisition request. For example, if the target data is user data, the target data identifier carried in the data acquisition request is also the user identifier.

[0181] Alternatively, the second server receives a recommendation request sent by the user terminal, obtains the user information carried in the recommendation request, and determines a target data identifier associated with the user information.

[0182] When the user terminal needs to display data, the user terminal obtains user information, which can represent relevant information of the user terminal, such as the user information such as the display time of the user terminal, the size of the display screen of the user terminal, the group of people to which the user account logged in by the user terminal belongs, and other information. Among them, the display time refers to the time when the user terminal displays the data. The group to which the user account belongs can be set as needed. Before providing the data recommendation service, the second server will associate and store the user information and the data identifier corresponding to the data. For example, the second server associates and stores the user information and the data identifier corresponding to the data in an inverted index. After obtaining the user information, the second server queries the target data identifier associated with the user information obtained from the recommendation request from the stored user information and the data identifier corresponding to the data.

[0183] In the embodiment of the present disclosure, after receiving a request sent by a user terminal, the target data identifier is determined based on the information carried in the request, so that the target data identifier for the user terminal can be determined, thereby improving the accuracy of the target data identifier.

[0184] 602. The second server queries the memory for location indication information associated with the target data identifier.

[0185] Since the data identifier and the location indication information are stored in association in the memory, the second server can query the corresponding location indication information from the memory based on the target data identifier.

[0186] 603. The second server obtains the target data at the storage location indicated by the location indication information from the first data file.

[0187] In the case where the location indication information is queried, the second server obtains the target data from the first data file. Optionally, the location indication information of any data includes the file identifier of the first data file where the data is located, the offset of the data in the first data file, and the length corresponding to the data; then the implementation method of the second server obtaining the data at the storage location indicated by the location indication information from the first data file includes: the second server obtains the target data of the length from the first data file corresponding to the file identifier, starting from the address indicated by the offset.

[0188] In the disclosed embodiment, since the data location indication information can indicate the storage location of the data, the target data corresponding to the offset and length can be obtained from the first data file corresponding to the file identifier, thereby improving data acquisition efficiency.

[0189] Optionally, the location indication information of the data also includes a status identifier (is static), and the implementation method for the second server to obtain the target data at the storage location indicated by the location indication information from the first data file includes: when the queried location indication information includes a status identifier, the second server obtains the target data at the storage location indicated by the location indication information from the first data file, and the status identifier is used to indicate that the data is stored in the first data file.

[0190] The location indication information of the data and the data identifier corresponding to the data are stored in association in the memory. Optionally, each piece of location indication information and the data identifier corresponding to the data are stored in association in the target memory block of the memory. Since other data may be stored in the target memory block, and the data is different from the location indication information and is not stored in the first data file, then by setting a status identifier in the location indication information, the location indication information and other data can be distinguished according to the status identifier.

[0191] In the embodiment of the present disclosure, by setting a status identifier in the location indication information, the processor can determine whether the data associated with the target data identifier is the location indication information or other data based on the status identifier after obtaining the data from the target memory block. Therefore, when the status identifier is included, the target data at the storage location indicated by the location indication information is obtained, thereby improving the accuracy of data acquisition.

[0192] In some embodiments, the data processing method provided by the embodiments of the present disclosure further includes: the second server obtains incremental data; and associates and stores the incremental data and the data identifier corresponding to the incremental data in the target memory block.

[0193] The incremental data is directly obtained by the second server from the database, that is, it is not transmitted by the first server. The implementation method of associating the incremental data with the data identifier corresponding to the incremental data and storing it in the target memory block is the same as the implementation method of associating the location indication information with the data identifier corresponding to the location indication information and storing it in the target memory block, which will not be repeated here. For example, see Figure 5 The second server acts as a loading module and associates and stores the incremental data with the data identifier corresponding to the incremental data.

[0194] In the embodiment of the present disclosure, the acquired incremental data and the data identifier corresponding to the incremental data are associated and stored in the target memory block, thereby ensuring that the incremental data and the full data previously acquired from the database can be stored in the same memory block, thereby achieving continuity of data storage.

[0195] In the case where the target memory block stores incremental data in addition to the location indication information, the second server obtains the target data corresponding to the target data identifier from the target memory block if the location indication information corresponding to the target data identifier is not found.

[0196] The target data identifier associated with the target memory block queried by the second server may be position indication information or incremental data. In this case, if the position indication information is not queried, the second server can directly obtain the target data stored in association with the target data identifier, which is incremental data.

[0197] Among them, the incremental data stored in the target memory block is stored at the heap address, that is, the incremental data in the target memory block is stored in the form of a pointer. Since the pointer value range corresponding to the heap address is limited, that is, the highest bit of the pointer can only be 0, and the state flag in the position indication information is also stored in the highest bit of the position indication information, the state flag can be set to 1, thereby distinguishing the incremental data stored at the position indication information and the heap address.

[0198] After the second server queries the information corresponding to the target data identifier in the target memory block, it determines whether the highest bit of the information is a status identifier. If it is a status identifier, it indicates that the information is location indication information. If it is not a status identifier, it indicates that the information is incremental data. The second server can directly obtain the incremental data to achieve the acquisition of the target data.

[0199] In the embodiment of the present disclosure, since incremental data is also stored in the target memory block, and the incremental data does not contain a status identifier, when the position indication information is not obtained, it means that the information associated with the target data identifier is incremental data. Then, the target data can be acquired by acquiring the incremental data, thereby improving the accuracy of data acquisition.

[0200] After acquiring the target data, the second server sends the target data to the user terminal so that the user terminal displays the target data. For example, if the target data is multimedia data, the user terminal can display the multimedia data on the display interface.

[0201] In the disclosed embodiment, during the data recommendation process, the storage location of the target data can be queried based on the target data identifier, thereby obtaining the target data at the storage location in the first data file. This enables data acquisition without associating the stored data and the data identifier corresponding to the data, thereby ensuring accurate data acquisition while improving data loading efficiency.

[0202] Figure 7 is a block diagram of a data processing device according to an exemplary embodiment. Figure 7 , the device comprises:

[0203] The file receiving unit 701 is configured to receive a first data file and a second data file sent by a first server, wherein the first data file includes a plurality of data items, and the second data file includes a data identifier and location indication information corresponding to each data item, and the location indication information of the data is used to indicate a storage location of the data;

[0204] A first loading unit 702 is configured to execute loading of a first data file into a memory;

[0205] The first loading unit 702 is further configured to execute the storage of the data identifier and the position indication information corresponding to each piece of data in the second data file in association with each other in the memory.

[0206] In some embodiments, the apparatus further comprises:

[0207] An identifier determining unit, configured to determine a target data identifier corresponding to the target data to be queried;

[0208] An information query unit is configured to query the location indication information associated with the target data identifier from the memory;

[0209] The first acquisition unit is configured to acquire target data at a storage location indicated by the location indication information from the first data file.

[0210] In some embodiments, the location indication information of any data includes a file identifier of a first data file where the data is located, an offset of the data in the first data file, and a length corresponding to the data;

[0211] The first acquisition unit is configured to acquire target data of length from the first data file corresponding to the file identifier, starting from the address indicated by the offset.

[0212] In some embodiments, the first acquisition unit is configured to execute, when the queried location indication information includes a status identifier, to acquire target data at the storage location indicated by the location indication information from the first data file, and the status identifier is used to indicate that the data is stored in the first data file.

[0213] In some embodiments, each piece of location indication information and a data identifier corresponding to the data are stored in association in a target memory block of the memory; the device further comprises:

[0214] A second acquisition unit is configured to acquire incremental data;

[0215] The second loading unit is configured to execute the storage of the incremental data and the data identifier corresponding to the incremental data in a target memory block in association with each other.

[0216] In some embodiments, the apparatus further comprises:

[0217] An identifier determining unit, configured to determine a target data identifier corresponding to the target data to be queried;

[0218] The information query unit is configured to query the location indication information associated with the target data identifier from the target memory block;

[0219] The data acquisition unit is configured to acquire target data corresponding to the target data identifier from the target memory block when the location indication information corresponding to the target data identifier is not found.

[0220] In some embodiments, the identifier determination unit is configured to execute receiving a data acquisition request sent by a user terminal, and obtain a target data identifier carried in the data acquisition request; or, the identifier determination unit is further configured to execute receiving a recommendation request sent by a user terminal, and obtain user information carried in the recommendation request, and determine a target data identifier associated with the user information;

[0221] The device also includes: a data sending unit configured to send target data to the user terminal.

[0222] In some embodiments, the first loading unit 702 is configured to execute, in response to the start of a target service, loading the first data file into the memory, and the target service is used for data recommendation.

[0223] In some embodiments, the first loading unit 702 is configured to execute mapping of the second data file into the memory.

[0224] In the embodiment of the present disclosure, the received first data file is directly loaded into the memory, and the location indication information of each data and the data identifier corresponding to the data are associated and stored in the memory. Therefore, in the data recommendation process, the storage location of the data can be queried based on the data identifier, and the data can be obtained at the storage location in the first data file. In this way, data acquisition can be achieved without loading the data and the data identifier corresponding to the data into the memory one by one, thereby greatly improving the data loading efficiency.

[0225] Figure 8 is a block diagram of another data processing device according to an exemplary embodiment. Figure 8 , the device comprises:

[0226] A data writing unit 801 is configured to write multiple data in a database into a first data file;

[0227] An information generating unit 802 is configured to generate location indication information for each piece of data, where the location indication information is used to indicate a storage location of the data;

[0228] The data writing unit 801 is further configured to write the data identifier corresponding to each piece of data and the data location indication information into the second data file in an associated manner;

[0229] The file sending unit 803 is configured to send the first data file and the second data file to the second server.

[0230] In some embodiments, the information generation unit 802 is configured to determine the file identifier corresponding to the first data file to which each data is written, the offset of the data in the first data file, and the length of the data; and determine the file identifier, offset, and length as location indication information of the data.

[0231] In some embodiments, the apparatus further comprises:

[0232] An information storage unit is configured to associate a data identifier corresponding to the data with the location indication information of the data and store it in a memory;

[0233] The data writing unit 801 is configured to write the data identification and location indication information corresponding to each piece of data stored in association in the memory into the second data file.

[0234] In some embodiments, the data writing unit 801 is configured to obtain multiple data from the database, associate each data with the data identifier corresponding to the data and store it in the memory; and write each data stored in the memory into the first data file.

[0235] In some embodiments, the data writing unit 801 is configured to perform a hash calculation on a data identifier corresponding to each piece of data to obtain a data index; and store the data index and the data in association in a memory.

[0236] In some embodiments, the data writing unit 801 is configured to map each piece of data stored in the memory to the first data file.

[0237] In the disclosed embodiment, multiple pieces of data are written into a first data file, and the data identifier and location indication information corresponding to each piece of data are written into a second data file, so that after receiving the first data file and the second data file, the second server can first determine the storage location of the data based on the location indication information during the data recommendation process, and then obtain the data at the storage location in the first data file. In this way, data acquisition can be achieved without loading the data and the data identifiers corresponding to the data into the memory one by one, thereby greatly improving the data loading efficiency.

[0238] It should be noted that: the data processing device provided in the above embodiment only uses the division of the above functional modules as an example when performing data processing. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the server is divided into different functional modules to complete all or part of the functions described above. In addition, the data processing device provided in the above embodiment and the data processing method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0239] In an exemplary embodiment, a server is also provided. Fig. 9 It is a block diagram of a server according to an exemplary embodiment. The server 900 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 901 and one or more memories 902, wherein the memory 902 is used to store the processor executable instructions, and the processor 901 is configured to execute the instructions to implement the data processing methods provided by the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server may also include other components for implementing device functions, which will not be described in detail here.

[0240] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, and the instructions in the computer-readable storage medium are executed by a processor in a server to implement the data processing method in the above embodiment. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0241] In an exemplary embodiment, a computer program product is also provided, including instructions, and when the instructions are executed by a processor, the data processing method in the above embodiment is implemented.

[0242] In some embodiments, the computer program involved in the embodiments of the present application may be deployed and executed on one server, or on multiple servers located in one location, or on multiple servers distributed in multiple locations and interconnected by a communication network. Multiple servers distributed in multiple locations and interconnected by a communication network may constitute a blockchain system.

[0243] It should be noted that the information (including but not limited to location indication information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions. For example, the data in the database involved in this application are all obtained with full authorization.

[0244] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0245] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data processing method, characterized in that: The data processing method comprises: Receive a first data file and a second data file sent by a first server, wherein the first data file includes a plurality of data items, and the second data file includes a data identifier and location indication information corresponding to each of the data items, wherein the location indication information of the data is used to indicate a storage location of the data; In response to the target service being started, loading the first data file into the memory, the target service being used for data recommendation; Mapping the second data file to the memory, the data identifier and the location indication information corresponding to each piece of the data in the second data file are associated and stored in the memory in the form of a hash table; wherein the location indication information of any piece of the data includes the file identifier of the first data file where the data is located, the offset of the data in the first data file, and the length corresponding to the data; Determine the target data identifier corresponding to the target data to be queried; query the location indication information associated with the target data identifier from the memory; obtain the target data at the storage location indicated by the location indication information from the first data file; Alternatively, each piece of the position indication information is associated with a data identifier corresponding to the data and stored in a target memory block of the memory; the method further comprises: acquiring incremental data; storing the incremental data and the data identifier corresponding to the incremental data in the target memory block in an associated manner; determining a target data identifier corresponding to the target data to be queried; querying the position indication information associated with the target data identifier from the target memory block; and if the position indication information corresponding to the target data identifier is not queried, acquiring the target data corresponding to the target data identifier from the target memory block; The method further comprises: sending the target data to a user terminal; Among them, determining the target data identifier corresponding to the target data to be queried includes: receiving a data acquisition request sent by the user terminal, and obtaining the target data identifier carried in the data acquisition request; or, receiving a recommendation request sent by the user terminal, obtaining user information carried in the recommendation request, and determining the target data identifier associated with the user information.

2. The data processing method according to claim 1, characterized in that: The acquiring, from the first data file, target data at the storage location indicated by the location indication information comprises: The target data of the length is obtained from the first data file corresponding to the file identifier, starting from the address indicated by the offset.

3. The data processing method according to claim 1, characterized in that: The acquiring, from the first data file, target data at the storage location indicated by the location indication information comprises: In a case where the queried location indication information includes a status identifier, the target data at the storage location indicated by the location indication information is acquired from the first data file, and the status identifier is used to indicate that the data is stored in the first data file.

4. A data processing method, characterized in that: The data processing method comprises: Acquire multiple pieces of data from a database, perform hash calculation on a data identifier corresponding to each piece of data, and obtain a data index; Associating the data index with the data in a hash table and storing them in a memory; Mapping each piece of data stored in the memory to a first data file, or serializing each piece of data stored in the memory and storing the obtained byte stream in the first data file; generating location indication information for each piece of the data, wherein the location indication information is used to indicate a storage location of the data; storing the data identifier corresponding to the data in association with the location indication information of the data in a memory; Mapping the data identifier and the position indication information corresponding to each piece of the data stored in the memory to the second data file, or serializing the data identifier and the position indication information corresponding to each piece of the data stored in the memory, and storing the obtained byte stream in the second data file; The first data file and the second data file are sent to a second server; the second server is used to receive the first data file and the second data file, load the first data file and the second data file into a memory, and recommend data to a user terminal based on the first data file and the second data file.

5. The data processing method according to claim 4, characterized in that: The generating the location indication information of each piece of the data comprises: Determine a file identifier corresponding to a first data file into which each piece of the data is written, an offset of the data in the first data file, and a length of the data; The file identifier, the offset and the length are determined as location indication information of the data.

6. A data processing device, characterized in that: The data processing device comprises: A file receiving unit is configured to receive a first data file and a second data file sent by a first server, wherein the first data file includes a plurality of data items, and the second data file includes a data identifier and location indication information corresponding to each of the data items, wherein the location indication information of the data is used to indicate a storage location of the data; A first loading unit is configured to execute, in response to the start of a target service, loading the first data file into a memory, the target service being used for data recommendation; The first loading unit is further configured to execute mapping of the second data file to the memory, wherein the data identifier and the position indication information corresponding to each piece of the data in the second data file are associated and stored in the memory in the form of a hash table; wherein the position indication information of any piece of the data includes the file identifier of the first data file where the data is located, the offset of the data in the first data file, and the length corresponding to the data; An identifier determining unit, configured to determine a target data identifier corresponding to the target data to be queried; An information query unit, configured to query the location indication information associated with the target data identifier from the memory; A first acquisition unit is configured to acquire target data at a storage location indicated by the location indication information from the first data file; Alternatively, each piece of the position indication information is associated with a data identifier corresponding to the data and stored in a target memory block of the memory, and the device further comprises: A second acquisition unit is configured to acquire incremental data; A second loading unit is configured to perform associative storage of the incremental data and a data identifier corresponding to the incremental data in the target memory block; The identification determination unit is further configured to determine the target data identification corresponding to the target data to be queried; The information query unit is further configured to query the location indication information associated with the target data identifier from the target memory block; A data acquisition unit is configured to acquire the target data corresponding to the target data identifier from the target memory block when the location indication information corresponding to the target data identifier is not found; The device further includes: a data sending unit configured to send the target data to a user terminal; Among them, the identification determination unit is configured to execute determination of the target data identification corresponding to the target data to be queried, including: receiving a data acquisition request sent by the user terminal, and obtaining the target data identification carried in the data acquisition request; or, receiving a recommendation request sent by the user terminal, obtaining the user information carried in the recommendation request, and determining the target data identification associated with the user information.

7. The data processing device according to claim 6, characterized in that: The location indication information of any of the data includes a file identifier of a first data file where the data is located, an offset of the data in the first data file, and a length corresponding to the data; The first acquisition unit is configured to acquire the target data of the length from the first data file corresponding to the file identifier, starting from the address indicated by the offset.

8. The data processing device according to claim 6, characterized in that: The first acquisition unit is configured to acquire, from the first data file, target data at a storage location indicated by the location indication information when the queried location indication information includes a status identifier, wherein the status identifier is used to indicate that the data is stored in the first data file.

9. A data processing device, characterized in that: The data processing device comprises: The data writing unit is configured to obtain multiple pieces of data from a database, perform hash calculation on a data identifier corresponding to each piece of data, and obtain a data index; associate the data index with the data in a hash table and store it in a memory; map each piece of data stored in the memory to a first data file, or perform serialization processing on each piece of data stored in the memory, and store the obtained byte stream in the first data file; an information generating unit, configured to generate location indication information of each piece of the data, wherein the location indication information is used to indicate a storage location of the data; An information storage unit, configured to associate a data identifier corresponding to the data with the location indication information of the data and store them in a memory; The data writing unit is further configured to map the data identifier and the position indication information corresponding to each piece of the data stored in the memory to the second data file, or to serialize the data identifier and the position indication information corresponding to each piece of the data stored in the memory, and store the obtained byte stream in the second data file; The file sending unit is configured to send the first data file and the second data file to the second server; the second server is used to receive the first data file and the second data file, load the first data file and the second data file into the memory, and recommend data to the user terminal based on the first data file and the second data file.

10. The data processing device according to claim 9, characterized in that: The information generating unit is configured to determine a file identifier corresponding to the first data file into which each piece of the data is written, an offset of the data in the first data file, and a length of the data; The file identifier, the offset and the length are determined as location indication information of the data.

11. A server, characterized in that: The server comprises: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the data processing method according to any one of claims 1 to 3, or the data processing method according to claim 4 or 5.

12. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of a server, the server is enabled to execute the data processing method as described in any one of claims 1 to 3, or the data processing method as described in claim 4 or 5.

Images