Data processing method and related device

Abstract

The embodiments of this specification provide a data processing method and a related device, wherein a data processing method includes: determining a mapping table corresponding to a data identifier of service data in a first storage system, and reading a parameter status of a parameter in the mapping table, reading a creation time, a parameter identifier, and a parameter value of the parameter in the mapping table according to the parameter status, determining a storage location of the parameter value in a second storage system according to the creation time, the data identifier, and the parameter identifier, and storing the parameter value in the storage location. The use of the embodiments of this application can improve the effectiveness of data storage.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method and related devices. Background Art

[0002] With the continuous development and progress of the Internet and information technology, more and more services are provided online, more and more data can be stored and processed by computers, and users can also query data through computers to avoid the manpower and material resources required to aggregate and record large amounts of data offline. As more and more users participate in the services, more and more data is generated by the services. In order to ensure the effective provision and management of services, a large amount of data needs to be processed. How to improve the effectiveness of data processing is an increasing focus of data processors. Summary of the invention

[0003] In a first aspect, an embodiment of the present application provides a data processing method, including: Determine a mapping table corresponding to a data identifier of the service data, and read a parameter status of a parameter in the mapping table; the mapping table is stored in a first storage system; According to the parameter status, read the creation time, parameter identifier and parameter value of the parameter in the mapping table; A storage location of the parameter value in a second storage system is determined according to the creation time, the data identifier, and the parameter identifier, and the parameter value is stored in the storage location.

[0004] It can be seen that in the embodiment of the present application, in the process of data processing, the mapping table corresponding to the data identifier of the service data is first determined in the first storage system and the parameter status of the parameter in the mapping table is read, so as to determine whether to store the parameter value in the mapping table to the second storage system through the parameter status, so as to avoid storing the parameter value in the second storage system once every time the data identifier of the service data is obtained, resulting in a waste of processing resources. According to the parameter status, the creation time, parameter identifier and parameter value of the parameter in the mapping table are read, and the storage location for storing the parameter value in the second storage system is determined using the creation time, data identifier and parameter identifier. In this way, the storage location is determined using the creation time to ensure that the parameter value at the same creation time remains unchanged in the storage location of the second storage system, so as to avoid occupying too much storage space of the second storage system. After determining the storage location, the parameter value is stored from the first storage system to the second storage system. In this way, the problem of limited memory usage and cache capacity of the first storage system is solved, and effective storage of service data is achieved on the basis of ensuring the performance of the first storage system.

[0005] In a second aspect, an embodiment of the present application provides a data processing device, including: A first reading module, used to determine a mapping table corresponding to a data identifier of service data, and read a parameter state of a parameter in the mapping table; the mapping table is stored in a first storage system; A second reading module, used for reading the creation time, parameter identifier and parameter value of the parameter in the mapping table according to the parameter state; A storage module is used to determine a storage location of the parameter value in a second storage system according to the creation time, the data identifier and the parameter identifier, and store the parameter value in the storage location.

[0006] In a third aspect, an embodiment of the present application provides a data processing device, comprising: a processor; and a memory configured to store computer-executable instructions, wherein the computer-executable instructions, when executed, cause the processor to execute the data processing method described in the first aspect.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium for storing computer-executable instructions, which, when executed by a processor, implement the data processing method as described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor. Figure 1 A schematic diagram of an implementation environment of a data processing method provided in an embodiment of the present application; Figure 2 A data processing method processing flow chart provided in an embodiment of the present application; Figure 3 A processing timing diagram of a data processing method provided in an embodiment of the present application; Figure 4 A state flow diagram provided for an embodiment of the present application; Figure 5 A schematic diagram of a data processing process applied to a data transfer scenario provided in an embodiment of the present application; Figure 6 A processing flow chart of a data processing method applied to a data transfer scenario provided in an embodiment of the present application; Figure 7 A schematic diagram of a data processing device provided in an embodiment of the present application; Figure 8A schematic diagram of the structure of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0009] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0010] In actual applications, if a large amount of data needs to be statistically summarized, real-time summary can be performed in memory to improve performance. For example, big data platforms such as Flink can be used for real-time summary. However, the deployment and maintenance of big data platforms such as Flink are relatively complex and the learning difficulty is also relatively high. In addition, distributed cache systems can also be used for real-time summary, but the memory of distributed cache systems requires a lot of resources to obtain. In order to achieve convenient and affordable data storage, an embodiment of the present application provides a data processing method, which configures a parameter state in a mapping table corresponding to a data identifier stored in a first storage system, stores the parameter value in the mapping table in a second storage system according to the parameter state, and uses the parameter state to control the execution of the transfer (from the first storage system to the second storage system) to ensure the effective execution of the transfer. In the specific process of storing the parameter value in the mapping table in the second storage system, first, the storage location of the parameter value in the second storage system is determined according to the creation time, the data identifier and the parameter identifier, and then the parameter value is transferred from the first storage system to the storage location of the second storage system for storage. In this way, the storage location is determined by using the creation time to ensure that the parameter value at the same creation time is stored in the same location of the second storage system, saving storage space in the second storage system. After determining the storage location, by storing the parameter value from the first storage system to the storage location of the second storage system, it is avoided that the memory of the first storage system is small and cannot flexibly and effectively store a large amount of data, and it is also avoided that a large amount of resources are spent on data storage in the first storage system. On the basis of ensuring the performance of the first storage system, the effective storage of the parameter value of the service data is achieved.

[0011] The data processing method provided in one or more embodiments of this specification may be applicable to the implementation environment of data processing, such as Figure 1 As shown, the implementation environment includes at least a server 101, and the server 101 may be one or more servers, a server cluster consisting of several servers, or a cloud server of a cloud computing platform; The server 101 may deploy a data statistics service 101-1 and a first storage system 101-2; the first storage system 101-2 stores a mapping table corresponding to each data identifier; the data statistics service 101-1 reads and processes the data recorded in the mapping table stored in the first storage system 101-2 by calling a processing function; In addition, the implementation environment may further include a second storage system 102, which is an external storage system of the server 101 and is used to store parameter values ​​transferred from the first storage system 101-2; The implementation environment may also include a terminal device 103, which may be a mobile phone, a personal computer, a tablet computer, an e-book reader, a device for information interaction based on VR (Virtual Reality, virtual reality technology), a vehicle terminal, an IoT device, a wearable smart device, a laptop computer and a desktop computer, etc., for sending service data to the server 101; or, the implementation environment may also include a server 104; the server 104 may be one or more servers, a server cluster composed of several servers, or a cloud server of a cloud computing platform; wherein the terminal device 103 and the server 104 may be data sources for providing service data in this embodiment; in addition, the service data in this embodiment may also be provided by other forms of devices, which are not limited in this embodiment; In this implementation environment, the terminal device 103 sends service data to the server 101. After the server 101 obtains the service data, the data statistics service 101-1 first determines the mapping table corresponding to the data identifier of the service data in the first storage system 101-2, and reads the parameter status of the parameter in the mapping table. Then, according to the parameter status, the creation time, parameter identifier and parameter value of the parameter in the mapping table are read, and the storage location of the parameter value in the second storage system 102 is determined according to the creation time, data identifier and parameter identifier, and the parameter value is transferred from the first storage system 101-2 to the storage location of the second storage system 102 for storage, that is, the parameter value is transferred from the first storage system 101-2 to the second storage system 102, thereby improving the effectiveness of data storage.

[0012] Figure 2 A processing flow chart of a data processing method provided in this embodiment, refer to Figure 2 , a data processing method provided in this embodiment specifically includes steps S202 to S206: Step S202: determine a mapping table corresponding to the data identifier of the service data, and read the parameter status of the parameters in the mapping table.

[0013] The service data in this embodiment includes data that needs to be statistically processed; for example, the short video viewing data of a user in half a day. The service data may be real-time data; the data identifier includes an identifier used to mark the uniqueness of the service data; for example, the dimensions in the short video viewing data of a user in half a day include the user (user-id) and the current date (current-day); then {user-id}-{current-day} can be used as a data identifier, that is, a key to characterize the uniqueness of the service data.

[0014] The first storage system includes a distributed cache system; in this embodiment, the distributed cache system can be used to perform data statistics on the parameter values ​​of the service data. The distributed cache system in this embodiment can be redis. In order to ensure the effectiveness and accuracy of data statistics, in this embodiment, the first storage system can configure a mapping table for a data identifier, and the mapping table stores data related to the data identifier in a mapping storage manner; for example, the mapping table can be hset, and hset stores mappings such as field-value; optionally, the mapping table stores parameter status, creation time, and at least one parameter value. Among them, parameter status, creation time, and at least one parameter value can all be stored through mappings such as field-value. For example, parameter status-initial state, creation time-t, parameter identifier 1-parameter value, parameter identifier 2-parameter value.

[0015] In specific implementation, this embodiment can be applied to a data statistics service. After the data statistics service obtains the service data, it can determine the data identifier of the service data according to a pre-configured data identifier determination strategy or a data dimension (user dimension and / or time dimension). After obtaining the data identifier, the parameter state of the parameter in the mapping table corresponding to the data identifier is read in the first storage system; that is, first determine the mapping table corresponding to the data identifier of the service data, and then read the parameter state of the parameter in the mapping table; in order to realize the reading of the parameter state from the first storage system, in an optional implementation manner provided by this embodiment, in the process of reading the parameter state of the parameter in the mapping table, perform the following operations: Inputting a data statistics instruction including the data identifier and the parameter value to be processed in the service data into a processing function to perform data statistics; Get the parameter status returned by the processing function.

[0016] Specifically, based on the data identifier and the parameter value to be processed in the service data, a processing function is called to perform data statistics on the data in the first storage system, and a parameter status returned by the processing function is obtained.

[0017] During the specific execution process, the processing function can perform data statistics in the following ways: Determine a mapping table based on the data identifier, and read an initial parameter value and a parameter state in the mapping table; Calculate a parameter value according to the parameter value to be processed and the initial parameter value, and update the calculated parameter value to the mapping table; Returns the parameter status.

[0018] In order to achieve real-time statistics of parameter values, the processing function first determines the mapping table based on the data identifier during data statistics based on the data statistics instruction, and then reads the parameter status and the initial parameter value of each parameter in the mapping table (the current parameter value of the parameter recorded in the mapping table). In order to perform parameter value statistics, the parameter value is calculated based on the parameter value to be processed and the initial parameter value, and the calculated parameter value is updated to the mapping table. In addition, the parameter status is returned to the data statistics service.

[0019] For example, after obtaining the service data, the data statistics service generates a corresponding data identifier according to the data dimension, that is, the key is {user-id}-{current-day}. According to the key and the two pending parameter values ​​of the number of views m1 and the viewing time n1, the redis Lua function is called to summarize the data. In the process of data aggregation, the Lua function first reads the parameter status, the initial parameter value m of the number of views and the initial parameter value n of the viewing time in the hset corresponding to {user-id}-{current-day}, and then calculates the sum of the initial parameter value m of the number of views and the pending parameter value m1 according to the summary method of the number of views recorded in the hset as summation, which is m+m1=m2. Similarly, according to the summary method of the viewing time recorded in the hset as summation, the sum of the initial parameter value n of the viewing time and the pending parameter value n1 is calculated as n+n1=n2. Further, the initial parameter value m mapped to the number of views in the hset is updated to m2, and the initial parameter value n mapped to the viewing time in the hset is updated to n2.

[0020] It should be noted that the aggregation method in this embodiment can be to find the maximum and minimum values ​​(max / min), find the average value (avg) and / or sum (sum / count), which is not limited in this embodiment. It should also be noted that the number of parameters in this embodiment can be one or more. When the number of parameters is one, the parameter can be processed. When the number of parameters is more than one, each parameter can be processed separately, which is not limited in this embodiment.

[0021] In the process of data aggregation, in addition to updating the parameter value, in order to improve the convenience of managing the mapping table, in an optional implementation manner provided by this embodiment, the processing function further performs the following operations in the process of data statistics: Detecting whether a mapping table update event is triggered according to the initial parameter value and the parameter state; If so, determining an event triggering object from the initial parameter value and the parameter state; and updating the mapping table according to an update strategy corresponding to the event triggering object; If not, no processing is done.

[0022] Specifically, if the read initial parameter value is empty or the parameter state is in the stored state, a mapping table update event is determined to be triggered, and an event triggering object is determined, and the mapping table is updated according to an update strategy corresponding to the event triggering object; If the read initial parameter value is empty, it is determined that the event trigger object is the initial parameter value, then the creation time is written in the mapping table according to the current time, and the parameter state in the mapping table is updated from the initial state to the state to be stored, thereby completing the mapping table update when the event trigger object is the initial parameter value, wherein the returned parameter state is the initial state; in an optional implementation manner provided by this embodiment, if the parameter state of the parameter in the read mapping table is the initial state, the creation time is written in the mapping table; If the parameter status read is a stored status, it is determined that the event trigger object is a parameter status, and the parameter status in the mapping table is updated from a stored status to a to-be-stored status, and the parameter deletion time of the mapping table is deleted. In this way, the mapping table update is completed when the event trigger object is a parameter status, wherein the returned parameter status is a stored status; that is, if the parameter status is updated during the data statistics process, the returned parameter status is the parameter status before the update.

[0023] It should be noted that when the processing function reads and writes data from the mapping table, all data can be read and written simultaneously, for example, the initial parameter value, creation time and parameter status can be read simultaneously, and the parameter value, creation time and parameter status can be written simultaneously.

[0024] In this embodiment, in order to prevent the mapping table or the parameter value in the mapping table from being stored in the first storage system for a long time and causing memory leakage, an expiration time may be configured for the mapping table, and the expiration time may be configured in the process of writing the parameter value to the mapping table for the first time, and may be reconfigured in the process of updating the parameter value in the mapping table each time, which is not limited in this embodiment. However, it should be noted that only one of the parameter deletion time and expiration time provided above can exist at the same time.

[0025] In the specific execution process, the processing function can perform the following operations in the process of data statistics: Determine a mapping table according to the data identifier carried in the data statistics instruction, and read the initial parameter value and parameter status recorded in the mapping table; If the read initial parameter value is empty, the pending parameter value, current time and state to be stored carried in the data statistics instruction are updated to the mapping table, and the parameter state of the initial state is returned; in addition, in the process of updating the pending parameter value, current time and state to be stored carried in the data statistics instruction to the mapping table, the expiration time of the mapping table can also be configured; for example, Figure 3 As shown in the figure, after the data statistics service obtains the first service data of the key, it calls the Lua function to perform data statistics. The Lua function stores the pending parameter values ​​in the service data into hset, sets the creation time and expiration time, and updates the parameter status to the pending storage status. At this time, the parameter status of hset is the pending storage status, and the parameter value is automatically deleted after the expiration time arrives; the Lua function returns the parameter status of the initial status to the data statistics service, and the data statistics service creates a delayed storage task based on the initial status; If the read initial parameter value is not empty and the parameter state is in the to-be-stored state, the parameter value is calculated based on the initial parameter value and the to-be-processed parameter value, and the parameter value is updated to the mapping table, and the parameter state of the to-be-stored state is returned; in addition, the expiration time can also be reconfigured; for example, Figure 3 As shown in the figure, after the data statistics service obtains the subsequent data of the key, it calls the Lua function to perform data statistics on the hset in redis. The Lua function summarizes the parameter values ​​to be processed in the service data and the initial parameter values ​​in hset, and keeps the parameter status as the state to be stored. At this time, the parameter status of hset is the state to be stored. The Lua function returns the parameter status of the state to be stored to the data statistics service, and the data statistics service does not process it according to the state to be stored. If the read initial parameter value is not empty and the parameter state is in the stored state, the parameter value is calculated based on the initial parameter value and the parameter value to be processed, and the parameter value and the state to be stored are updated to the mapping table, and the parameter state of the stored state is returned; in addition, the expiration time can also be reconfigured; for example, Figure 3As shown, after the data statistics service obtains the first service data whose parameter value of key has been transferred (stored in the second storage system), it calls the Lua function to perform data statistics. The Lua function summarizes the parameter values ​​to be processed in the service data and the initial parameter values ​​in hset, updates the parameter status in hset from the stored status to the to-be-stored status, and updates the parameter deletion time to the expiration time. At this time, the creation time in hset is the creation time set in the data statistics process of the first service data before the transfer, and the parameter status is the to-be-stored status; the Lua function returns the parameter status of the stored status to the data statistics service, and the data statistics service creates a delayed storage task according to the stored status; In this embodiment, if the initial parameter value in the mapping table is empty, the parameter state is the initial state; each time the mapping table is updated with a pending parameter value, the parameter state is updated from the initial state or the stored state to the pending state; if the parameter value is transferred, the parameter state in the mapping table is the stored state. Figure 4 In the state flow diagram shown, when the parameter value in the mapping table is empty, the parameter state is the initial state. After the first pending parameter value is updated to the mapping table, the parameter state of the mapping table is the to-be-stored state. The expiration time can be configured in the to-be-stored state. After the transfer, the parameter state in the mapping table is updated to the stored state. The expiration time can be modified to the parameter deletion time in the stored state. If there are still pending parameters updated to the mapping table before the parameter deletion time arrives, the parameter state in the mapping table is updated from the stored state to the to-be-stored state, and the parameter deletion time can also be modified to the expiration time.

[0026] The above-mentioned process of data statistics of the processing function is described. In addition, the above-mentioned data statistics can also be performed by a data statistics service. Specifically, after obtaining the data identifier of the service data, a mapping table is determined in the first storage system based on the data identifier. In addition to reading the parameter status in the mapping table, the initial parameter value in the mapping table can also be read. Further, the parameter value is calculated according to the parameter value to be processed and the initial parameter value in the service data, and the calculated parameter value is updated to the mapping table. Correspondingly, it also includes detecting whether a mapping table update event is triggered according to the initial parameter value and the parameter status. If so, the event triggering object is determined in the initial parameter value and the parameter status, and the mapping table is updated according to the update strategy corresponding to the event triggering object. The specific process is similar to the execution process of the above-mentioned processing function. The relevant content of the execution process of the above-mentioned processing method can be referred to, and this embodiment will not be repeated here. In addition, when the parameter status read is the initial status or the stored status, in addition to calculating the parameter value based on the pending parameter value in the service data and the initial parameter value in the mapping table, and updating the calculated parameter value to the mapping table, the following step S204 can also be executed, according to the parameter status, reading the creation time, parameter identifier and parameter value of the parameter in the mapping table; if the parameter status read is the to-be-stored status, calculating the parameter value based on the pending parameter value in the service data and the initial parameter value in the mapping table, and updating the calculated parameter value to the mapping table.

[0027] Step S204: read the creation time, parameter identifier and parameter value of the parameter in the mapping table according to the parameter status.

[0028] In the above steps, after obtaining the data identifier of the service data and reading the parameter status in the mapping table corresponding to the data identifier in the first storage system, the creation time, parameter identifier and parameter value of the parameter in the mapping table are read according to the parameter status; specifically, if the parameter status meets the storage condition, that is, the parameter status meets the storage condition from the first storage system to the second storage system, the creation time, parameter identifier and parameter value of the parameter in the mapping table are read.

[0029] In specific implementation, if the parameter state is the initial state or the stored state, it is determined that the parameter state meets the storage condition; if the parameter state is the to-be-stored state, it is determined that the parameter state does not meet the storage condition; if the parameter state does not meet the storage condition, no processing is performed. Figure 4 As shown, if the Lua function is called to obtain the initial state, a delayed storage task is created. If the Lua function is called to obtain the state to be stored, no processing is done. If the Lua function is called to obtain the stored state, a delayed storage task is created.

[0030] In order to avoid wasting resources by transferring data every time service data is acquired, a storage condition may be introduced to determine whether to store parameter values ​​in the second storage system. When the parameter status meets the storage condition, a delayed storage task may be created, and the transfer may be performed when the task execution time of the delayed storage task is reached. In an optional implementation provided by this embodiment, if the parameter status meets the storage condition, a delayed storage task is created based on the data identifier; when it is detected that the task execution time of the delayed storage task has arrived, the creation time, parameter identifier, and parameter value recorded in the mapping table are read. That is, if the parameter status is an initial state or a stored state, a delayed storage task is created based on the data identifier; when it is detected that the delayed storage task has reached the task execution time, the creation time, parameter identifier, and parameter value of the parameter in the mapping table are read; In order to improve the effectiveness of the management of the mapping table, in an optional implementation manner provided in this embodiment, after creating a delayed storage task based on a data identifier, the parameter state in the mapping table (from an initial state or a stored state) is also updated to a state to be stored. Furthermore, when it is detected that the delayed storage task has reached the task execution time, after reading the creation time, parameter identifier and parameter value of the parameter in the mapping table, the parameter state in the mapping table can also be updated from the state to be stored to the state to be stored.

[0031] For example, Figure 3 As shown, the data statistics service calls the delayed scheduling service to create a delayed storage task. When the delayed scheduling service detects that the task execution time of the delayed storage task has arrived, it calls the Lua function to query the data. The Lua function reads the creation time and parameter value recorded in the hset corresponding to the key and returns it to the delayed scheduling service. In addition, the Lua function also updates the parameter status in the hset from the to-be-stored status to the stored status, and sets the parameter deletion time. After the delayed scheduling service obtains the parameter value, parameter identifier and creation time returned by the Lua function, it transfers the parameter value to the external storage system (i.e., the second storage system) according to the creation time, data identifier and parameter identifier. It should be noted that the above-mentioned delayed scheduling service can also be a data statistics service; at this time, the parameter status in the hset is the stored status, and the parameter value is automatically deleted after the parameter deletion time arrives.

[0032] In the specific execution process, if the parameter state is the initial state or the stored state, a delayed storage task is created according to the preset delay time. Before the task execution time of the delayed storage task arrives, the parameter state in the mapping table is the to-be-stored state. When it is detected that the delayed storage task has arrived at the task execution time, the delayed storage task is executed to store the parameter value in the second storage system. Specifically, the delayed scheduling service can be called to create the delayed storage task. When it is detected that the delayed storage task has arrived at the task execution time, the delayed scheduling service executes the delayed storage task or calls the data statistics service to execute the delayed storage task. The specific execution process of the delayed storage task is similar. Specifically, the creation of the delayed storage task can be implemented by using a delay queue or other methods, which is not limited in this embodiment.

[0033] Further, similar to the above-mentioned reading of the parameter state, the creation time, parameter identifier and parameter value recorded in the mapping table can also be read by calling a processing function; in an optional implementation provided by this embodiment, in the process of reading the creation time, parameter identifier and parameter value recorded in the mapping table, a data query instruction containing a data identifier is input into the processing function to perform data query, and the creation time, parameter identifier and parameter value returned by the processing function are obtained. Optionally, the processing function performs the following operations in the process of performing data query: Read the creation time, parameter identifier and parameter value recorded in the mapping table and return them; Based on the data query instruction, updating the parameter status in the mapping table to a stored status; In addition, the parameter deletion time of the mapping table may be configured so that the parameter value recorded in the mapping table is deleted after the parameter deletion time is reached.

[0034] Specifically, the processing function first reads the creation time, parameter identifier and parameter value recorded in the mapping table and returns them to the data statistics service. In addition, in order to accurately represent the storage situation with the parameter status in the mapping table, based on the data query instruction, the parameter status in the mapping table is updated to the stored status. When the parameter value is stored in the second storage system, in order to avoid occupying the space of the first storage system, the parameter deletion time of the mapping table is configured, and the parameter value in the mapping table is deleted after the parameter deletion time arrives. In addition, the Lua function can be used for data query without calling it, and the data statistics service can perform data query.

[0035] In this embodiment, the parameter deletion time is configured. On the one hand, it is ensured that the parameter value is unidirectionally flowing (flowing only from the first storage system to the second storage system), and the performance and concurrency problems of the first storage system caused by the bidirectional flow are avoided. The parameter deletion time is configured, and the parameter value recorded in the mapping table is deleted after the parameter deletion time is reached. In this way, after the parameter value is deleted, when a new parameter value arrives, a new storage location is determined by a new creation time, and the parameter value stored in the second storage system and has reached the parameter deletion time will not be directly overwritten in the storage location of the second storage system, resulting in the loss of the stored parameter value; if the mapping table or the parameter value in the mapping table is deleted directly after storage, when a new parameter value arrives later, it will cause multiple storage locations of the second storage system to be occupied for storage; in actual applications, service data is either continuously acquired for a period of time or occurs in small quantities. For data that occurs continuously, the parameter deletion time is configured to ensure continuous updating of the same storage location to avoid storage location explosion. For high-cardinality service data with small occurrences, the parameter deletion time is configured to avoid long-term occupation of the first storage system.

[0036] Step S206: determine the storage location of the parameter value in the second storage system according to the creation time, the data identifier and the parameter identifier, and store the parameter value in the storage location.

[0037] The second storage system in this embodiment includes an external storage system, that is, a storage system independent of the server running the data statistics service and the first storage system; the second storage system can be a distributed storage system, such as hbase (Hadoop Database). In addition, the second storage system can also be a storage system of other structures, such as MySQL (an open source relational database management system).

[0038] In the specific implementation, after reading the creation time, parameter identifier and parameter value, first determine the storage location of the parameter value in the second storage system based on the creation time, data identifier and parameter identifier, and then store the parameter value to the storage location, thereby realizing the transfer of the parameter value from the first storage system (distributed cache system) to the second storage system (distributed storage system).

[0039] In order to perform effective transfer, the storage location of the parameter value in the second storage system is determined according to the creation time, the data identifier and the parameter identifier; in an optional implementation manner provided by this embodiment, in the process of determining the storage location of the parameter value in the second storage system according to the creation time, the data identifier and the parameter identifier, the following operations are performed: Determine, in the second storage system, row data whose row identifier matches the data identifier, and determine, in the second storage system, column data whose column identifier matches the creation time and the parameter identifier; The positions corresponding to the row data and column data obtained by matching are determined as the storage positions.

[0040] Specifically, first determine the storage row whose row identifier matches the data identifier, and determine the storage column whose column identifier matches the creation time and parameter identifier, and then determine the corresponding position of the storage row and the storage column as the storage position. In order to avoid the inability to determine the storage position due to the absence of a row identifier that matches the data identifier or the absence of a storage column that matches the creation time and parameter identifier, in an optional implementation provided by this embodiment, if the row data obtained by matching is empty, then generate a storage row based on the data identifier to obtain the row data, and / or, if the column data obtained by matching is empty, then generate a storage column based on the creation time and parameter identifier to obtain the column data.

[0041] For example, the key is { user-id}-{ current-day}. In the process of determining the storage location, the storage row (rowkey) in hbase is first determined according to the key: view-state-{ user-id}-{ current-day}, where view-state is the data type of the service data; according to the parameter identifier view-count of the parameter viewing number and the creation time t, the storage column of the viewing number parameter is determined to be view-count-t, and according to the viewing duration view-duration and the creation time t, the storage column of the viewing duration parameter is determined to be view-duration-t. In this way, the position in hbase with view-state-{ user-id}-{ current-day} as the row and view-count-t as the column is determined as the storage location of the viewing number parameter, and the position with view-state-{ user-id}-{ current-day} as the row and view-duration-t as the column is determined as the storage location of the viewing time parameter. It should be noted that if there is a location with view-state-{ user-id}-{ current-day} as the row and view-count-t as the column, then this location is directly determined as the storage location. If there is only a location with view-state-{ user-id}-{ current-day} as the row but no location with view-count-t as the column, then a location with view-state-{ user-id}-{ current-day} as the row and view-count-t as the column is created as the storage location.

[0042] After the storage location is determined, the parameter value in the first storage system is stored in the storage location in the second storage system.

[0043] In addition to using the above method to determine the storage location and then store the parameter value, when the second storage system is another storage system, the parameter value can be stored according to the storage type of the second storage system, for example, stored in the second storage system through a key-value pair structure; this embodiment is not limited here.

[0044] In addition, a data retrieval service may be provided to the user, and the user may retrieve the parameter value corresponding to the relevant data identifier through the data retrieval service, thereby conveniently and accurately performing data retrieval, and performing retrieval in the second storage system when the retrieval result in the first storage system is empty, thereby avoiding the retrieval failure after the expiration time of the first storage system reaches and affecting the user's retrieval experience. In an optional implementation manner provided in this embodiment, data retrieval may be performed in the following manner: Obtaining a data retrieval request, and determining an identifier of data to be retrieved according to the data retrieval request; According to the identifier of the data to be retrieved, querying in the first storage system a corresponding first search parameter value in the mapping table to be retrieved; If the query result is not empty, generating a search result according to the first search parameter value and the second search parameter value found in the second storage system according to the identifier of the data to be searched; If the query result is empty, the second search parameter value is used as the search result.

[0045] Optionally, the mapping table to be retrieved includes a mapping table corresponding to the identifier of the data to be retrieved in the first storage system.

[0046] Specifically, during the data retrieval process, on the one hand, the first retrieval parameter value in the corresponding to-be-retrieved mapping table is queried in the first storage system according to the to-be-retrieved data identifier, and on the other hand, the second retrieval parameter value is queried in the second storage system according to the to-be-retrieved data identifier, and then the first retrieval parameter value and the second retrieval parameter value are merged to obtain the retrieval result. In this process, the first retrieval parameter value in the to-be-retrieved mapping table in the first storage system may be empty due to transfer or other circumstances, and the second retrieval parameter value can be directly used as the retrieval result.

[0047] It should be noted that determining the identifier of the data to be retrieved based on the data to be retrieved in the data retrieval request can be similar to the above-mentioned process of determining the data identifier based on the service data, and this embodiment will not be repeated here. The above-mentioned data retrieval process can be executed by the data retrieval service, and can also be executed by the data statistics service, which is not limited in this embodiment.

[0048] In summary, the data processing method provided by the present embodiment is that after the data statistics service obtains the service data, it first determines the data identifier of the service data. When the data identifier is obtained, the data statistics instruction containing the data identifier and the parameter value to be processed in the service data is input into the processing function to perform data statistics. After receiving the data statistics instruction, the processing function first determines the corresponding mapping table in the first storage system based on the data identifier, and reads the initial parameter value and parameter status of the parameter in the mapping table. After that, on the one hand, the parameter status is returned to the data statistics service, and on the other hand, if the initial parameter value is empty, the current time is written into the mapping The table is used as the creation time, the parameter value to be processed is written into the mapping table, the parameter status in the mapping table is updated from the initial status to the status to be stored and the expiration time is set; if the initial parameter value is not empty and the parameter status is the status to be stored, the parameter value is calculated according to the initial parameter value and the parameter value to be processed, and the calculated parameter value is updated to the mapping table; if the initial parameter value is not empty and the parameter status is the stored status, the parameter value is calculated according to the initial parameter value and the parameter value to be processed, the calculated parameter value is updated to the mapping table, the parameter status in the mapping table is updated from the stored status to the status to be stored, and the parameter deletion time is modified to the expiration time; The data statistics service obtains the parameter status returned by the processing function. If the parameter status is a to-be-stored status, no processing is performed. If the parameter status is an initial status or a stored status, a delayed storage task is created. When it is detected that the task execution time of the delayed storage task has arrived, the data query instruction containing the data identifier is input into the processing function for data query. The processing function reads the parameter value, parameter identifier and creation time of the parameter in the mapping table corresponding to the data identifier in the first storage system and returns it to the data statistics service. In addition, the processing function also updates the parameter status in the mapping table from a to-be-stored status to a stored status. If the expiration time is not set in advance, A parameter deletion time is set for the mapping table. When an expiration time is set in advance, the expiration time is modified to the parameter deletion time. In this way, the data statistics instructions and data query instructions in the first storage system are serially executed through the processing function to ensure thread safety; and the data query in the data statistics and transfer process is realized through the processing function, which is executed in a single-threaded sequence and the parameter value is transferred from the first storage system to the second storage system unidirectionally to avoid concurrency problems and ensure performance; it should be noted that when the parameter deletion time or the expiration time is reached, the corresponding mapping table or the parameter value in the mapping table in the distributed cache system is deleted; After the data statistics service obtains the creation time, parameter identifier and parameter value returned by the processing function, it first determines the storage location in the second storage system based on the creation time, data identifier and parameter identifier, and then stores the parameter value to the storage location. In this way, the parameters in the first storage system are transferred to avoid occupying the first storage system, and the parameter values ​​are stored in the second storage system, thereby improving the storage time and storage efficiency.

[0049] The following takes the application of a data processing method provided in this embodiment in a data transfer scenario as an example, combined with Figure 5 and Figure 6 For further explanation of the data processing method applied to the data transfer scenario provided in this embodiment, see Figure 6 , a data processing method applied to a data transfer scenario specifically includes the following steps.

[0050] like Figure 5 As shown in the figure, after the data statistics service obtains the service data stream provided by the data source, it first calls the Lua function according to the key and the parameter value to be processed to summarize the parameter values ​​in the corresponding hset in redis and return the parameter status. If the parameter status returned by the Lua function is the initial status or the stored status, the delayed scheduling service is called to create a delayed storage task. After detecting that the task execution time of the delayed storage task has arrived, the delayed scheduling service calls the data statistics service to store the parameter value externally. In response to the call of the external storage, the data statistics service calls the Lua function according to the key to query the parameter value, parameter identifier and creation time in the corresponding hset in redis, and updates it to the stored status. After the statistics service obtains the parameter value, parameter identifier and creation time, it stores the parameter value in the external storage system according to the creation time, data identifier and parameter identifier; if the user retrieves data through the data retrieval service, the data is first retrieved from redis. If the retrieval result is empty, the data is retrieved from the external storage system, and the retrieval result is obtained and returned to the user; if the retrieval result is not empty, the first retrieval result obtained from the data retrieval in redis and the second retrieval result obtained from the data retrieval in the external storage system are generated into a final retrieval result and returned to the user; in the process of retrieving data from redis, data can be queried in hset through the hgetall command.

[0051] It should be noted that the above-mentioned data statistics service, delay scheduling service and data retrieval service can be deployed on the same server, that is, the above-mentioned process can be directly executed by the server without distinguishing between services.

[0052] Step S602: Acquire user service data and generate a data identifier according to the user service data.

[0053] Step S604: determine a mapping table corresponding to the data identifier in the first storage system, and read the parameter status of the parameters in the mapping table.

[0054] In addition, step S604 may also be replaced by calling a processing function based on the data identifier and the parameter value to be processed in the parameter value to be processed to perform data statistics in the mapping table of the first storage system, and obtaining the parameter status returned by the processing function.

[0055] Optionally, data statistics can be implemented as follows: Determine a mapping table in the first storage system based on the data identifier, read an initial parameter value and a parameter status of a parameter in the mapping table and return the parameter status; If the initial parameter value read is empty, the current time is written into the mapping table as the creation time, the parameter value to be processed is written into the mapping table, the parameter state in the mapping table is updated from the initial state to the state to be stored and the expiration time is set; If the initial parameter value is not empty and the parameter state is in the to-be-stored state, the parameter value is calculated according to the initial parameter value and the to-be-processed parameter value, and the calculated parameter value is updated to the mapping table; If the initial parameter value is not empty and the parameter status is in the stored state, the parameter value is calculated based on the initial parameter value and the parameter value to be processed, the calculated parameter value is updated to the mapping table, the parameter status in the mapping table is updated from the stored state to the to-be-stored state, and the parameter deletion time is modified to the expiration time.

[0056] If the read initial parameter value is empty, it is determined that the parameter state recorded in the mapping table is the initial state.

[0057] In addition, step S604 may also be replaced by performing data statistics in a mapping table in the first storage system based on the data identifier and the parameter value to be processed, and forming a new implementation method with one or more other processing steps provided in this embodiment.

[0058] Step S606: If the parameter state is an initial state or a stored state, a delayed storage task is created based on the data identifier.

[0059] Step S608: If it is detected that the task execution time of the delayed storage task has arrived, a processing function is called based on the data identifier to perform data query in the mapping table.

[0060] Optionally, data query can be implemented as follows: Read the parameter value, parameter identifier and creation time recorded in the mapping table corresponding to the data identifier in the first storage system and return it; The parameter status in the mapping table is updated from the to-be-stored status to the stored status. If no expiration time is set in advance, the parameter deletion time is set for the mapping table. If the expiration time is set in advance, the expiration time is modified to the parameter deletion time.

[0061] In addition, step S608 can also be replaced by, if it is detected that the task execution time of the delayed storage task has arrived, performing data query in the first storage system based on the data identifier, and forming a new implementation method with one or more other processing steps provided in this embodiment.

[0062] Step S610, obtaining the creation time, data identifier and parameter value returned by the processing function.

[0063] Step S612: Determine the storage location of the parameter value in the second storage system according to the creation time, the data identifier and the parameter identifier.

[0064] Step S614, storing the parameter value in a storage location.

[0065] It should be noted that any one of steps S602 to S614 or any combination of multiple steps can be combined with any one of steps S202 to S206 to form a new implementation method according to the needs of implementation deployment; in addition, according to the needs of actual deployment, any one or multiple technical features can be selected from steps S602 to S6146 and combined with any one or multiple technical features provided by steps S202 to S206 to form a new implementation method; or, any one or multiple technical features in steps S602 to S614 can be replaced with any one or multiple technical features provided by steps S202 to S206 to form a new implementation method according to the needs of actual deployment, which will not be repeated here.

[0066] An embodiment of a data processing device provided in this specification is as follows: In the above-mentioned embodiment, a data processing method is provided, and correspondingly, a data processing device is also provided, which will be described below with reference to the accompanying drawings.

[0067] Reference Figure 7 , which shows a schematic diagram of the structure of a data processing device provided by this embodiment. Since the device embodiment corresponds to the method embodiment, the description is relatively simple, and the relevant parts can refer to the corresponding description of the method embodiment provided above. The device embodiment described below is only illustrative.

[0068] This embodiment provides a data processing device, including: A first reading module 702, used to determine a mapping table corresponding to a data identifier of service data, and read a parameter state of a parameter in the mapping table; the mapping table is stored in a first storage system; A second reading module 704 is used to read the creation time, parameter identifier and parameter value of the parameter in the mapping table according to the parameter state; The storage module 706 is used to determine the storage location of the parameter value in the second storage system according to the creation time, the data identifier and the parameter identifier, and store the parameter value in the storage location.

[0069] In one embodiment, when the second reading module 704 reads the creation time, parameter identifier and parameter value of the parameter in the mapping table according to the parameter state, it performs the following steps: If the parameter state is an initial state or a stored state, creating a delayed storage task based on the data identifier; When it is detected that the task execution time of the delayed storage task has arrived, the creation time, parameter identifier and parameter value of the parameter in the mapping table are read.

[0070] In one embodiment, after creating a delayed storage task based on the data identifier, the second reading module 704 also updates the parameter status in the mapping table to a to-be-stored state; after reading the creation time, parameter identifier and parameter value of the parameter in the mapping table, the second reading module 704 also updates the parameter status in the mapping table to a stored state.

[0071] In one embodiment, after determining the mapping table corresponding to the data identifier of the service data and reading the parameter status of the parameter in the mapping table, the first reading module 702 further performs the following steps: If the parameter state is a to-be-stored state, calculating the parameter value based on the to-be-processed parameter value in the service data and the initial parameter value in the mapping table; The calculated parameter values ​​are updated to the mapping table.

[0072] In one embodiment, after determining the mapping table corresponding to the data identifier of the service data and reading the parameter status of the parameter in the mapping table, the first reading module 702 further performs the following steps: If the parameter state is an initial state, the creation time is written into the mapping table.

[0073] In one embodiment, when determining the storage location of the parameter value in the second storage system according to the creation time, the data identifier and the parameter identifier, the storage module 706 performs the following steps: Determine, in the second storage system, row data whose row identifier matches the data identifier, and determine, in the second storage system, column data whose column identifier matches the creation time and the parameter identifier; The positions corresponding to the row data and column data obtained by matching are determined as the storage positions.

[0074] In one embodiment, when determining the storage location of the parameter value in the second storage system according to the creation time, the data identifier and the parameter identifier, the storage module 706 further performs the following steps: If the row data obtained by matching is empty, a storage row is generated based on the data identifier to obtain the row data; or, If the column data obtained by matching is empty, a storage column is generated based on the creation time and the parameter identifier to obtain the column data.

[0075] In the data processing device provided in the embodiment, during the process of data processing, firstly, a mapping table corresponding to a data identifier of service data is determined in the first storage system and the parameter status of the parameter in the mapping table is read, so as to determine whether to store the parameter value in the mapping table to the second storage system through the parameter status, so as to avoid storing the parameter value in the second storage system once every time the data identifier of the service data is obtained, resulting in a waste of processing resources. According to the parameter status, the creation time, parameter identifier and parameter value of the parameter in the mapping table are read, and the storage location for storing the parameter value in the second storage system is determined using the creation time, the data identifier and the parameter identifier. In this way, the storage location is determined using the creation time, so as to ensure that the parameter value at the same creation time remains unchanged in the storage location of the second storage system, so as to avoid occupying too much storage space of the second storage system. After determining the storage location, the parameter value is stored from the first storage system to the second storage system, so as to solve the problem of limited memory occupation and cache capacity of the first storage system, and realize effective storage of service data on the basis of ensuring the performance of the first storage system.

[0076] An embodiment of a data processing device provided in this specification is as follows: Corresponding to the data processing method described above, based on the same technical concept, an embodiment of the present application also provides a data processing device, which is used to execute the data processing method provided above, Figure 8 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application.

[0077] This embodiment provides a data processing device, including: like Figure 8As shown, the data processing device may have relatively large differences due to different configurations or performances, and may include one or more processors 801 and memory 802, and the memory 802 may store one or more storage applications or data. Among them, the memory 802 may be a short-term storage or a persistent storage. The application stored in the memory 802 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the data processing device. Furthermore, the processor 801 may be configured to communicate with the memory 802 and execute a series of computer executable instructions in the memory 802 on the data processing device. The data processing device may also include one or more power supplies 803, one or more wired or wireless network interfaces 804, one or more input / output interfaces 805, one or more keyboards 806, etc.

[0078] In a specific embodiment, a data processing device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for the data processing device, and the one or more programs configured to be executed by one or more processors include computer-executable instructions for performing the following: Determine a mapping table corresponding to a data identifier of the service data, and read a parameter status of a parameter in the mapping table; the mapping table is stored in a first storage system; According to the parameter status, read the creation time, parameter identifier and parameter value of the parameter in the mapping table; A storage location of the parameter value in a second storage system is determined according to the creation time, the data identifier, and the parameter identifier, and the parameter value is stored in the storage location.

[0079] In one embodiment, when the processor reads the creation time, parameter identifier and parameter value of the parameter in the mapping table according to the parameter state, the processor performs the following steps: If the parameter state is an initial state or a stored state, creating a delayed storage task based on the data identifier; When it is detected that the task execution time of the delayed storage task has arrived, the creation time, parameter identifier and parameter value of the parameter in the mapping table are read.

[0080] In one embodiment, after creating the delayed storage task based on the data identifier, the processor also updates the parameter state in the mapping table to a to-be-stored state; After reading the creation time, parameter identifier and parameter value of the parameter in the mapping table, the processor also updates the parameter status in the mapping table to a stored status.

[0081] In one embodiment, the processor further performs the following steps: If the parameter state is a to-be-stored state, calculating the parameter value based on the to-be-processed parameter value in the service data and the initial parameter value in the mapping table; The calculated parameter values ​​are updated to the mapping table.

[0082] In one embodiment, the processor further performs the following steps: If the parameter state is an initial state, the creation time is written into the mapping table.

[0083] In one embodiment, when determining the storage location of the parameter value in the second storage system according to the creation time, the data identifier, and the parameter identifier, the processor performs the following steps: Determine, in the second storage system, row data whose row identifier matches the data identifier, and determine, in the second storage system, column data whose column identifier matches the creation time and the parameter identifier; The positions corresponding to the row data and column data obtained by matching are determined as the storage positions.

[0084] In one embodiment, when the processor determines the storage location of the parameter value in the second storage system according to the creation time, the data identifier and the parameter identifier, the processor further performs the following steps: If the row data obtained by matching is empty, a storage row is generated based on the data identifier to obtain the row data; or, If the column data obtained by matching is empty, a storage column is generated based on the creation time and the parameter identifier to obtain the column data.

[0085] In the data processing device provided in this embodiment, during the process of data processing, firstly, a mapping table corresponding to a data identifier of service data is determined in the first storage system and a parameter status of a parameter in the mapping table is read, so as to determine whether to store the parameter value in the mapping table to the second storage system through the parameter status, so as to avoid storing the parameter value in the second storage system once each time the data identifier of the service data is obtained, resulting in a waste of processing resources. According to the parameter status, the creation time, parameter identifier and parameter value of the parameter in the mapping table are read, and the storage location for storing the parameter value in the second storage system is determined using the creation time, data identifier and parameter identifier. In this way, the storage location is determined using the creation time, so as to ensure that the storage location of the parameter value at the same creation time remains unchanged in the second storage system, so as to avoid occupying too much storage space of the second storage system. After determining the storage location, the parameter value is stored from the first storage system to the second storage system, so as to solve the problem of limited memory occupation and cache capacity of the first storage system, and realize effective storage of service data on the basis of ensuring the performance of the first storage system.

[0086] An embodiment of a computer-readable storage medium provided in this specification is as follows: Corresponding to the data processing method described above, based on the same technical concept, an embodiment of the present application also provides a computer-readable storage medium.

[0087] The computer-readable storage medium provided in this embodiment is used to store computer-executable instructions, and the computer-executable instructions implement the following process when executed: Determine a mapping table corresponding to a data identifier of the service data, and read a parameter status of a parameter in the mapping table; the mapping table is stored in a first storage system; According to the parameter status, read the creation time, parameter identifier and parameter value of the parameter in the mapping table; A storage location of the parameter value in a second storage system is determined according to the creation time, the data identifier, and the parameter identifier, and the parameter value is stored in the storage location.

[0088] In the computer-readable storage medium provided in this embodiment, during the process of data processing, firstly, a mapping table corresponding to a data identifier of service data is determined in the first storage system and a parameter status of a parameter in the mapping table is read, so as to determine whether to store the parameter value in the mapping table to the second storage system through the parameter status, so as to avoid storing the parameter value in the second storage system once each time the data identifier of the service data is obtained, resulting in a waste of processing resources. According to the parameter status, the creation time, parameter identifier and parameter value of the parameter in the mapping table are read, and the storage location for storing the parameter value in the second storage system is determined using the creation time, the data identifier and the parameter identifier. In this way, the storage location is determined using the creation time, so as to ensure that the storage location of the parameter value at the same creation time remains unchanged in the second storage system, so as to avoid occupying too much storage space of the second storage system. After determining the storage location, the parameter value is stored from the first storage system to the second storage system, so as to solve the problem of limited memory occupation and cache capacity of the first storage system, and realize effective storage of service data on the basis of ensuring the performance of the first storage system.

[0089] It should be noted that an embodiment of a computer-readable storage medium in this specification and an embodiment of a data processing method in this specification are based on the same inventive concept, so the specific implementation of this embodiment can refer to the implementation of the aforementioned corresponding method, and the repeated parts will not be repeated.

[0090] Another embodiment of the present disclosure further provides a computer program product, the computer program product comprising a computer program, and when the computer program is executed by a processor, the following process is implemented: Determine a mapping table corresponding to a data identifier of the service data, and read a parameter status of a parameter in the mapping table; the mapping table is stored in a first storage system; According to the parameter status, read the creation time, parameter identifier and parameter value of the parameter in the mapping table; A storage location of the parameter value in a second storage system is determined according to the creation time, the data identifier, and the parameter identifier, and the parameter value is stored in the storage location.

[0091] In the computer program product provided in this embodiment, during the process of data processing, firstly, a mapping table corresponding to a data identifier of service data is determined in the first storage system and a parameter status of a parameter in the mapping table is read, so as to determine whether to store the parameter value in the mapping table to the second storage system according to the parameter status, so as to avoid storing the parameter value in the second storage system once every time the data identifier of the service data is obtained, resulting in a waste of processing resources. According to the parameter status, the creation time, parameter identifier and parameter value of the parameter in the mapping table are read, and the storage location for storing the parameter value in the second storage system is determined using the creation time, the data identifier and the parameter identifier. In this way, the storage location is determined using the creation time, so as to ensure that the storage location of the parameter value at the same creation time remains unchanged in the second storage system, so as to avoid occupying too much storage space of the second storage system. After determining the storage location, the parameter value is stored from the first storage system to the second storage system, so as to solve the problem of limited memory occupation and cache capacity of the first storage system, and realize effective storage of service data on the basis of ensuring the performance of the first storage system.

[0092] The computer program product in the embodiments of the present disclosure can implement the various processes of the above-mentioned data processing method embodiments and achieve the same effects and functions, which will not be repeated here.

[0093] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0094] It should be understood by those skilled in the art that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the embodiments of the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0095] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable test equipment to produce a machine, so that the instructions executed by the processor of the computer or other programmable test equipment generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0096] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable test equipment to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, the instruction device being implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0097] These computer program instructions may also be loaded onto a computer or other programmable test device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0098] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0099] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0100] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0101] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0102] Embodiments of the present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0103] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0104] The above description is only an embodiment of this document and is not intended to limit this document. For those skilled in the art, this document may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this document should be included in the scope of the claims of this document.

Claims

1. A data processing method, characterized in that: The method comprises: Determine a mapping table corresponding to a data identifier of the service data, and read a parameter status of a parameter in the mapping table; the mapping table is stored in a first storage system; According to the parameter status, read the creation time, parameter identifier and parameter value of the parameter in the mapping table; A storage location of the parameter value in a second storage system is determined according to the creation time, the data identifier, and the parameter identifier, and the parameter value is stored in the storage location.

2. The method according to claim 1, characterized in that The step of reading the creation time, parameter identifier and parameter value of the parameter in the mapping table according to the parameter state includes: If the parameter state is an initial state or a stored state, creating a delayed storage task based on the data identifier; When it is detected that the task execution time of the delayed storage task has arrived, the creation time, parameter identifier and parameter value of the parameter in the mapping table are read.

3. The method according to claim 2, characterized in that After the operation of creating a delayed storage task based on the data identifier is executed, the method further includes: Updating the parameter status in the mapping table to a to-be-stored state; After the operation of reading the creation time, parameter identifier and parameter value of the parameter in the mapping table is performed, the method further includes: The parameter status in the mapping table is updated to a stored status.

4. The method according to claim 1, characterized in that: The method further comprises: If the parameter state is a to-be-stored state, calculating the parameter value based on the to-be-processed parameter value in the service data and the initial parameter value in the mapping table; The calculated parameter values ​​are updated to the mapping table.

5. The method according to claim 1, characterized in that After the operation of determining a mapping table corresponding to the data identifier of the service data and reading the parameter status of the parameter in the mapping table is performed, the method further includes: If the parameter state is an initial state, the creation time is written into the mapping table.

6. The method according to claim 1, characterized in that The determining, according to the creation time, the data identifier, and the parameter identifier, a storage location of the parameter value in the second storage system includes: Determine, in the second storage system, row data whose row identifier matches the data identifier, and determine, in the second storage system, column data whose column identifier matches the creation time and the parameter identifier; The positions corresponding to the row data and column data obtained by matching are determined as the storage positions.

7. The method according to claim 6, characterized in that The method further comprises: If the row data obtained by matching is empty, a storage row is generated based on the data identifier to obtain the row data; or, If the column data obtained by matching is empty, a storage column is generated based on the creation time and the parameter identifier to obtain the column data.

8. A data processing device, characterized in that: The device comprises: A first reading module, used to determine a mapping table corresponding to a data identifier of service data, and read a parameter state of a parameter in the mapping table; the mapping table is stored in a first storage system; A second reading module, used for reading the creation time, parameter identifier and parameter value of the parameter in the mapping table according to the parameter state; A storage module is used to determine a storage location of the parameter value in a second storage system according to the creation time, the data identifier and the parameter identifier, and store the parameter value in the storage location.

9. A data processing device, characterized in that: The device comprises: A processor; and a memory configured to store computer executable instructions, which, when executed, cause the processor to perform the data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store computer-executable instructions, and the computer-executable instructions, when executed by a processor, implement the data processing method according to any one of claims 1 to 7.

Images